Issuu on Google+


Hotel Map & Table of Contents

Ambassador 10

6

EAST FOYER

WEST FOYER

BREAKOUTS

5

GENERAL SESSIONS

7

EAST PATIO

AMBASSADOR FOYER PATIO

PATIO

Celebrity BUSINESS CENTER

MOROCCAN BOARDROOM CELEBRITY PLAZA WEST

CELEBRITY PLAZA EAST

3

1 F

2

C

G

EXHIBITORS & POSTER SESSIONS

B

H

MAIN ENTRANCE

4

7

A

5 6

5 6 7

4

8

3 2

9

1

LOADING DOCK LOBBY

1. MISSION HILLS BOARDROOM

6. MIRAGE ROOM

2. CELEBRITY BALLROOM

7. OASIS PATIO

3. CELEBRITY FOYER

8. OASIS

4. POLO ROOM 5. RANCHO ROOM

EXECUTIVE OFFICES

LOBBY LOUNGE RESTAURANT BELLE VISTA

9. OASIS COURTYARD 10. AMBASSADOR WORKSHOPS & BREAKOUTS

MASTER’S PLAZA

Table of Contents

Welcome Letter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 Special Thanks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 2009 Conference Sponsors . . . . . . . . . . . . . . . . . . .7 Schedule-at-a-Glance . . . . . . . . . . . . . . . . . . . . . . . .8 Sunday Conference Program . . . . . . . . . . . . . . . . .9 Monday Conference Program . . . . . . . . . . .10-21

Tuesday Conference Program . . . . . . . . . . .22-42 Wednesday Conference Program . . . . . . . .43-49 Speaker Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 ATP Members List . . . . . . . . . . . . . . . . . . . . . . . . . . 54 Exhibit Hall Map & Exhibitors . . . . . . . . . . . . . . . 55 3


Welcome Letter Dear 2009 ATP Conference Attendees, On behalf of the Association of Test Publishers (ATP) Board of Directors and the Conference Planning Committee, we welcome you to the 10th Anniversary of ATP’s Innovations in Testing Conference. The Conference Planning committee has worked hard over the past year to prepare an informationpacked event covering a great array of topics that influence our industry. We’re excited to offer 13 workshops, 65 breakout sessions, and 21 poster presentations this year! Be sure to join us on Tuesday, February 24th at 8:30 AM for our opening Keynote session, Dr. Mary Teagarden, eminent professor and globalization expert from the Thunderbird School of Global Management will discuss global competition in the current economy. We’re thrilled to have her insight and perspective. Also, don’t miss the panel discussion on Wednesday morning, February 25th at 8:30 AM. ATP veteran Cyndy Fitzgerald is moderating a panel discussion on Testing in a Global Marketplace: Key Learnings that Create Assessment’s Future. Panelists include: Barbara Nichols, CEO, Commission on Graduates of Foreign Nursing Schools David Foster, CEO, KRYTERION, Inc. Mark Anderson, President, Pearson Language Tests Ronald K. Hambleton, Distinguished University Professor, University of Massachusetts Thomas Abrams, Attorney, Vedder Price P.C. We hope that you will also join us for the ATP General Meeting and 2009 Award Presentations on Tuesday morning at 8:00 AM to learn more about the value of ATP and its contribution to the industry. Visit the ATP booth in the exhibit hall to learn more about the benefits that membership brings to you and your organization! If you are not currently a member, this is a great opportunity to join. Lastly, we want to acknowledge this conference would not be possible without the financial support of our valued sponsors and the exceptional time, expertise, and talent the planning committee volunteers have contributed. We are truly thankful to those of you who value what this conference does for the test publishing/assessment services community and have shown your support through participating in the planning of this important event. Innovations are happening every day in the exciting world of testing and assessment. This is the industry event of the year, a place to find new ideas, absorb a wealth of valuable information, and learn practical solutions to testing problems. We are thrilled that you have joined us!

2009

Sincerely,

Casey Marks ks Chair, 2009 ATP Board d off Directors COO, National Council of State Boards of Nursing

Andrew Wiley Chair, 2009 ATP Conference Committee Senior Director, The College Board 4


Special Thanks Executive Committee

Program Committee

Andrew Wiley, Chair, 2009 ATP Conference Committee Senior Director, Client Services, The College Board

Jamie Mulkey, Chair, 2009 Program Committee Granite Bay Associates

Cyndy Fitzgerald, Vice-Chair, 2009 ATP Conference Committee KRYTERION, Inc.

Christine Niero, Vice-Chair, 2009 Program Committee Professional Testing, Inc.

Eugene Burke, SHL Group Limited Chris Gruber, Western Psychological Services William G. Harris, Association of Test Publishers Reid Klion, pan – A TALX Company Casey Marks, National Council of State Boards of Nursing Jamie Mulkey, Granite Bay Associates Christine Niero, Professional Testing, Inc. Lauren Scheib, Association of Test Publishers Cor Sluijter, Cito Dick Soule, Schroeder Measurement Technologies, Inc. Natalie Tracy, Hogan Assessment Systems Denny Way, Pearson Educational Measurement Hazel Wheldon, Multi-Health Systems, Inc. Heather Young, Pearson VUE

Brian Adams, Alpine Testing Solutions Annette Baisden, US Navy Lance Blackstone, Pearson VUE Jay Breyer, Prometric Jim Brinton, Certification Management Services Mary Browne, Professional Examination Service Alan Bugbee, American Society for Clinical Pathology Stephanie Dille, Pearson VUE Arbelle Dolev, McCann Associates Ken Doucet, Professional Examination Services Amanda Evans, PreVisor Laura Ferrando Matt Forster, Integral7, Inc. Naomi Gafni, National Institute for Testing and Evaluation Mark Gierl, Professor of Educational Psychology Jerry Gorham, Educational Testing Service Sandra Greenberg, Professional Examination Service Steve Greenfield, Pearson VUE Shelby Grieve, Microsoft Fanmin Guo, GMAC Ron Hancher, American Health Information Management Association J. Christine Harmes, James Madison University Bill Hogan, Applied Measurement Professionals, Inc. Paul Howarth, Pearson VUE Ted Jackson, SIGMA Assessments Systems, Inc. Neal Kingston, University of Kansas Nelson Lacey, Chartered Alternative Investment Analyst Association Julia Leahy, Prometric Shirley Lewis-Brown, NETPDTC Joy Matthews-Lopez, Prometric Marvin McCray, MRM Consulting Group Mary Beth Mohn, Pearson VUE Liberty Munson, Microsoft Adisack Nhouyvanisvong, Data Recognition Corporation Feng Pan, China National Medical Examination Center Jake Parizek, CTB McGraw-Hill David Pascal, Pearson VUE Shannon Persad, CASTLE Worldwide, Inc. Geri Ray, CTB/McGrawa-Hill Shungwon Ro, Prometric Ryan Ross, Hogan Assessments Jarin Schmidt, Pearson VUE Linjun Shen, National Board of Osteopathic Medical Examiners Anne Sikorski, Pearson VUE Michelle Sparacino, Medical College Admission Test Association of American Medical Colleges Susan Steinkamp, Pearson VUE Carl Swartz, Meta Metrics Chole Torres, Measured Progress Linda Waters, Prometric Patti Wolf, Internet Testing Systems

Sponsorship Committee William G. Harris, Chair, 2009 Sponsorship Committee Association of Test Publishers Lauren Scheib, Vice-Chair, 2009 Sponsorship Committee Association of Test Publishers Ayanna Brady, Vantage Learning Cathy Donath, The Donath Group, Inc. Kirk Lundeen, Integral7, Inc. Bill West, Pearson VUE

Marketing Committee Heather Young, Chair, 2009 Marketing Committee Pearson VUE Natalie Tracy, Vice-Chair, 2009 Marketing Committee Hogan Assessments Jim Brinton, Certification Management Services (CMS) Ben Cantlon, Integral7, Inc. Jocelyn Courtney, pan – A TALX Company Brigid Flood, Integral7, Inc. Bridget Herd, Pearson VUE Tracie Hickman, CASTLE Worldwide Krista Mattern, The College Board Kerri O’Meara, Pearson VUE Karen Paavola, Data Recognition Corporation Robert Pedigo, CASTLE Worldwide

5


2009 Conference Sponsors Platinum Sponsors ________________________________________________________________________

Gold Sponsors __________________________________________________________________________

Silver Sponsors ________________________________________________________________________

Conference Contributor _________________________________________________________________

7


Schedule-at-a-Glance Sunday, February 22nd Registration Open

3:00 PM -

7:00 PM

Celebrity Foyer

10th Anniversary Kick Off

7:00 PM -

8:30 PM

Oasis Courtyard

Registration Open

7:30 AM -

6:30 PM

Celebrity Foyer

Continental Breakfast

7:30 AM -

8:30 AM

Ambassador Foyer

Session Coordinator Meeting

7:30 AM -

8:15 AM

Celebrity G

Opening Keynote Address: Playing for Chinese and Indian Talent: Opportunities, Challenges, and Paths Forward

8:30 AM -

9:45 AM

Ambassador 1-4

Morning Break

9:45 AM -

10:00 AM

Ambassador Foyer

Morning Workshops

10:00 AM -

12:30 PM

Lunch with Exhibitors

12:30 PM -

2:00 PM

Celebrity Foyer Celebrity Ballroom

Monday, February 23rd

Exhibits Remain Open

2:00 PM -

4:30 PM

Afternoon Workshops

2:00 PM -

4:30 PM

Innovation Celebration Reception, Poster Sessions and Exhibits Open

5:00 PM -

6:30 PM

Celebrity Ballroom

Registration Open

8:00 AM -

5:00 PM

Celebrity Foyer

ATP General Meeting and 2009 Award Presentations with Continental Breakfast

8:00 AM -

9:30 AM

Breakout Sessions

9:45 AM -

10:45 AM

Morning Break

10:45 AM -

11:00 AM

Breakout Sessions

11:00 AM -

12:00 PM

Lunch with Exhibitors

12:00 PM -

1:30 PM

Celebrity Foyer

1:30 PM -

2:45 PM

Celebrity Foyer

Tuesday, February 24th

Exhibits Remain Open Breakout Sessions

1:30 PM -

2:30 PM

Afternoon Break

2:30 PM -

2:45 PM

Breakout Sessions

2:45 PM -

3:45 PM

ATP Divisional Meetings

3:45 PM -

5:00 PM

Ambassador 1-4

Celebrity Foyer

Celebrity Foyer

Sponsor Debrief Meeting

4:30 PM -

5:30 PM

Celebrity F

The Next Big Thing

5:30 PM -

7:00 PM

Masters Plaza

Registration Open

7:30 AM -

12:30 PM

Celebrity Foyer

Continental Breakfast with Exhibitors

7:30 AM -

8:30 AM

Celebrity Ballroom

Closing Keynote Panel: Testing in a Global Marketplace: Key Learnings that Create Assessment’s Future Landscape

8:30 AM - 10:00 AM

Ambassador 1-4

Morning Break

10:00 AM -

10:15 AM

Ambassador Foyer

Breakout Sessions

10:15 AM -

11:15 AM

Breakout Sessions

11:30 AM -

12:30 PM

2009

Wednesday, February 25th

Key for Session Types

Clinical  International 

Certification/Licensure  Industrial/Organizational  8

Education  Cross Divisional 


10 th Anniversary Kick Off & Sunday Conference Program

Sunday, February 22nd Registration Open

3:00 PM -

7:00 PM

Celebrity Foyer

10th Anniversary Kick Off

7:00 PM -

8:30 PM

Oasis Courtyard

Registration Open 3:00 PM – 7:00 PM Location: Celebrity Foyer

__________________________________________________________________________________________________________

10th Anniversary Kick Off 7:00 PM – 8:30 PM Location: Oasis Courtyard

Join us on Sunday evening from 7:00 PM – 8:30 PM in the Oasis Courtyard to see just how far we’ve come over the past decade. 9


Monday Conference Program

Monday, February 23rd Registration Open

7:30 AM -

6:30 PM

Celebrity Foyer

Continental Breakfast

7:30 AM -

8:30 AM

Ambassador Foyer

Session Coordinator Meeting

7:30 AM -

8:15 AM

Celebrity G

Opening Keynote Address: Playing for Chinese and Indian Talent: Opportunities, Challenges, and Paths Forward

8:30 AM -

9:45 AM

Ambassador 1-4

Morning Break

9:45 AM -

10:00 AM

Ambassador Foyer

Morning Workshops

10:00 AM -

12:30 PM

Lunch with Exhibitors

12:30 PM -

2:00 PM

Celebrity Foyer

Exhibits Remain Open

2:00 PM -

4:30 PM

Celebrity Ballroom

Afternoon Workshops

2:00 PM -

4:30 PM

Innovation Celebration Reception, Poster Sessions and Exhibits Open

5:00 PM -

6:30 PM

Registration Open 7:30 AM - 6:30 PM Location: Celebrity Foyer

____________________________________________________

Continental Breakfast 7:30 AM - 8:30 AM Location: Celebrity Foyer

____________________________________________________

Session Coordinator Meeting 7:30 AM - 8:15 AM Location: Celebrity G

2009

All who are acting as session coordinators should plan to attend this instructional session.

10

Celebrity Ballroom


Monday Conference Program Morning Break 9:45 AM - 10:00 AM Location: Ambassador Foyer

Opening Keynote Address: Playing for Chinese and Indian Talent: Opportunities, Challenges, and Paths Forward 8:30 AM - 9:45 AM Location: Ambassador 1-4

China and India cannot be ignored based on sheer population size--these two countries have ten times the population of the United States--37% combined compared to 3.8% for the United States. More importantly, China and India cannot Mary B. Teagarden be ignored because of the sheer size of their talent pools compared to that of industrialized countries--healthcare and IT professionals abound. Playing the China or India game and tapping into these talent pools is not for the feint hearted. While the scale of talent opportunity in these gigantic markets is formidable the price of access to talent in China and India is high. Success in tapping into these talent opportunities presents major challenges including understanding the real opportunities in these markets versus the hype; clarifying your company’s specific reasons for entering these markets and understanding your needs; understanding the obstacles you are most likely to encounter there and here; and devising paths around those obstacles. This is a good news-bad news story--in China and India everything is possible, but nothing is easy. Why should your organization play on this field? How can your organization win this talent access game?

Hold on to your sombreros, it’s Fiesta Time! Come join us for margaritas as we celebrate today’s advancements in testing. Be sure to visit the exhibit hall and poster sessions while you’re here. Monday @ 5:00 PM – 6:30 PM

Speaker: Mary B. Teagarden, Professor, Thunderbird School of Global Management

Celebrity Ballroom

See you there amigos!

Morning Workshops 10:00 AM - 12:30 PM Clinical Division Strategic Planning Workshop Location: Celebrity F

Type: Clinical 

Level: Beginner

The Clinical Division warmly invites new, current, and potential members to join us as we continue developing the Division’s strategic plan. In this workshop, we will discuss the steps taken since the strategic planning session in August 08 and continue to explore how to best serve the current and future needs of ATP Clinical Division members. First time attendees are welcome. This is excellent opportunity to put the ATP Clinical Division to work for you and your organization. Speaker: Hazel Wheldon, Publisher, Multi-Health Systems, Inc.

11


Monday Conference Program Morning Workshops 10:00 AM - 12:30 PM Cont.

Cognition and Technology: Test Design for Online Cognitive Diagnostics Location: Celebrity G

Type: Education 

Level: Intermediate

Cognitive diagnostic assessments (CDA) are designed to measure the knowledge structures and processing skills of students in a given subject domain. The CDA model links the test score interpretations to the cognitive skills required to solve different test items. The attribute hierarchy method (AHM) is a cognitive diagnostic assessment procedure that evaluates the cognitive skills or attributes measured by test items. Test items are designed for each attribute based on declarative or procedural knowledge requirements. The test items are then computer scored and a diagnostic report is generated based upon the attribute structures. Total test score and diagnostic sub-scores can then be provided for each examinee. The test score interpretations link cognitive and learning psychology with instruction. This allows decisions to be made for improving or re-mediating specific cognitive skills for each examinee. In this session, an innovative, technologybased system for diagnostic testing is described and examples at two grade levels are shared.

most common statistical analysis performed on test results (classical test theory) to de-mystify the process and to help attendees to make future, informed decisions based on analysis results. Using both summary item statistics and option analyses, attendees receive instruction on what such statistics mean, how to interpret them and how they can use these results to decide whether to keep or delete an item from their test. Additionally, they learn how to use statistics to identify ways to improve the content of their test items for future use, and how to identify potentially mis-keyed items, poorly-performing distracters, ambiguity, misinterpretation and more. Finally, the session provides an overview of how these statistics are used to create balanced forms. Speakers: Kim Thayn, Psychometrician & Partner, Certification Management Services Scott Thayn, President & Psychometrician, Certification Management Services

____________________________________

If You Build it, Will They Come? Why Some Programs Fail While Others Succeed Location: Oasis 4

Type: Certification/Licensure

Level: Beginner Speaker: Renate Majeau, Exam Manager, Alberta Education

____________________________________

Head to Head - Online Technology Vendors Sound Off: The View Location: Rancho Level: Beginner

Type: Certification/Licensure

Tired of listening to the same old viewpoints? Tired of marketing hype? This session brings technologists from the industry’s most innovative companies together to discuss key online assessment topics. The panel members discuss innovations in online assessment and their views on the future of online testing. Topics include test security, use of online testing for high-stakes and credentialing assessment; (the real) online testing costs; use of online testing in formative assessment and online practice testing communities; international online testing; supporting innovative item types; and other topics identified by a survey of previous ATP workshop participants. For the past two years at ATP, this workshop has been well attended and well received by participants. This year’s workshop builds on that success with an enhanced format and expanded topic list based on input from previous participants. Speakers: David Foster, CEO, KRYTERION, Inc. Eric Shepherd, CEO, Questionmark Patrick Ward, President, Internet Testing Systems Scott Elliot, President, SEG Assessment, SEG Research

____________________________________

How to Read and Interpret Statistical Analyses of Test Data

2009

Location: Mirage Level: Beginner

Type: Certification/Licensure

As soon as some people hear the word statistics, their muscles stiffen, their hearts race and a feeling of impending doom overcomes them. Then the thought strikes them: “Do I really need to analyze these data?” This session breaks down the 12

While a good assessment can be challenging to build, a successful program can be even more difficult. Even when exams are valid, reliable and legally defensible, they may not “take off” in the marketplace. Why do good exams sometimes perform poorly? In this session, the presenters discuss how marketing principles can be applied to make programs more successful in the marketplace. This session is designed to help everyone, whether the entire test development, program management and promotion department or individuals who need to work with them, better position the exams internally and externally for the greatest success. Speakers: Don Sorensen, VP, Marketing, Caveon Test Security Jerry Christensen, Director, Partnerships, KRYTERION, Inc. Quinn Sutton, VP, Marketing, KRYTERION, Inc. Stephanie Dille, VP, Marketing, Pearson VUE

____________________________________

Measurement Decision Theory Location: Oasis 5-7

Type: Certification/Licensure

Level: Intermediate

If you are is interested in classifying examinees into one of a finite number of discrete categories, such as pass/fail or proficient/ basic/below-basic, then decision theory, rather than classical measurement theory or item response theory, can be the model of choice. Relative to other models, decision theory requires fewer test questions to achieve the same level of precision and has less stringent requirements to properly calibrate the test questions. This workshop presents the use of decision theory as a tool for classifying examinees based on their item response patterns and pre-calibrated test questions. Scoring, classification rules and adaptive testing are discussed. This is a hands-on workshop and attendees should bring a laptop. An Excel spreadsheet is provided. The only pre-requisite mathematics skill is the ability to multiply and divide decimals. (For more information, visit http:// edres.org/mdt/.) The model is quite appropriate for performance testing, programs with a limited number of test-takers and for diagnostic testing. Speakers: Johnny Weissmuller, US Air Force Lawrence Rudner, VP, R&D, GMAC


Conference Map & Table of Contentss

Don’t Miss the Innovations in Testing Keynotes and General Session!

Opening Keynote Address: Playing for Chinese and Indian Talent: Opportunities, Challenges, and Paths Forward 8:30 AM - 9:45 AM Location: Ambassador 1-4 Speaker: Mary B. Teagarden, Professor, Thunderbird School of Global Management

Monday Congratulations Dr. David Arnold

Dr. John Jones

Lawrence M. Rudner

ATP General Meeting and 2009 Award Presentations with Continental Breakfast 8:00 AM - 9:30 AM Location: Ambassador 1-4 Speakers: Casey Marks, William G. Harris Congratulations to The Award for Professional Contributions and Service to Testing Recipients: Dr. David Arnold, Dr. John Jones Hats off to The Career Achievement Award Recipient: Lawrence M. Rudner

Tuesday Closing Keynote Panel: Testing in a Global Marketplace: Key Learnings that Create Assessment’s Future Landscape 8:30 AM – 10:00 AM Cyndy Fitzgerald

Barbara Nichols

David Foster

Location: Ambassador 1-4 Moderator: Cyndy Fitzgerald Panelists: Barbara Nichols, David Foster, Mark Anderson, Ronald K. Hambleton, Thomas Abrams

Thomas Abrams

Ronald K. Hambleton

Mark Anderson

Wednesday 13


Monday Conference Program Afternoon Workshops 2:00 PM - 4:30 PM Cont.

Where’s the Thief?: Getting Results from Your Exam Security Investment Location: Oasis 1-3

Type: Certification/Licensure 

Level: Intermediate

Participants in this working session hear how two very different organizations have gained significant benefits from their exam security programs. The lessons learned from these organizations form the basis for discussion of each participant’s specific exam security challenges and the design of security solutions that bring value to the unique situations. In the first half of the session, EMC and the Graduate Management Admission Council (GMAC) discuss the processes they have implemented to investigate and take action against exam theft and cheating. The discussion focuses on relevant business cases that were used to help each participant design solutions that brought a return to their business. In the second half of the workshop, participants divide into “Birds of a Feather” groups, led by a facilitator, to discuss how the EMC and GMAC solutions can be applied to and modified for their unique challenges. Individuals are encouraged to discuss their specific exam security challenges and design solutions that bring a return to their business. Speakers: Ashok Sarathy, Director, GMAT Operations, GMAC Daniel Eyob, Associate Director, GMAT Operations, GMAC Gene Radwin, Assessment Program Manager, EMC Liz Burns, Program Manager, EMC Proven Professional Program, EMC

____________________________________________________

Lunch with Exhibitors 12:30 PM - 2:00 PM Location: Celebrity Foyer

____________________________________________________

Exhibits Remain Open 2:00 PM - 4:30 PM Location: Celebrity Ballroom

____________________________________________________

all departments. The workshop also reviews the financial impact and major cost drivers associated with developing a new exam, such as item writing, field testing and software. Statistical requirements are woven throughout. A reference list of useful books, documents and websites is included and serves as a basis for the discussions and question-answer periods. Speakers: Andrew Wiley, Senior Director, The College Board Jerry Melican, Chief Psychometrician, The College Board Wayne Camara, VP, Research & Psychometrics, The College Board

____________________________________

Ensuring the Right Candidates Pass Your Examination Location: Oasis 1-3

Type: Certification/Licensure

Level: Intermediate

The purpose of licensure and certification exams is to identify and pass examinees who are qualified to practice, and to ensure that examinees who are not qualified do not pass. Incorrect decisions carry huge consequences, regardless of whether unqualified applicants are awarded passing scores or qualified applicants fail to achieve them. As a result, all possible avenues must be explored in order to eliminate erroneous results. This presentation takes a global perspective in exploring how the test development processes (developing a test blueprint, identifying item writers, reviewing test items, automated test assembly, etc.) and test administration practices (test time and length, computerized testing options, security analyses, item rotation, standard setting, etc.) can be integrated with post-administration activities to help ensure that the “right” examinees pass. The session focuses on instituting best practices throughout the testing process to achieve this result. Speakers: Chuck Kunce, VP, Assessment & Technical Programs, National Institute for Automotive Service Excellence Deborah Harris, Director, Measurement Research, ACT, Inc. Renee Sullivan Warias, Senior Manager, Inventory Development & Production, Certified Public Accountants

____________________________________

Afternoon Workshops

Measuring Continued Competence through Recertification

2:00 PM - 4:30 PM Developing a New Test from Conceptualization to Maintenance Location: Rancho Type: Certification/Licensure

2009

Level: Beginner

This session is for beginners developing a new test. Key decision points and the processes involved from start to maintenance are explained. Questions and issues that need to be addressed in developing a reliable, fair and valid test are raised and discussed. Quizzes and hands-on opportunities are included. Emphasis is on outlining necessary steps, identifying appropriate stakeholders, determining the construct to be measured, identifying resource requirements, and setting a realistic schedule and costs. Additional topics covered include steps to determine item types, delivery system (p-p or CBT), necessity for regular communication, prep materials for each constituent group and integrating the workflow among 14

Location: Mirage

Type: Certification/Licensure

Level: Intermediate

Recertification is a requirement of most certification programs, including ANSI ISO/17024 and other accreditation programs. Although recertification has many meanings and models of practice, it is frequently used as a means of assuring end users of certifications that the individuals holding the designation remain current in their knowledge and skills. In reality, many certification programs do not or cannot deliver on this. This workshop is designed to assist participants in developing and implementing recertification programs based on the concept of continued competence. Participants interactively take part in exercises designed to assist them in assessing the viability of their recertification programs and steering a community through the political, financial and implementation challenges. Strategies assessing continued competence through retesting as well as other models for recertification assessment are addressed. Time is allotted for questions,


Monday Conference Program group discussion, and the exchange of experiences and ideas. Speakers: Christine Niero, VP, Professional Certification, Professional Testing, Inc. Cynthia Woodley, VP, Professional Testing, Inc. Dale Cyr, CEO, ARDMS Gwen Henderson, Director, Registrant Services, American Registry for Diagnostic Medical Sonography

____________________________________

Quality Design, Control, Monitoring and Improvement for Certification Testing Programs Location: Oasis 4 Type: Certification/Licensure Level: Beginner

All certification testing programs must have a focus on quality. Typically, though, quality is thought of only as having wellwritten test questions and reporting accurate test results. However, quality needs to be built into all aspects of the certification program, whether conducted by the certifying agency or their vendors. The purpose of this workshop is to introduce several process tools and techniques that can be used to assure high quality tests and services essential to process quality design, control, monitoring and improvement. In the design and control phases, process identification and mapping is covered, along with the identification of critical output measures or metrics. The second section of the workshop covers tools and techniques for monitoring, analyzing, reporting, and interpreting the metrics and the subsequent process improvement activities. Best practices and examples of process identification and mapping techniques, important metrics and reporting methodologies from current testing programs are presented and discussed throughout the workshop. Speakers: Cheryl Wild, President, Wild & Associates David Anderson, Senior Psychometrician, Educational Testing Service Rohit Ramaswamy, VP, Client Relations, Oriel, Inc.

ATP Innovations Networking Event 3:00 PM - 4:30 PM Location: Ambassador 5 & 6

Test Development in a Global Economy: A Practical Guide to Best Practices in Test Construction, Localization and Adaptation Location: Celebrity G

With the global economy driving multi-language and multigeography employment testing programs, the issue of equivalence and comparability of test scores is becoming increasingly important. This session shares learning from a large-scale localization program covering both self-report questionnaires and ability tests, and spanning 35 languages illustrated through case study material covering Arabic and Chinese. This is an extension of the session run at last year’s ATP conference that received very positive ratings and requests for it to be run again. The session will be of interest to test developers, test development and localization program managers, translators as well as psychometricians. Speakers: Carly Rebello, Science & Innovation Consultant, SHL Group Limited Eugene Burke, Science & Innovation Director, SHL Group Limited

____________________________________

The Design and Development of Innovative Items - From Soup to Nuts Location: Oasis 5-7

Type: Certification/Licensure

Level: Intermediate

This session begins with the presentation of a 6-step process for the design of innovative item types. This process takes you from an initial analysis of the construct needs of an exam to the final production of exam materials. This full approach will help exam programs add innovative items that are of high measurement quality, logistically practical, and acceptably affordable. The second presentation illustrates several aspects of this process, using data and examples from several large operational programs. Our experience with innovative items has shown that implementing the following processes improves the quality of the assessment and provides a framework for developing these items as part of the normal test development process:

Got questions? Your peers have answers!

t

$MBSJGZJOHUIFDPHOJUJWFTLJMMTBOELOPXMFEHF areas for assessment prior to software development

t

%FWFMPQJOHTQFDJĂśDNBUFSJBMT JF UFNQMBUFT   item writing guidelines, and item writer training) for each innovative item type

t

$POEVDUJOHTFWFSBMSPVOETPGVTBCJMJUZUFTUTXJUI a representative sample of candidates early in the test development cycle

t

"OBMZ[JOHCPUIDPSSFDUBOEJODPSSFDUSFTQPOTFT  from trialing data for each innovative item type

Join us for refreshments and conversation at our inaugural networking event.    

Type: Industrial/Organizational 

Level: Intermediate

t/FUXPSLXJUIJOEVTUSZDPMMFBHVFT t4IBSFUFTUJOHCFTUQSBDUJDFT t#VJMEOFXSFMBUJPOTIJQT t$POUJOVFUIFDPOGFSFODFACFZPOEUIFDPOGFSFODF

Don’t miss this chance to connect with your colleagues. See you there!



Speakers: Cynthia Parshall, Consultant, Measurement Consultant J. Christine Harmes, Assistant Professor, Graduate Psychology, James Madison University Kirk Becker, Manager, Psychometric Services, Pearson VUE

15


Monday Conference Program Poster Sessions 5:00 PM - 6:30 PM Cont.

Innovation Celebration Reception, Poster Sessions and Exhibits Open

A Picture is Worth a Thousand Words: How Enhanced Candidate Authentication Procedures Deter Proxy Testing and Restore Employer Confidence Location: Celebrity Ballroom

5:00 PM - 6:30 PM

Type: Certification/Licensure

Level: Beginner

Poster Session #3

Location: Celebrity Ballroom

Poster Sessions Everything You Always Wanted to Know about Getting a Session Accepted at ATP but Were Afraid to Ask Location: Celebrity Ballroom

Type: Cross Divisional 

Level: Beginner

Poster Session #1

Ever wonder what it takes to get a session accepted for the ATP conference? Do you ever think, “wow, why wasn’t my session accepted?” The answers to your questions are here! ATP 2009 Program Chairs Jamie Mulkey and Christine Niero will provide you with an informative session on how abstracts are selected for the ATP Innovations Conference. Topics such as complete submission forms, clear learning objectives, and creating a compelling, interactive session description will all be discussed. This session is for both new and existing session submitters. Presenters: Christine Niero, VP, Professional Certification, Professional Testing, Inc. Jamie Mulkey, President, Granite Bay Associates

____________________________________

The Use of Innovative Items and Their Implications for Test Security Location: Celebrity Ballroom Level: Intermediate

Type: Cross Divisional  Poster Session #2

2009

Items on computer-based tests that incorporate elements that cannot be included on paper-and-pencil tests (e.g., audio and video) are referred to as innovative items. Innovative items can expand an exam’s construct representation by providing opportunities to better measure constructs difficult to measure by traditional text-based items. As such, the use of innovative items has been increasing. Recently, the claim has been made that innovative items are more memorable than traditional items, therefore threatening test security and compromising the validity of inferences made from test scores (Impara and Foster, 2006; Sireci and Zenisky, 2006). However, this claim has not yet been empirically tested. The current study being discussed employs a mixed-methods approach to test the claim that innovative items are more memorable, and to examine what specific item content is recalled by test takers. Preliminary results suggest that innovative items may not be more memorable than traditional items. However, interesting patterns are emerging regarding the actual item content that was recalled. These results have important security and validity implications for testing programs considering the use of innovative items. Presenters: Carol Barry J. Christine Harmes, Assistant Professor of Graduate Psychology, James Madison University Megan France Pamela Kaliski, Graduate Student, James Madison University 16

Proxy testing is a growing threat to certification. How can testing programs ensure that the credential recipient is the same person who passed the certification exams? Cisco and Pearson VUE raised the bar for information technology (IT) certification by adding a new candidate authentication feature to the admissions process. Digital photos and digital signatures are captured during check-in at the test center, and the same photo appears on the score report and on an authentication website. This feature helps protect the integrity of the exam by reducing incidents of proxy testtaking. As an added benefit, test takers can share access to the authentication site with others, so employers can verify that the interviewee is the same person who took the exam. Presenters: Fiona Collins, Cisco Implementation Manager, Pearson VUE Amy Brooks, Cisco Systems

____________________________________

ARRT’s Experience: Applying Six Sigma to the Test Development Process Location: Celebrity Ballroom

Type: Certification/Licensure

Level: Intermediate

Poster Session #4

The American Registry of Radiologic Technologists found that a large number of Six Sigma investigative and solution-based tools were highly applicable to the test development process. The session shows how ARRT uses these tools to enhance the understanding of the complexities within and surrounding the test development cycle and how these tools can be applied to other programs. By applying the Six Sigma techniques, organizations can achieve large gains in the efficiency of test development processes and reduce the possibility of errors. Presenter: Lauren Wood, Director, Psychometrics, American Registry of Radiologic Technologists

____________________________________

Best Practices: Stages in Designing Competency-Based Assessments Location: Celebrity Ballroom

Type: Certification/Licensure

Level: Intermediate

Poster Session #5

The Certified General Accountants Association of Canada (CGA) is a professional accounting body that offers a certification program modeled on competency-based assessments. The CGA model is based on industry best practices starting with the practice analysis, identifying core competencies that are mapped to the courses, designing blueprints and developing the assessment tools. Validation of the assessment tool is conducted through a verifiable process that involves training assessors, developing marking keys, a standard-setting process that determines the cut score, borderline reads and reporting final scores. The assessment process involves a performance measurement tool that ensures fairness and consistency in assessing the performance of individual assessors and an audit process that


Monday Conference Program Poster Sessions 5:00 PM - 6:30 PM Cont.

allows for verification of any challenges to the assessment process. This presentation illustrates a road map for the effective implementation of a competency-based assessment, including examples of a practice analysis, competency mapping, assessment blueprint, item development, and examination assembly, assessment and reporting. Presenter: Nazir Nenshi, Director, Assessments, Certified General Accountants Association of Canada

____________________________________

Classical and IRT Analyses for Emergency Vehicle Driver Simulation Performance Location: Celebrity Ballroom

Type: Certification/Licensure

Level: Intermediate

Poster Session #6

Classical and IRT measurement analyses can be used to evaluate individuals and tasks delivered via performance simulations and to analyze the dependencies among the task performance scores. In this study, classical and IRT analyses are used to evaluate candidate performance with a pool of 35 simulated emergency vehicle driving tasks presented with increasingly more difficult driving scenarios. The sample size is 214 drivers with an average of 14 alternative driving scenarios. Simulation tasks are polytomously scored. Classical and IRT analyses were conducted with the incomplete matrix of simulation task scores. The 35 overall driver simulation tasks are ordered by empirical task difficulty values, by point-biserial discrimination correlations and Rasch calibration values across driving scenarios. IRT curves are presented for each task and information analysis identifies which simulation tasks provide the most information at different points across the proficiency continuum. The simulation system specifies three types of driver errors (critical, collision and completion). The classical and IRT analysis task results are compared with the frequency of different types of driver errors. Presenter: James Olsen, VP & Chief Scientist, Alpine Testing Solutions

____________________________________

Incorporating Test Users in the Test Development Process: A Case Study Location: Celebrity Ballroom Type: Certification/Licensure Level: Intermediate

Poster Session #7

Test developers are constantly facing challenges in terms of new clients, new content areas and new purposes for the test outcome and interpretation. Clients often have a limited understanding of the necessity and complexity of proper test development procedures. Clients might know the need for a test and how they plan to use its outcome but do not know how to define the objectives. Without this definition, it is difficult to develop a blueprint and, hence, almost impossible to end up with a valid test, regardless of how much time and effort is given to item construction. In cases like this, it is necessary for both parties to be involved in the test development process. This presentation describes a model for analyzing and defining a new content area, and for developing the blueprint in collaboration with the test user and the test developer. Empirical evidence is provided in a case study describing a test development project conducted at Enlight. The presentation discusses the advantages and disadvantages of this model.

2009

Presenter: Lise-Lotte Appelgren, Test Consultant, Enlight AB

18

Proxy Test Takers: Paying Someone to Take Your Test... Really Location: Celebrity Ballroom Level: Intermediate

Type: Certification/Licensure Poster Session #8

Proxy test takers are becoming more brazen by offering services through websites. As part of a sting, the presenter hired a proxy test taker to see if he could acquire a certain IT certification. This led him to an individual in China who promised the certification for only $500 up front and $500 after the test was taken. This poster session outlines how programs can find out if proxy test takers are offering their tests and what to do about it. The presenter’s company also conducted a data forensic analysis of the test results to determine where the actual test was administered and whether or not this individual could be matched to other completed tests. Presenter: Don Sorensen, VP, Marketing, Caveon Test Security

____________________________________

Replacing Physical-Based Testing with Advanced Performance Item Types Location: Celebrity Ballroom Level: Beginner

Type: Certification/Licensure Poster Session #9

This poster session demonstrates replacing a physical-based exam with a performance exam that utilizes interactive 3D models. Three-D technology (Glass3D) was created to allow real-time 3D interactive test items to be developed. Currently Glass3D is deployed in testing centers such as Pearson VUE. ExamStudio, item authoring software, supports advanced item types that utilize 3D technology. Glass3D was embedded into ExamStudio to allow examiners to author and view 3D performance-based items including all candidate interactions. To support 3D-based performance item types, Glass3D allows interactions between 3D models (e.g., between network devices and cabling models). These interactions generate intersection points during candidate testing. This technology allows candidates to place cables into network equipment. The location of cables is able to be accurately measured. Traditionally this test would be given with real equipment. With the 3D technology, the physical exam has been replaced with a performance-based test. Presenter: John Baker, CEO, Zoomorphix Systems

____________________________________

Security Awareness for Test Takers Location: Celebrity Ballroom Level: Beginner

Type: Certification/Licensure Poster Session #10

Test publishers are constantly challenged by test takers who, knowingly or unknowingly, participate in illegal testing behavior from test preparation through test completion. While corrective action is always an option for addressing these threats to the integrity of the testing program, there recently has been a strong focus on how to proactively address these behaviors at the test taker level before they occur. This session highlights the different methods that some test publishers have used to provide security-awareness education to


Monday Conference Program test takers. Particular consideration is given to educating test takers on what constitutes illegal behavior, what the consequences of these actions are and the different methods that can be used to communicate this information to the test taker. The experiences of two large scale testing programs are used to illustrate efforts in this area of test security. Presenters: Layne Pethick, Test Security Administrator, GMAC Mark Poole, Director, Test Security, Pearson VUE

Virtual Lab Exams: An Update from Microsoft Location: Celebrity Ballroom Level: Intermediate

Type: Certification/Licensure Poster Session #15

Microsoft is making progress in performance-based testing with a virtual lab emulation exam solution. This session provides a glimpse of the future and a candid discussion of the challenges that lie ahead.

____________________________________

The Ordered Item Booklet Angoff: Combining the Best of Both Worlds Location: Celebrity Ballroom Level: Intermediate

Type: Certification/Licensure Poster Session #11

Standard setting is a critical component of any testing program, and numerous methods exist for systematically determining an appropriate cut score. Two test-centered methods, Angoff and Bookmark, have become prominent in setting passing standards on credentialing exams. Despite their numerous applications, criticisms of both methods have arisen. In response, this session describes the use of a hybrid method that combines the ideal features of the Angoff and Bookmark methods and addresses the concerns of both methods. This new method, referred to as the ordered item booklet Angoff, presents standard setting panelists with the test items in order of difficulty but allows them to provide yes-no Angoff ratings for each item. This presentation describes the theoretical justification for this method and an implementation of the method for a credentialing testing program. Presenters: Alejandra Garcia, Exam Developer, Citrix Systems Russell Smith, Senior Psychometrician, Alpine Testing Solutions Susan Davis, Psychometrician, Alpine Testing Solutions

____________________________________

Top 10 Reasons Why Your Website May Be Killing Your Certification Program Location: Celebrity Ballroom Type: Certification/Licensure Level: Beginner

Poster Session #14

Web users form first impressions of Web pages in as little as 50 milliseconds (1/20th of a second). Is your website making a good first impression? Do you give your customers a good online experience, or do they have to suffer every time they visit your site? Even for those who hold an exclusive program, making the site more customer friendly is beneficial. Candidates who can find the information they need won’t eat up valuable employee time that could be spent on more productive activities. This session starts by explaining the most common ways a website may actually be hindering a certification program. It opens participants’ eyes to new ideas and approaches that can be implemented immediately. Presenter: Kaiya Karges, Senior Web Designer, Pearson VUE

Presenters: Amy Tipton, Program Manager, Microsoft Learning Julie Devine, Content Development Manager, Microsoft Learning - Assessments and Certification Exams

____________________________________

Garnering Voluntary Participation in Test Development Studies Location: Celebrity Ballroom Type: Education  Level: Beginner

Poster Session #16

Many testing programs endeavor to obtain optimal sample sizes, while facing the challenge of recruiting voluntary participants for studies. One such program is the GED testing program, which provides a second opportunity for adults to certify that they possess high-school level academic knowledge and skills. As part of the test development process, performance data are collected from graduating high school seniors through the voluntary participation of high schools. This session shows how the GED Testing Service modified both study designs and incentives, and presents the impact on voluntary participation of high schools in test development. Presenter: Carol George-Ezzelle, Senior Psychometrician, GED Testing Service

____________________________________

Operational Considerations for Implementing Automated Essay Scoring in K-12 Testing Location: Celebrity Ballroom Type: Education  Level: Intermediate

Poster Session #17

Automated essay scoring (AES) systems have been in development for more than 40 years but have only recently begun to be widely utilized in large-scale testing. Using an AES system, greatly reduces the time involved in scoring this performance task. In addition to being highly efficient, they have proven to be consistent, objective and impartial. The results of many studies have proven AES to be capable of producing valid and reliable scores that accurately reflect the writing skills of students who produce the essays. Although the various AES systems have been investigated extensively and implemented in a number of large-scale and high-stakes testing situations, to date very few states use AES for their NCLB assessments. This presentation addresses operational considerations for implementing AES in a high-stakes, K-12 testing program, including a presentation of its advantages and disadvantages. It serves as a point of reference for testing companies and state departments of education, when considering whether to implement such a system. Presenters: Laurie Davis, Manager, Psychometric Services, Pearson Paul Nichols, Director, Research Services, Pearson Jason Meyers, Research Scientist, Pearson

19


Monday Conference Program Poster Sessions 5:00 PM - 6:30 PM Cont.

An Application of Two Differential Item Functioning Detection Methods: An Integral Step in the Computer-Adaptive Test Development Process Location: Celebrity Ballroom Type: Industrial/Organizational  Level: Intermediate

Poster Session #19

The identification and revision of biased items should be integral parts of the development process of computeradaptive tests (CATs), in which it is possible for examinees to be presented with varying combinations of items. Although some combinations may not lead to adverse impact, it is possible for alternate combinations, which may include biased items, to do so. At the root of item bias is differential item functioning (DIF), and there are numerous DIF detection techniques. The purpose of this study is to present two DIF detection methods and demonstrate their utility within the CAT development process. The Mantel-Haenszel (MH) and Differential Functioning of Items and Test (DFIT) methods will be applied to empirical data to identify any items that may function differently for men and women. The test was designed to measure abstract, numerical, and verbal reasoning skills at an executive, managerial, and graduate level. Preliminary results indicate that there are true ability differences between the groups on all three scales. The study will help to emphasize the importance and the utility of DIF analyses within the CAT development process. The final results may help developers determine appropriate methods of DIF detection. Presenters: Sara Gutierrez, Research Scientist, PreVisor

____________________________________

Face Validity of Innovative Items: Test-Taker Perceptions of a Multimedia Simulation Assessment Location: Celebrity Ballroom Level: Intermediate

Type: Industrial/Organizational  Poster Session #20

As technology advances, computer-based assessment and innovative items are being utilized more and more. Innovative items have the ability to change the test-taking experience and potentially assess a construct more completely than tests consisting of traditional items. As long as technology provides developers with the ability to produce construct cutting-edge and flashy item types, test developers will continue to utilize them. It is essential that research in the area of innovative items continues to grow. Although one assumed benefit of innovative items is the increased face validity, or authenticity of the task within the assessment, little research has addressed this empirically. The current study investigates test-takers’ perceptions of face validity, constructs representativeness and overall opinion of an assessment containing innovative items (a multimedia simulation). The results are compared and contrasted with the perceptions of examinees taking a textbased version of the same item set of items.

Innovation and Going Green: Use of WebEnabled Technologies in Applicant Screening Location: Celebrity Ballroom Level: Intermediate

Type: Industrial/Organizational  Poster Session #21

In today’s world of eco-friendly and budget conscious business, having job applicants travel for testing and interviewing is costly, time-consuming and difficult to manage. With technologies for online test-taking already in place, online interviewing is becoming a more attractive option. Web-enabled cameras can be used to eliminate the cost of travel, saving money and resources, while still maintaining face contact. Utilizing electronic reports, meetingroom websites and phone conference services to consult when providing feedback on hire/not-to-hire decisions also reduces travel costs, while minimizing the environmental impact of the consultation. This poster outlines issues related to maintaining test security, the benefits of online testing centers, ways to verify the identity of the applicant, receiving test results electronically and consulting to agencies via the Web. Presenter: Stephen Griffin, Chief Psychologist, IPAT, Inc.

____________________________________

What Would My Employer Want Me to Say? Location: Celebrity Ballroom Level: Beginner

Type: Industrial/Organizational  Poster Session #22

Personality tests have a long history of successfully predicting workplace performance; however, personality tests are frequently criticized for containing items in which applicants can respond in a socially desirable manner. The Craft Personality Questionnaire has a built-in, behind-thescenes faking pattern, which identifies when candidates are attempting to respond favorably. If the faking pattern is “tripped,” candidates receive a real-time warning message that prompts them to return to the questionnaire and recheck their answers. At that time, candidates are allowed the opportunity to change their answers. The purpose of this research study is to analyze data for more than 8,000 candidates who changed answers once the warning message was displayed. The goal of the analysis is to determine what personality traits are consistently changed by applicants. This could have a beneficial impact on workplace personality assessments, as many personality assessments measure similar constructs. If employers and test publishers understand what constructs applicants feel are the most socially desirable, researchers can narrow their focus in test development on constructs that may be harder to fake. Presenter: Amanda Evans, Professional Services Consultant, PreVisor

Presenter: Sara Gutierrez, Research Scientist, PreVisor

21


Tuesday Conference Program

Tuesday, February 24th Registration Open

8:00 AM -

5:00 PM

ATP General Meeting and 2009 Award Presentations with Continental Breakfast

8:00 AM -

9:30 AM

Breakout Sessions

9:45 AM -

10:45 AM

Morning Break

10:45 AM -

11:00 AM

Breakout Sessions

11:00 AM -

12:00 PM

Lunch with Exhibitors

Celebrity Foyer Ambassador 1-4

Celebrity Foyer

12:00 PM -

1:30 PM

Celebrity Foyer

Exhibits Remain Open

1:30 PM -

2:45 PM

Celebrity Foyer

Breakout Sessions

1:30 PM -

2:30 PM

Afternoon Break

2:30 PM -

2:45 PM

Breakout Sessions

2:45 PM -

3:45 PM

ATP Divisional Meetings

3:45 PM -

5:00 PM

Sponsor Debrief Meeting

4:30 PM -

5:30 PM

Celebrity F

The Next Big Thing

5:30 PM -

7:00 PM

Masters Plaza

Celebrity Foyer

Registration Open 7:00 AM - 5:30 PM Location: Celebrity Foyer

__________________________________________________________________________________________________________

Sponsored By:

Tuesday night we will focus the spotlight on what the future holds for the testing industry. Join us from 5:30 PM – 7:30 PM in the Masters Plaza to find out.

WIN A GPS

2009

Enter for a chance to win a GPS navigational system! *Must be present to win. 22


Tuesday Conference Program ATP General Meeting and 2009 Award Presentations with Continental Breakfast 8:00 AM - 9:30 AM Dr. David Arnold

Location: Ambassador 1-4

Dr. John Jones

Lawrence M. Rudner

Meet your colleagues and hear from leaders about legislative, regulatory and judicial initatives that impact testing. Network with colleagues and hear about the advances and best practices that are shaping the industry. Speakers: Casey Marks, Chief Operating Officer, NCSBN (National Council of State Boards of Nursing) William G. Harris, Executive Director, Association of Test Publishers

The Awards for Professional Contributions and Service to Testing Dr. David Arnold, General Counsel for Wonderlic, Inc. and Dr. John Jones, a Vice President and Chief Scientist at Vangent, Inc., were nominated as a team for the Professional Contributions and Service to Testing award. Drs. Arnold and Jones have made significant contributions to the test publishing industry that span the last three decades. They were heavily involved in saving/legitimizing the integrity testing industry during the late 1980s and early 1990s when many states had proposed broad language to severely restrict and even outlaw these types of job-related tests. These two psychologists provided lobbying leadership and support at the state and federal level and teamed with top government lobbyists to ensure that their lobbying efforts were effective. Moreover, Drs. Arnold and Jones worked with leading statisticians to ensure that the most advanced types of applied research (e.g., meta-analytic studies) were conducted and published to optimally legitimize this area of practice. Drs. Arnold and Jones were among the original founders of the Association of Personnel Test Publishers, which ultimately became ATP. Dr. Arnold has continuously served as General Counsel of ATP helping to ensure its continuity, and Dr. Jones chaired the I-O Division and various missioncritical committees (e.g., Testing Standards Committee) early on when ATP faced some of its toughest challenges from legislators.

The Career Achievement Award Lawrence M. Rudner, Vice-President of Research and Development at the Graduate Management Admission Council and recipient of the ATP Career Achievement Award has a long career of innovation and service to the measurement community. His work at GMAC® has included test validation, adaptive testing, professional standards, QTI specifications, test security, data forensics, and contract monitoring for the GMAT® exam. He conducted some of the first research on several lasting measurement topics, including the use of IRT to assess item bias, parameter invariance, assessment of person fit, validity of a composite measure, and classification accuracy. Dr. Rudner’s impact has also been through promoting the exchange of information, tools, and ideas. For 18 years he served as the Director of the immensely popular ERIC Clearinghouse on Assessment and Evaluation. In that role, he created one of the first quality websites for the measurement community. At one point, that site was ranked in the top 10,000 sites on the entire Internet. He is well known as the founder and co-editor of the online journal Practical Assessment Research and Evaluation (PAREonline.net), which is now the most widely read journal in its field.

Breakout Sessions 9:45 AM - 10:45 AM ATP Best Practice Working Group: Update on Operational Best Practices Location: Oasis 6 & 7 Type: Education  Level: Beginner

As charged by the leadership of ATP, the Best Practice Working Group has created a document outlining operational best practices. This session presents the background, actions taken, and philosophy regarding the development of the document, and articulates what the assessment industry identifies as best-of-breed regarding implementation. This session includes a demonstration of the review website, an example of the reporting best practice, a discussion of how these practices could be used by states and testing organizations, how the best practices document will be disseminated and plans for periodic updates. The session concludes with a reaction and critique by a respected university researcher and practitioner. Speakers: Jon Twing, Executive VP, Pearson Lisa Ehrlich, VP, Measured Progress Wayne Camara, VP, Research & Psychometrics, The College Board Wes Bruce, Assistant Superintendent for Accreditation, Assessment & Licensing, Indiana Department of Education

Automated Scoring Technologies Location: Polo Room Level: Intermediate

Type: International 

The Pearson Test of English (PTE) is an innovative, valid and reliable tool that uses 20 integrated item types to assess the ability of learners of English to use the English language for oral and written communication in the context of Englishmedium higher education. The instrument requires candidates to engage in a wide range of interactive and integrative tasks based on naturalistic examples of English-language use in academic settings. The test deploys innovative item formats such as various types of selected-response (multiple-choice, hotspots, highlight, and drag and drop) for the oral and written receptive skills and constructed-response (short-answer and extended discourse) for the oral and written productive skills. There is much research to substantiate the reliability and validity of Pearson’s automated scoring technologies. Automated scoring eliminates potential bias of human markers and provides more consistent grading. Speakers: Ernie Anastasio, Senior Advisor, Graduate Management Admission Council Helene Duvin, VP, University Relations, Pearson John H.A.L. De Jong, Director, Test Development, Pearson Language Assessments

23


Tuesday Conference Program Breakout Sessions 9:45 AM - 10:45 AM Cont.

Case Studies in Large Scale TechnologyAssisted Oral Language Assessment Location: Celebrity G Type: Education 

Developing and Validating Accessible Reading Assessments for Students with Disabilities Location: Celebrity F Type: Education 

Level: Beginner

Level: Beginner

English language learners (ELLs) represent one of the fastest growing segments of the K-12 student population in the United States. In the past, most assessments in education were designed for fluent English speakers. Thus, in order to help ELLs meet the academic achievement requirements of NCLB, their diverse needs must be addressed in educational reform efforts. One such way is to use settings, techniques and technology, familiar in both instruction and assessment, including English language proficiency exams, which often include an oral language component. This session outlines international case studies of the use of an innovative voice-based mobile phone application in the teaching and assessment of spoken language skills. With oral language fluency of the utmost importance to secondary language acquisition, this application allows students to use any phone to dial into a voice application and answer a series of voicebased questions. It also allows students to communicate oneto-one in real time (using voice or text), allowing them to use the target language in role plays based on real-life scenarios. The session also includes the opportunity to participate in a live hands-on demonstration of the assessment tools.

This session focuses on the development and validation of accessible reading assessments for students with disabilities. The development and validation of accessible reading assessments is the fourth and final goal of the National Accessible Reading Assessments Project (NARAP), a federally funded project that focuses on research to make largescale assessments of reading proficiency more accessible for students with disabilities. Goal four of this project was preceded by a series of steps that included a definition of accessible reading, researching the accessibility of reading assessments, and the development of principles and guidelines for increasing accessibility. The presenters describe their experiences developing and field testing prototypes of accessible reading assessments. They discuss the principles and procedures they used to develop the assessments and the design of the field trial studies that are used to provide empirical evidence of the validity of the prototypes.

Speakers: Gavin Cooney, CEO, Learnosity Jason Altman, Research Coordinator, National Center on Educational Outcomes

____________________________________

Cheating: Personality Predictors and Psychological Mediators Location: Oasis 1 & 2 Level: Intermediate

Type: Clinical 

Cheating remains a significant problem in myriad testing environments. Despite a wealth of studies, personality predictors of scholastic cheating remain elusive. However, several highly relevant variables have been overlooked. Furthermore, these studies neglect naturalistic, behavioral indicators of cheating in favor of self-report estimates or contrived laboratory studies. These voids were addressed in a series of studies. In Study 1, a large-scale survey isolated psychopathy, Machiavellianism, narcissism, low agreeableness and low conscientiousness as personality predictors of cheating, with psychopathy as the strongest predictor. Study 2 replicated this pattern using the innovative S-Check computer program as a measure of multiple-choice answercopying. Study 3 further supported these results using the Internet service, turnitin.com as an index of plagiarism. The psychopathy link held up after accounting for intelligence. Finally, Study 4 identified unmitigated achievement and moral inhibition, but not fear of punishment, as partial explanations of the psychopathy-cheating link. The implications of these studies for researchers and educators are discussed. Speaker: Kevin Williams, Senior Research Associate, Multi-Health Systems, Inc.

Speakers: Linda Cook, Principal Research Scientist, Educational Testing Service Martha Thurlow, Director, National Center on Educational Outcomes

____________________________________

Developments in Innovative Technology-Based Assessments Location: Oasis 3 Type: Industrial/Organizational  Level: Intermediate

Over the past decade, the concept of using Web-based technologies to deliver assessments has gone from cuttingedge to commonplace. Advances in technology have created new capabilities and spurred changes in the way tests are designed, delivered and used. The Internet has become the delivery medium of choice for a large number of assessment programs. In many instances, though, the assessments themselves are simply legacy versions of earlier paper-andpencil tests, and technology is simply used as a “page turner.� While this approach continues to bring significant benefits from a number of perspectives, it does not make full use of all the potential benefits of Internet technology. Over the past few years, however, some models of assessment have emerged that are starting to leverage these technological benefits. This session brings together a diverse panel of assessment experts who draw from real-world experience and research to examine new technology-driven assessment models, applications, challenges and future trends. This session also addresses the course of development of these models and how this may unfold over the next several years. Speakers: Reid Klion, Chief Science Officer, pan - A TALX Company Tim Kroecker, Partner, Cambria Consulting, Inc. Tracy Kantrowitz, Manager, Content Development, PreVisor

25


Tuesday Conference Program Breakout Sessions 9:45 AM - 10:45 AM Cont.

Integrating Cognitive Process and Complex Items Types in Exam Development Location: Celebrity H Level: Advanced

Prosecuting a Brain Dump Site: A Case Study Location: Mirage Level: Intermediate

Type: Cross Divisional

The primary validity concern of credentialing programs is that examination content reflects the knowledge, skills and abilities required for job performance. Testing programs must consider the cognitive processes required by job performance in several steps in exam development, including exam purpose definition, content definition, item writing, item review and standard setting, to provide such evidence. This presentation discusses different models of cognitive processes and how cognitive complexity was successfully incorporated in the exam development process for one credentialing program. Sometimes building cognitive complexity into a test program involves the addition of more complex item types. This presentation also addresses the specific issues that arise when including complex item types on a given exam to assess the range of cognitive abilities (understanding and analysis evaluation) and the challenge of combining scores from multiple item types (dichotomous and polytomous). Specifically, different scoring models are presented that allow for the creation of separate sub-scale scores for grouped items (e.g., scenario based, testlets, etc.). Speakers: Liberty Munson, Psychometrician Program Manager, Microsoft Corporation Russell Smith, Senior Psychometrician, Alpine Testing Solutions Sierra Hampton, Manager, Exam Development, Citrix Systems, Inc. Susan Davis, Psychometrician, Alpine Testing Solutions

____________________________________

Moving Beyond the Ropes Course: Using Assessments for Team Development Location: Oasis 4 Level: Beginner

Type: Industrial/Organizational 

Authors discussing the current state of organizational functioning emphasize the importance of teams in the obtainment of capital gain. In response, human resource practitioners are often called upon to create strategic programs designed to enhance team performance. Prior to engaging in team development activities, practitioners frequently use assessments to gather individual and grouplevel data. Although a profusion of literature addresses teamwork in organizational settings, there is a dearth of available resources that provide well-founded guidance for using assessments to create and implement meaningful teambased development programs. The purpose of this session is to: t $BQJUBMJ[FPOUIFFYQFSJFODFTPGQSBDUJUJPOFSTXIP  have successfully used assessments to enhance the team-development process t 8PSLUPXBSEUIFFTUBCMJTINFOUPGCFTUQSBDUJDFT

2009

Speakers: Jamie Bomer, Consultant, Hogan Assessment Systems Kevin Meyer, Consultant, Hogan Assessment Systems

26

Type: Certification/Licensure

Through a case study, the presenters of this session explore issues related to the targeting and civil prosecution of illegal brain dump sites. This session touches on the best practices for investigating targets, as well as utilizing pre-litigation techniques to obtain relief and effective strategies for litigating against online pirates. The presenters address not only actions that can be taken with respect to the brain dump site itself, but also available strategies for dealing with an offending website’s registrar, private registration company, host and/or payment processor(s). Test sponsors who attend this session come away with a range of investigative and legal strategies that may pertain to their own programs. Speakers: Brian Roche, Attorney, Roche Pia, LLC Gerald Pia, Attorney, Roche Pia, LLC

____________________________________

The Role of a Research Group in the Development and Extension of a Simulations Testing Program Location: Rancho Level: Intermediate

Type: Certification/Licensure

The AICPA Uniform CPA examination has used simulations of accountancy practice for the past four years. While the item types used for this purpose have served well, the lay committees developing content for the test continue to request new and innovative designs in order to assess the full range of skills identified in the practice analysis of the profession. In recognition of this need, the AICPA examinations team has assembled a research group whose role is to identify, design, develop and usability test new item types. The group consists of a test developer, three systems developers and a usability expert. The group has developed a sophisticated piece of software that allows for rapid prototyping and usability testing. This presentation describes the role of the individuals in this group, indicates how they interact with the operational testing staff, demonstrates the prototyping software and illustrates how a new item type goes in a principled way from concept to operational use in the test. Other organizations may find that this model for development and extension of a simulations testing program would work for them as well. Speakers: Peter Brittingham, Strategic Architect - Exams, AICPA Richard DeVore, Simulations Development Leader, AICPA


Tuesday Conference Program Breakout Sessions 9:45 AM - 10:45 AM Cont.

Questionmark Platinum Sponsor Session

To Catch a Thief: A Case Study on How GMAC Squashed a Brain Dump Site Location: Ambassador 5 Level: Intermediate

Type: Cross Divisional

Failure Is Not an Option: Ensuring Assessment Security Using Questionmark Perception Location: Ambassador 7

Nearly everyone has heard about brain dump sites and the impact that they have on testing by raising questions about the integrity of such tests. Testing organizations have long agonized over how to get these websites to stop harvesting questions from their tests. In this session, the presenters discuss a successful case that one organization had against a brain dump site. The Graduate Management Admission Council (GMAC) was awarded a $2.35 million judgment by the U.S. District Court for the Eastern District of Virginia in a copyright infringement case against the operator of Scoretop. com, a U.S.-based website that sold access to questions used on the GMAT exam. GMAC seized the site’s domain name, shut down the site and obtained a hard drive containing subscriber information. In this case study, members of GMAC discuss the process by which the company successfully took down an infringing website. The breakout session highlights the following :identification of the website, investigation of the website, compiling the evidence, the legal process, forensic analysis of the hard drive, consequences for individual test takers and lessons learned. Speakers: Daniel Eyob, Associate Director, GMAT Operations, GMAC Layne Pethick, Test Security Administrator, GMAC

____________________________________

Top 10 Costly Mistakes Certification Programs Make and How to Avoid Them Location: Oasis 5 Type: Certification/Licensure

Type: Cross Divisional 

Cheating on tests and breaches of security jeopardize the foundations of fair and equitable testing. This session will explore how Questionmark’s technologies and services proactively address security challenges at every stage of authoring, scheduling, delivering and reporting. t &YBNJOFUIFOFFEPGUFTUQVCMJTIFSTUPQSPWJEF role-based access to test content and administrative capabilities. t %JTDPVSBHFDIFBUJOHCZSBOEPNJ[JOHRVFTUJPOTBOE shuffling choices. t ,FFQQBSUJDJQBOUTGSPNTFFJOHDPOUFOUUIFZNBZOPU yet be ready for. t 7FSJGZUIFJEFOUJUJFTPGUFTUUBLFST QSFEFUFSNJOF dates, times, and numbers of attempts. t 4VQQPSUQSPDUPSJOHBOENPOJUPSJOHPGIJHITUBLFT tests. t -JNJUUIFMPDBUJPOTBOEPSNBDIJOFTGSPNXIJDI participants can access tests. t 1SFWFOUOFUXPSLQJSBDZBOEQSPIJCJUUBTLTXJUDIJOH content capturing, printing and other activities. This session will demonstrate how Questionmark’s technologies and services can address these and other security-related issues. Speakers: Greg Pope, Analytics & Psychometrics Manager, Questionmark Jeffery Place, US Sales Manager, Questionmark

Level: Beginner

____________________________________

There is no doubt about it—mistakes can be costly. Whether a new or an existing program, there is a need to not only watch out for the big “gotchas,� but there is also a need to know how to fix them. During this session, we discuss the most costly mistakes made in program design, test design, project management and more. Participants learn from peers and discover ways to avoid the pain experienced by others. Speakers: Cyndy Fitzgerald, Director, Psychometric Services, KRYTERION, Inc. Leslie Thomas, Senior Director, Client Services, KRYTERION, Inc.

Pearson VUE Platinum Sponsor Session Testing Trends and Tasty Treats Location: Ambassador 6

Type: Cross Divisional 

Back by popular demand, Pearson VUE’s interactive exploration into the latest trends in testing. Bring your curiosity and your questions as our experts address the hottest innovations within the CBT industry. Hear the results of our successful palm vein recognition pilot and learn about our Secure Testing Framework™. Discover what’s new in test development and see how unique item types are currently being used. If you don’t want to sit through another static presentation, this is the session for you. Our experienced team of industry leaders will guide you through individual stations that explore the new frontiers of global test security, as well as highlight some of the innovative tools currently available for test development and data management. And we’ll round out our menu of testing trends and informative tidbits with some tasty treats.

2009

Speaker: Donald Wagner, VP, Business Development, Pearson VUE

28


Tuesday Conference Program Balancing Validity and Practicality: Implications on Test Design

Morning Break 10:45 AM - 11:00 AM Location: Celebrity Foyer

____________________________________________________

Breakout Sessions 11:00 AM - 12:00 PM A Next Generation for Automated Scoring: Product Definition and Usability for K-12 Applications Location: Celebrity F Type: Education  Level: Intermediate

Automated scoring of essays and extended response items is a well-established technology, but it has a greater presence in academic and professional testing programs than in K-12 applications. This session explores the potential for expanding the use of automated scoring technology through consideration of product definitions and usability standards in balanced assessment systems for the K-12 market at the state and district levels. Tools of this type have existed for almost 10 years, yet they remain grossly underutilized. What are some ways we can use automated scoring tools can be used to move toward the balanced assessment systems that districts and states are asking for? How can such tools be integrated with other assessments to produce the goal of “balance”? Factors related to decisions based on assessment information in formative and summative contexts and how they are presented to user audiences is addressed in the presentation. The discussion also takes into consideration the challenges and benefits of automated scoring in terms of user needs for measurement services, customer support and interpretive guidance. Speakers: Catherine Welch, Professor, Iowa Testing Programs Richard Swartz, President, Riverside Publishing Company Stephen Dunbar, Director, Iowa Testing Programs

____________________________________

Assessing the Effects of an Item Error in a Computer Adaptive Test (CAT) Location: Celebrity G Level: Intermediate

Location: Mirage Level: Intermediate

Type: Certification/Licensure

The success of a licensure or certification program depends, in part, on how the validity and relevance of its testing program is perceived by the profession or industry that it represents. A common problem faced by certifying bodies and testing vendors is how to introduce new content. New content is critical to keep the exam current with the latest industry regulations and trends. However, many times certifying bodies and testing vendors need to implement new content quickly. In this session, the presenters explore methods for updating content on high-stakes certification exams, while maintaining validity and candidate fairness. Speaker: Susan Steinkamp, Manager, Psychometric Services, Pearson VUE

____________________________________

Business Strategy and Planning to Make the Decision to Go International Location: Ambassador 5 Type: International  Level: Intermediate

Taking a certification global presents risks. Translating, localizing and expanding the reach of your certification on a global scale can be a terrifying proposition for many associations. IMA has thought long and hard about the best ways to make their CMA certification a truly global certification – offered in different language and through different testing channels. Hear how one US based association began the process, developed strategies to mitigate risk, and successfully expanded the certification on an international scale and in different languages. Launching a certification globally however is only the beginning - ongoing maintenance, expansion, validity testing and promotion also need to be planned from the beginning and then completely rewritten as various challenges are met. Speaker: Christina Turner, Managing Director, Global Professional Skills, Inc.

____________________________________

Type: Education 

When an error is found with an operational item in a computer adaptive test (CAT), assessing its effect is especially challenging, because items selected after the erroneous item are closely dependent on the examinee’s performance on the erroneous item, as stipulated by CAT. Using both true and simulated examinee responses on a real international CAT assessment, this study investigates the effects of an erroneous operational CAT item on (a) score accuracy, (b) degrees of difference in item selections and (c) the potential to recover an examinee’s true ability from the error, while controlling for the relative positions of the erroneous item in a CAT assessment. In addition to presenting the research findings, this session provides an overview of current theory, practice and methodology. Speakers: Sung Lee, University of Oklahoma Xiang Bo Wang, VP, ACT, Inc.

Guess What? Score Differences with Rapid Replies Versus Omissions on a Computerized Adaptive Test Location: Celebrity H Level: Beginner

Type: Cross Divisional

One of the strategies for completing a standardized test is to guess, whether the test taker is unsure of the answer or running out of time. If the penalty for incorrect responses is the same across all test items, then the impact of guessing can be easily measured. However, in the case of a computerized adaptive test, the effect of guessing on any single item would depend on the individual’s ability estimate and the item’s parameters. Rapid guessing as time expires may have extreme effects depending on current estimates or may result in only slight changes. The current study uses data from an operational computer adaptive test to look at 29


Tuesday Conference Program Breakout Sessions 11:00 AM - 12:00 PM Cont.

differences among scaled scores in three cases: (1) if the score is calculated before guessing; (2) if the score is calculated based on completing the last few items by guessing; and (3) if the score is calculated before guessing but adjusted with a penalty for omissions. Using this data, the relative benefits and possible penalties of guessing or leaving questions blank is explored for different ability levels. Speaker: Eileen Talento-Miller, Psychometrician, Graduate Management Admission Council Fanmin Guo, Director, Psychometrician Research, Graduate Management Admission Council

____________________________________

Guidelines for Using Statistics in Legal Defense of Test Program Integrity Location: Oasis 1 & 2 Level: Advanced

Type: Certification/Licensure

Today’s testing environment is becoming more and more diverse in an effort to accommodate test takers and to improve security. Depending on the market, surprisingly high numbers of test takers are attempting to improve their scores using unscrupulous techniques. The age-old practice of live proctors is somewhat useful in encouraging honesty, but is not adequate in and of itself. Recent developments in data forensics allow test fraud to be detected statistically with a very high degree of accuracy. This session explores the use of statistics in legal arguments to defend the integrity of testing instruments. Speakers: Alan Thiemann, Legislative Counsel, Law Office of Alan J. Thiemann Jennifer Ancona Semko, Partner, Baker & McKenzie, LLP Robert Burgoyne, Partner, Fulbright & Jaworski, LLP Dennis Maynes, Chief Scientist, Caveon Test Security

____________________________________

Justifying the Use of Employee Selection Procedures: Issues, Challenges and Strategies Location: Oasis 3 Level: Intermediate

Type: Industrial/Organizational 

As enforcement activity among federal regulatory agencies increases, organizations are under increasing pressure to justify their use of assessment procedures in making selection decisions, particularly when they result in adverse impact against a protected demographic group. While professional standards and principles delineate good practice, justification is often not straightforward because legal requirements (guidelines and case law) may not be consistent in and of themselves, or with professional practice. This session reviews current legal issues, enforcement trends and cases related to uses of selection instruments (e.g., cut scores, scoring and decision strategies, and alternative content and methods), along with corresponding measurement issues, research and considerations for practitioners who work with organizations to design and implement staffing systems.

2009

Speakers: John Weiner, Chief Science Officer, PSI Keith Pyburn, Fisher & Phillips, LLP

30

Technology, Assessments and Globalization: TAG You’re It Location: Oasis 4 Level: Beginner

Type: Industrial/Organizational 

A core issue in the world of work facing industrialized societies is technological development. Tracing the diffusion of technology throughout the world is an important aide to understanding the application of assessments internationally. Technology clearly influences the implementation of certain assessments—like the relationship between item response theory (IRT) and computer-based development and administration. In locations with limited resources and access to technology, practitioners may be forced to focus primarily on the practicality and utility of selection methods, rather than their validity and reliability. This session includes a discussion of assessment-related technological advances, and the impact technology has on the international implementation of assessments. Speakers: Jamie Bomer, Consultant, Hogan Assessment Systems Kevin Meyer, Consultant, Hogan Assessment Systems

____________________________________

Test Security: Taking Action and Seeing Results Location: Oasis 6 & 7 Type: Certification/Licensure Level: Intermediate

In this session, representatives of major certification programs, which are giving a high priority to protecting their exams and combating cheating, report on a number of issues, including: t

8IZUIFZDPODMVEFUIBUEFWPUJOHNPSFUJNFBOE resources to these issues is essential?

t

8IBUTUSBUFHJFTUIFZFNQMPZ

t

)PXSFTQPOTJCJMJUZJTBMMPDBUFE

t

)PXUIFZNPOJUPSUIFFòFDUJWFOFTTPGUIFJS actions?

t

8IBUBQQSPBDIFTTFFNUPIBWFUIFCFTUSFUVSOT

t

8IBUUIFZIBWFUSJFEUIBUIBTOPUCFFOBT successful?

t

8IFSFUIFJSHSFBUFTUWVMOFSBCJMJUJFTBSF

t

)PXUIFZXPVMEBEWJTFPUIFSTUPQSPDFFEXIP  want to learn from the panel’s experiences?





The types of individuals and programs that are participating in the session provide a thorough overview of key steps being taken to protect items and tests and to support the value of the test-based decisions that are being made. Speakers: Alejandra Garcia, Exam Developer, Citrix Systems Bob Whorton, Manager, Compliance and Security, NCEES Gary Clark, VP, Sales & Business Development, Caveon Test Security Liz Burns, Senior Program Manager, EMC Jill Burroughs, Senior Management, Exam Development and Security, CompTIA


Tuesday Conference Program Breakout Sessions 11:00 AM - 12:00 PM Cont.

Testing in a European Context: Latest Developments Location: Polo Room Level: Intermediate

The Importance of Fidelity in Professional Certification Testing

Type: International 

Location: Rancho

Type: Certification/Licensure

Level: Beginner

This session provides insights into the latest developments in the field of testing in Europe, an economic zone spanning 25 member countries and a population of half a billion. Europe presents a dynamic environment for testing with geographical, cultural, linguistic as well as educational, legal and best practice challenges. Meeting these challenges also provides an incubator for innovations in testing theory and practice. Europe’s measurement scientists contribute significantly to the advancement of the field of measurement science and its related testing technologies, as well as a range of testing issues, such as Internet testing, intellectual property protection, privacy, best practices for the cultural adaptation of tests, interoperability and innovative testing methodologies. Speaking from their respective fields (from employment to education to skills and qualifications testing), the panel shares their latest experiences in one of the world’s most diverse contexts for the development and deployment of testing programs. This is a sequel to the well received 2008 panel presentation on the same topic. Speakers: Cor Sluijter, Director, Vocational, Adult & Higher Education, Cito Eric Shepherd, CEO, Questionmark Eugene Burke, Science & Innovation Director, SHL Group Limited

____________________________________

While advancements in technology enable simulation of life-like environments, certifying bodies need to evaluate the benefits of increased fidelity in their assessments. We compare the design and development processes of a high and low fidelity simulation and explore practical and measurement pros and cons. Interventional Cardiology involves complex medical procedures and life-threatening complications. Full-immersion simulation should enhance the validity of certification by providing an evaluation of physicians’ decision making skills under the stress of dealing with multiple events and high sensory load. Dr. Rebecca Lipner will present the process undertaken by the American Board of Internal Medicine to develop the simulation assessment. Engineering technologists perform a variety of quality assurance tests and critical inspections to ensure our nation’s infrastructure. Low fidelity computer simulation of the field and laboratory environments should allow more thorough and uniform evaluation without the cost and risks of actual hands-on testing. Buck Chaffee and John Baker will discuss challenges with computer simulation exams for the National Institute for Certification in Engineering Technology. Speakers: Clarence Chaffee, President, The Caviart Group David Cox, President, Professional Testing, Inc. John Baker, CEO, Zoomorphix Systems Rebecca Lipner, VP, Psychometrics and Research Analysis, ABIM

____________________________________

Testing Program Management: Best Practices to Help You Successfully Manage Change and Growth Location: Oasis 5

Certification Management Services Platinum Sponsor Session

Type: Certification/Licensure

Level: Intermediate

Leveraging Technology for Distributed High-Stakes Test Development

This session will be led by representatives from three premier program sponsors and a global testing vendor who have successfully guided large-scale testing programs through a variety of challenges such as rapid growth, vendor transition, and new test construction. In this session, we will discuss the management of transitions, candidate communication and expectations, vendors, and data (what do you own, what do you get?)—all elements of a growing program. The session will encourage audience participation throughout the presentation to address current issues attendees may be facing.

Location: Ambassador 6

Speakers: Bill West, Director, Business Development, Pearson VUE Charles Snead, Information Systems Manager, Excelsior College Dania Eter, Associate VP, Research and Test Development, National Commission on Certification of Physician Assistants Scott Arbet, VP, Research & Test Development, National Commission on Certification of Physician Assistants

Type: Cross Divisional

By leveraging technology, test sponsors can securely develop high quality exams in significantly less time and at a fraction of the cost than most traditional test development methods. This session explores how to overcome security concerns of utilizing distributed item writing workshops and webfacilitated meetings for developing high-stakes exams. Processes most impacted by effective use of technology include: blueprinting, item writing training and item development, technical and peer reviews, statistical analyses, forms builds and standard setting. Not only can you reduce the travel cost and time required to bring content experts together for in-person meetings, but the actual time-on-task can also be greatly reduced. Technology can also significantly reduce the administrative time and cost required to track and manage the development process. The overall test quality is improved by involving more high-level experts, many of which may not be available to travel to in-person workshops.

2009

Speakers: Kim Thayn, Partner & Psychometrician, Certification Management Services Scott Thayn, President & Psychometrician, Certification Management Services

32


Tuesday Conference Program KRYTERION, Inc. Platinum Sponsor Session Online Secured Testing: Extending the Reach of High Stakes Testing Location: Ambassador 7

Type: Cross Divisional

Online testing was once considered not secure. With the advent of new item types and test delivery technologies you no longer have to sacrifice security for the convenience of delivering tests where you test takers live, learn and work. In this session you will discover how you can extend the reach of your current testing channel and add layers of security to your online tests.

learning), Learning outcome assessment t 5SBOTGFSPGMFBSOJOH Certifying agencies can use the model presented in this program to evaluate their existing programs and to gain insights into areas that need improvement. Speakers: Roy Swift, Program Director, American National Standards Institute Vijay Krishna, Program Manager, American National Standards Institute

____________________________________

Speaker: David Foster, CEO, KRYTERION, Inc.

Global Assessment Challenges: Case Studies and Lessons Learned

____________________________________________________

Location: Polo Room

Type: International 

Level: Intermediate

Lunch with Exhibitors Sponsored by: 12:00 PM - 1:30 PM Location: Celebrity Ballroom

____________________________________________________

Exhibits Remain Open 1:30 PM - 2:45 PM Location: Celebrity Ballroom

____________________________________________________

Breakout Sessions 1:30 PM - 2:30 PM A Framework for the Development of a Quality Certificate Program Using an Integrated Learning and Development Approach Location: Celebrity G

Type: Education 

Level: Intermediate

ASTM International is working towards the creation of a new standard for certificate programs. A certification program is based on a testing paradigm, while a certificate program is based on a learning and development paradigm, which requires alternative approaches to measure the quality of learning experienced. The ultimate goal of a certificate program should be a positive transfer of learning to the workplace, however, research has shown that only 10 percent of learning actually transfers to job performance (Lim & Morris, 2006). Drawing from learning transfer theories and adult and workplace learning theories, the paper by Lim and Morris develops a model that outlines critical success factors (CSFs) of a quality certificate program. The key elements of this model are: t -FBSOJOHOFFETBTTFTTNFOU t .FBTVSBCMFMFBSOJOHPCKFDUJWFT t &òFDUJWFMFBSOJOHQSPDFTT FYQFSJFOUJBMBOECMFOEFE

As organizations continue to span geographical boundaries, business approaches need to be adapted to local cultures and customs. While this can be a challenging situation for any business attempting to coordinate across borders, time zones, languages and cultures, the situation becomes increasingly more complex when dealing with psychological assessment products that are especially sensitive to cultural and linguistic differences. The purpose of this session is to review common issues encountered by test publishers and other practitioners when implementing global assessment programs. The session explores issues that are common to test publishers and practitioners using assessments in employee selection, leadership development and talent management contexts. Common issues to be discussed include (but are not necessarily limited to) translation strategies, development of assessment norms, sources of error in cultural adaptation, validation strategies and cultural biases. The presenters supplement this discussion with examples from their own experiences in global assessment contexts. Speakers: Jamie Bomer, Consultant, Hogan Assessment Systems Kevin Meyer, Consultant, Hogan Assessment Systems

____________________________________

Innovative Forensics for Exam Security Location: Celebrity H

Type: Cross Divisional

Level: Intermediate

No matter how clever cheaters are, they leave invisible clues or patterns that can be recovered through the use of forensic tools and methods. This session discusses a number of innovative forensic methods and how they have been used defensibly to protect testing programs from cheaters, proxy testers and exam-content thieves. Examples include aberrant response analysis, program forensics, real-time interventions, digital watermarking and Trojans. Security experts present a number of case studies based on both statistical and nonstatistical forensic approaches. Speakers: Dennis Maynes, Chief Scientist, Caveon Test Security Mark Poole, Director, Test Security, Pearson VUE Wayne Patience, Assistant VP, ACT, Inc.

33


Tuesday Conference Program Breakout Sessions 1:30 PM - 2:30 PM Cont.

Management of Assessment Results: Practical, Ethical and Legal Considerations Location: Oasis 4

Type: Industrial/Organizational 

Online Testing: The Green Scene Location: Oasis 6 & 7

Type: Certification/Licensure

Level: Beginner

Level: Intermediate

In contrast with the rather clear ethical and legal guidelines that exist for the management of clinical assessment data, significant attention has not been paid to this issue within the context of either pre-employment selection or corporate leadership development testing programs. This session explores some of the related practical, legal and ethical issues. First, attention is paid to existing practice standards, such as the SIOP Principles, APA Ethical Standards, Joint Standards and the International Test Commission guidelines. Some of the broader underlying legal issues and societal concerns related to both privacy and the desire for process transparency are also discussed. Second, there is a discussion of test-taker perceptions of confidentiality and the effect that a clear versus ambiguous statement of privacy may have upon a willingness to engage in assessment and do so in a reliable manner. Finally, the practical implications for the management of assessment data is discussed in terms of post-hire reuse for employee development and promotion, specific issues related to civil service and other public sector hiring situations, and practical guidelines for practice and client education. Speakers: Julie Carswell, Senior Consultant, SIGMA Assessment Systems, Inc. Reid Klion, Chief Science Officer, pan - A TALX Company Ryan Ross, Director, Alliance & Partners, Hogan Assessment Systems

____________________________________

New Testing Paradigms in Technology-Based Assessment Location: Mirage

Type: Industrial/Organizational 

Level: Intermediate

Advances in technology have created new capabilities and spurred changes in the way tests are designed, delivered and used. Many of these changes are a natural outgrowth of the continuous pressure for organizations to be increasingly efficient and effective, as well as addressing changes like the daily presence and use of the Internet. This session brings together a diverse panel of assessment experts who draw from real-world experience and research to examine new technology-driven assessment models, applications, challenges and future trends. The topics addressed include: talent assessment models and practices enabled by technology, and issues resolved and raised; new applications in technology-based assessment, highlighting simulation testing and adaptive testing in an unproctored online environment; technology-assisted test construction, delivery and validation; and the next generation of technology-based assessment—the “real” use of the Internet in testing

Paper-based and brick-and-mortar testing have been traditionally associated with assessment. But how do these historical approaches impact the greater environment? Are there ways to achieve similar or even better results with green friendly technologies? The answer is “Yes”! In this session, discover how to incorporate the latest in online testing technologies without sacrificing program control or security, and ways to make assessments green. Presenters will lead the audience in a discussion that identifies the areas of testing that leave the largest footprint, such as climate control, travel and computers. Participants share their experiences in greening their practice. Participants will learn how to set up a green infrastructure, focusing on procurement and recycling and will learn how to interpret common green concepts such as life-cycle analysis, ecological footprint, regenerative design and renewable energy. The speakers will show how personnel certifications can promote sustainability by incorporating sustainable KSAs into pre-existing assessment programs. Speakers: Brian Bontempo, Principal Consultant, Mountain Measurement, Inc. Michael Sammet Quinn Sutton, VP, Marketing, KRYTERION, Inc.

____________________________________

Practical Applications of Response-Time Modeling Location: Celebrity F Type: Education  Level: Intermediate

Test theorists have always been intrigued by the relationship between responses to test items and the time used by the test taker to produce them. Both seem indicative of the same behavior on test items. Nevertheless, their relationship appears to be difficult to conceptualize, let alone represent coherently in a statistical model. In this presentation, the attendees are guided on a tour through the theoretical landscape of response-time modeling to clarify several of the issues that have always confused test theorists, including the definitions of speed and power tests, the role of the speedaccuracy in testing, the distinction between response time and speed, and the question of why correct responses take a different time than incorrect responses. The results of several empirical studies are presented to illustrate the huge potential benefits of using response times as an additional source of information in item analysis, test design, estimation and control of test speediness; detection of cheating; and item selection in adaptive testing. Speaker: Wim J. van der Linden, Chief Research Scientist, CTB/McGraw-Hill

Speakers: David Foster, CEO, KRYTERION, Inc. Doug Reynolds, VP, Assessment Technology, Development Dimensions International (DDI) Eugene Burke, Science & Innovation Director, SHL Group Limited John Weiner, Chief Science Officer, PSI

35


Tuesday Conference Program Breakout Sessions 1:30 PM - 2:30 PM Cont.

Prevention, Detection and Management of Falsified Test Development Data in the Clinical Test Publishing Industry: An Interactive Forum

The 2009 Revision of the ATP’s Model Guidelines for Pre-Employment Integrity Testing Location: Oasis 3 Type: Industrial/Organizational 

Location: Oasis 1 & 2

Level: Intermediate

Type: Clinical 

Level: Intermediate

Attendees receive the draft copy of the ATP’s revised Model Guidelines for Pre-employment Integrity Testing (Model Guidelines) that were first introduced in their original form in 1991 and in an updated form in 1996. The following areas requiring revision are reviewed during this session: integrity testing research update, alignment of Model Guidelines with new testing standards, new classes of integrity tests, international adaptation of integrity tests, legal compliance update and technology innovations related to integrity testing.

In the test publishing industry, the authenticity of the data collected during test development is paramount. Test publishers generally rely on external sites to collect data, but there are no firm guidelines for verifying data integrity. The presenters propose several steps for discovering/managing falsified data, and welcome feedback and discussion based on other publishers’ experiences in this interactive forum. Fraudulent data may be discarded easily, but if left undetected, its negative impact can be substantial. It is the presenters’ hope that discussion of this topic helps propel implementation of industry-wide standards for contending with fraudulent data. The presenters encourage information sharing and audience participation throughout the presentation.

Speakers: David Arnold, General Counsel, Wonderlic, Inc. John Jones, VP & Chief Scientist, Vangent Inc.

____________________________________

Value of Virtual Machine Technologies and Tools for Improving Testing/Assessment

Speakers: David Herzberg, Senior Project Director, Western Psychological Services Gill Sitarenios, Chief Scientist, Multi-Health Systems, Inc. Hazel Wheldon, Publisher, Multi-Health Systems, Inc. Kevin Williams, Senior Research Associate, Multi-Health Systems, Inc.

Location: Rancho Level: Intermediate

____________________________________

Security Trends in Large Scale Testing in Education Location: Oasis 5 Type: Education  Level: Beginner

Organizations involved in large scale testing in Education have initiated many activities to help protect their examinations and to combat efforts to cheat. Representatives of major organizations in the US and the UK will review the strategies that they have applied and share what they have learned. Issues such as the following will be addressed: t

8IBUTUSBUFHJFTBSFCFJOHFNQMPZFE

t

8IPIBTSFTQPOTJCJMJUZGPSUIFTFTUSBUFHJFTXJUIJO the testing entities?

t

8IBUTFFNTUPCFUIFNPTUFòFDUJWFQBDLBHFPG activities?

t

8IBUBSFUIFHSFBUFTUDIBMMFOHFT

t

8IBUJTEJòFSFOUBCPVUQSPUFDUJOHBHBJOTU

Virtualization software comes in many sizes and shapes and has many uses in todays increasingly more compact and digital computer/software/communications world. Virtualization is widely employed in the computer and networking industries for clustering, storage networks, server consolidation, application virtualization and other uses. This session focuses on the value of virtualization and virtual machine software and tools for improving testing and assessment. The speakers exhibit a wide array of expertise and interest in the current and future use of virtualization technologies and tools for improving performance testing, remote application management, virtual and live application testing, and test and score management. This session shows several practical applications of virtualization technologies and tools for testing and assessment. The speakers also highlight potential research directions for these technologies and tools, which may offer a high payoff in terms of improving the types, richness, modularity and scalability of future tests and assessments.

cheating now compared to ten years ago?

Speakers: Dean Hardy, Business Development, Summit Consulting Group Matthew Strongin, Manager, Courseware Development for Citrix Education, Citrix Systems, Inc. Robert Cromartie, VMware, Inc Steven Moore, Content Publishing Manager, Microsoft

t

8IBUIBWFUIFQBSUJDJQBOUTMFBSOFEBCPVUEFBMJOH

____________________________________

t

8IBUBEWJDFDBOCFTIBSFEXJUIQFPQMFKVTU starting out to tackle test security in an educational setting?

with the media in security cases?

West Meets East - Executive Coaching in China Location: Ambassador 5 Type: International  Level: Intermediate

Speakers: Ardeshir Geranpayeh, Senior Research & Validation Coordinator, University of Cambridge ESOL Examinations Don Sorensen, VP, Marketing, Caveon Test Security Peg Jobst, VP, GMAT Program, GMAC

2009

Type: Certification/Licensure

In response to delegate feedback, we are presenting an VQEBUFEWFSTJPOPGUIFA8FTU.FFUT&BTU#SFBLPVU4FTTJPOUIBU was presented at ATP 2008. The use of psychometric assessments in the United States has a long and distinguished history, but in China the science

36


Tuesday Conference Program Breakout Sessions 1:30 PM - 2:30 PM Cont.

is a new one. As China emerges from a centrally controlled state economy to a more flexible and market-driven business environment, the need to identify talent and support joint ventures has become more urgent. Everyone knows that Chinese culture is different, but do they know how different and why? More importantly, do they know how Western behavioral norms impact Chinese managers? In this session, the results from a number of assessment projects in China are presented and compared to results from Western managers. Key differences are highlighted between Chinese managers working in Chinese companies, Chinese managers working in joint ventures and Chinese managers working outside China. All are compared to Western norms, and the presenters describe how this analysis supports a successful executive coaching program. Speaker: Chris Sale, Director, EDAC

____________________________________

Defining and Measuring College Success Type: Cross Divisional

Admissions and Selection Testing: New Techniques for Facing Test Security Challenges and Protecting Program Integrity Location: Celebrity G Type Education  Level: Intermediate

Admissions and selection testing professionals are faced with new challenges to keep standards high. Cheating threatens the testing system by calling into question the validity of test scores and the validity of the test. At this session, top testing experts responsible for admissions and selection testing present their new techniques to reduce test fraud. Candidate fraud issues and prevention techniques are also discussed.

____________________________________

Many states have developed separate standards for college readiness and are increasingly concerned about preparing students for college success. This session will examine various definitions of college readiness and how it can be measured in a multi-method approach. Test scores are predictors of success– they are necessary, but not sufficient in determining is students are prepared. Both speakers will discuss the multiple facets of college readiness: assessments, previous grades in school, academic rigor, financial and family support, as well as other cognitive and non-cognitive skills. Speakers: Andrew Wiley, Senior Director, The College Board Krista Mattern, Assistant Research Scientist, College Board Wayne Camara, VP, Research & Psychometrics, The College Board

____________________________________

Internet Testing Systems Platinum Sponsor Session Innovation in Online Testing Location: Ambassador 6

2:45 PM - 3:45 PM

Speakers: Deborah Harris, Director, Measurement Research, ACT, Inc. Kathy Hinshaw, Senior Manager, Test Administration, American Dental Association Peg Jobst, VP, GMAT Program, GMAC

The College Board Platinum Sponsor Session

Location: Ambassador 7

Breakout Sessions

Type: Cross Divisional

Find out the latest features and functionality provided by Internet Testing Systems, a leader in online testing. This session will provide technology demonstrations of item types, performance based testing, diagnostic reporting, offline test delivery, channel management, and more.

An Investigation of the Idiosyncratic Computerized Adaptive Testing (ICAT) Procedure Location: Celebrity F Level: Intermediate

Type: Education 

Idiosyncratic computerized adaptive testing (ICAT) provides teachers with information to target instruction for students with unique knowledge patterns. For instance, if a teacher knows that a particular student tends to be stronger in geometry and weaker in algebra, she can frame mathematics geometrically to take advantage of the student’s strengths. This allows students with unusual patterns of knowledge to flourish. At the same time, the technique can be used to give education providers information about how well their instructional programs prepare individuals for certification or licensure. This study investigates the ICAT procedure designed to tease out individual patterns of performance within subgoal areas, in order to provide feedback for use in remediation and instruction. The study investigates the performance of ICAT using data that models real student data applied to a fixed-form test, a traditional adaptive test and an ICAT test. Results indicate how accurately each test type identifies idiosyncratic patterns. Additional analyses investigate whether any information loss occurs using ICAT. Speaker: Gage Kingsbury, Senior Research Fellow, NWEA

____________________________________ Speaker: Patrick Ward, President, Internet Testing Systems

____________________________________________________

Case Closed: Stories from Three of the Greatest Test Detectives

Afternoon Break

Location: Celebrity H Level: Intermediate

Type: Cross Divisional

2:30 PM - 2:45 PM Test security is an ongoing concern. For example, how can testing program managers connect the clues around test security issues to get the cases of test theft resolved? Come

2009

Location: Celebrity Foyer

38


Tuesday Conference Program hear from three of the greatest test detectives of our time; Sherlock Holmes, Nancy Drew, and Ms. Marple. These test security experts discuss their toughest cases; their cunning strategies, their clinchers in catching the culprits and their swift action in bringing these miscreants to justice. Cases are sighted from certification, licensure and educational settings. Speakers: Ardeshir Geranpayeh, Assistant Director, Research and Validation, University of Cambridge ESOL Examinations Liz Burns, Senior Program Manager, EMC

____________________________________

Computerized Adaptive Testing for the Singapore Employability Skills System (ESS) Location: Polo Room Type: International  Level: Beginner

This session presents innovations in computer adaptive testing of adult workplace literacy and numeracy skills developed by CASAS and customized for the Singapore Employability Skills System (ESS). The Singapore Workforce Development Agency plays a key role in the implementation of ESS “to enhance the employability and competitiveness of employees and job seekers, building a work force to meet changing needs of Singapore’s economy.” CASAS has developed CAT tests for math, reading, listening and computer-delivered tests for writing and speaking. The CAT tests are administered in secure proctored locations using LANs. The CAT tests use the following psychometric procedures: selection of initial item from a random proficiency value near the center of the proficiency distribution of the item bank; Rasch model calibration and proficiency estimation; and stopping rules based on a minimum standard error or administration of a specified maximum number of items. The session presents an overview of the project, demonstrations of sample items and the test delivery and administration system, review of test score results and psychometric analyses, and plans for future enhancements and extensions. Speakers: James Olsen, VP & Chief Scientist, Alpine Testing Solutions Linda Taylor, Director, Assessment Development, CASAS Patricia Rickard, President, CASAS Richard Ackermann, President, Team Code, Inc.

____________________________________

Defending Accountability Testing: Countering Anti-Testing Arguments and Educating the Public about the Benefits of Testing Location: Oasis 5 Type: Education  Level: Intermediate

The No Child Left Behind (NCLB) Act of 2001 will be reauthorized in either 2009 or 2010. With this reauthorization, both critics and friends of accountability testing will propose changes to the law. All the changes will be promoted as ways to strengthen the teaching and learning process leading to higher overall achievement, a narrowing of the achievement gap, and promoting new knowledge and skills for the 21st Century. Our panel will review and comment on the various proposed changes to NCLB that have come to light recently, and comment on whether they will strengthen or weaken educational accountability, and the likelihood that the change will be implemented. Participants should be well informed

about our national policy on K-12 achievement testing and accountability and be the first line of defense against attempts to weaken educational accountability. ATP members will take away from the discussion arguments that can be used to defend testing; including why a well-constructed assessment is the most fair and efficient way of ensuring accurate measurement and public accountability. Come refresh your knowledge in this area by listening to a lively discussion and interacting with our distinguished panel. Speakers: Chuck Melley, Executive Director, Government and External Relations, ETS Doug Kubach, President & CEO, Pearson’s Assessment & Information Group Larry Snowhite, VP, Public and Government Affairs, McGraw Hill Education Phil Young, Senior Director, Large Scale Assessment, ETS William G. Harris, Executive Director, Association of Test Publishers

____________________________________

Defense by Association, Part II: Exploring Strategies for Collaborative Action to Combat Test Piracy Location: Ambassador 5 Level: Intermediate

Type: Cross Divisional

The theft and piracy of intellectual property (IP) costs the US economy $250 million per year. In the United States and elsewhere, companies that produce IP employ multiple strategies to safeguard these valuable assets, including membership in industry associations such as the Association of American Publishers (AAP) and the Software & Information Industry Association (SIIA). These organizations support their memberships with educational programs, legislative advocacy and enforcement. Members of these trade groups recognize the necessity of collective action to address the accelerating threats of theft and piracy in black markets that have been enabled by the instant transfer of data. Beyond common financial interests, Internet trafficking of tests is an active and growing threat to the fairness and validity of testing. As a follow up to last year’s “Defense by Association,” this presentation draws examples from other industries to examine the methods, costs and impact of association-led IP protection efforts. Topics include the mechanics of association-led legal action, as well as the legal and fee structures needed to support such activities Speakers: Alan Thiemann, Legislative Counsel, Law Office of Alan J. Thiemann Jennifer Ancona Semko, Partner, Baker & McKenzie LLP

____________________________________

Improving Testing Security with Global Identity Management Systems (GIMS) Location: Oasis 6 & 7 Type: Certification/Licensure Level: Beginner

The ability to uniquely identify a testing candidate within a testing program or population offers many benefits, such as the ability to deter or eliminate testing fraud by systematically disallowing candidates from having multiple testing accounts. In addition, proxy and professional test takers can be identified and banned from accessing more test content. A global identity management system (GIMS) can be implemented with the use of advanced non-invasive 39


Tuesday Conference Program Breakout Sessions 2:45 PM - 3:45 PM Cont.

biometric technologies. For example, palm vein biometric technology offers the ability to uniquely identify a person without the stigmas that are attributed to other biometric technologies. There are many obstacles and challenges that have to be overcome to successfully implement GIMS. These include selection of the right biometric modality for test administration applications, operational usage and candidate privacy.

Search and Destroy! Web Searching Strategies and Techniques that Work Location: Rancho Type: Certification/Licensure Level: Intermediate

Speakers: Daniel Eyob, Associate Director, GMAT Operations, GMAC Michael Thieme, Director, Special Projects, International Biometric Group Ruben Garcia, Product Manager, Pearson VUE

____________________________________

Privacy 101 for Testing Programs Location: Mirage Level: Beginner

Type: Certification/Licensure

Testing programs collect and process significant volumes of personally identifiable information (PII). While technology innovation and the Internet provide the tools to collect and utilize this data, the spotlight is on organizations to implement these tools in a manner that sustains and does not erode individuals’ privacy rights. This interactive session will explore ways to balance program needs with social responsibility and rigorous privacy protections, with an emphasis on both domestic and international regulatory regulations and guidelines. Best practices will be addressed for developing, publishing, and maintaining clear and comprehensive privacy policies, and for managing compliance with these policies.

____________________________________

Recycling Applicants: Using Multi-Job Fit Assessment Profiles to Leverage Your Applicant Pool Type: Industrial/Organizational 

In many large scale recruitment processes, assessments are used early in the sourcing process—typically after the initial education/experience review but before extensive interviews—to identify candidates who meet the minimum employability standards. While a personality assessment may be used to screen at this basic level of employability, this in of itself does not provide sufficient information to determine if the applicant is a good fit for a specific job. A job-specific profile is often developed to determine if candidates who meet the basic employability profile fit a particular job. For positions with many applicants, such as customer service jobs, there are likely to be candidates who meet the employability requirements but not the specific job profile. For those candidates, the option of a multi-job assessment solution avoids classifying them as a “poor fit” and permits recruiters to approach these applicants about jobs in the firm where the multi-job profile indicated they are a good fit. If the candidate is interested in applying for the other job(s), their application, assessment results, etc., would be applied toward that position.

2009

Speakers: Benjamin Mannes, Director, Test Security, American Board of Internal Medicine Dwan Bovell, Manager, Computer Based Testing, American Board of Internal Medicine Joe Cannata, Certification Manager, Brocade Joe McClintock, Manager, American Board of Anesthesiology

____________________________________

Taking Another Look at Norms, Their Construction and Their Value in Enabling Valid Decisions Location: Oasis 3 Level: Intermediate

Speakers: Cordell Jung, Senior Counsel, Pearson VUE Jay Cline, President, Minnesota Privacy Consultants Mark Poole, Director, Test Security, Pearson VUE

Location: Oasis 4 Level: Intermediate

The information super highway is rapidly becoming a speedway for sharing and selling stolen test intellectual property. No sooner are exams published, then they are available at lightning speed on Web pages, at auction houses and in chat room forums. Testing programs are scrambling to stop this unlawful activity from damaging their test content and their program’s image. Fortunately, some testing programs have come up with effective strategies that help reduce item exposure and remove test content from these illicit venues. In this session, three distinguished testing programs are discussed along with what’s working, what’s not and how to get test content removed from the Internet.

Speakers: Craig Haas, Consultant, Hogan Assessment Systems Jamie Bomer, Consultant, Hogan Assessment Systems 40

Type: Industrial/Organizational 

This session shares recent thinking and methods in the development and use of norms in educational and employment testing. While norms have become a convention in test development and testing practice, new developments in IRT and meta-regression allow us to test models in terms of the appropriateness of a norm for a population or a range of populations of interest, as well as to be able to produce customized norms on demand from the psychometric properties of the instrument and the demographics of the candidates. Furthermore, with the advent of global testing programs, new questions in how a norm is configured and applied to cross-national and multi-language populations have come to light. The symposium develops these themes by first considering models of factors influencing test scores through the application of meta-regression models that account for demographic and firmographic variables, crossborder testing programs, and finally how norms can be used to provide talent benchmarks against which clients can interpret scores and also measure the effectiveness of their talent acquisition processes. Speakers: Eugene Burke, Science & Innovation Director, SHL Group Limited Ronald K. Hambleton, Distinguished University Professor, University of Massachusetts


Tuesday Conference Program Breakout Sessions 2:45 PM - 3:45 PM Cont.

ACT, Inc. Platinum Sponsor Session

ATP Divisional Meetings

America’s Jobs: How ACT’s WorkKeys® System and JobPro Database Supports Workforce Development Initiatives in High-Demand Occupations

All are Welcome to Attend!

Location: Ambassador 7

3:45 PM - 5:00 PM Certification/Licensure Division  Location: Oasis 4

Type: Cross Divisional 

Speaker: Shelby Grieve

ACT’s WorkKeys® assessment program puts jobs’ foundation skill requirements and individuals’ skill proficiencies on a common scale, enabling employers to gauge employee job readiness and job seekers to understand and achieve needed skills. WorkKeys® foundation assessments (reading, writing, math, locating information, applied technology, listening, observation, teamwork) and a job analysis system are used to identify skills and associated proficiency levels required to perform job tasks. Using ACT’s SkillPro System, analysts profile job requirements and upload the resulting profiles to ACT’s JobPro Database. The JobPro database provides a rich source of data describing critical tasks and attendant foundational skill levels associated with a broad cross-section of America ’s jobs. This presentation uses a research-based example from the healthcare industry to illustrate the significant benefits of using foundational skills concepts to help healthcare professionals improve quality of care, gain higher job success, and expand career opportunities. Applications with other high-demand occupations are explored. Speaker: Oliver W. Cummings, Assistant VP, ACT, Inc.

____________________________________

Speaker: Ted Jackson

Education Division  Location: Rancho

Speaker: Jake Parizek

European Division  Location: Mirage Speaker: Paul Howarth

Industrial/Organizational Division  Location: Oasis 6 & 7 Speaker: Ryan Ross

____________________________________________________

Sponsor Debrief Meeting Location: Celebrity F

Innovations in Educational Assessment: An ETS Perspective Type: Cross Divisional 

This session will present some potential innovations in approaches to educational assessment coming from the interface of ETS’s research into psychometrics and educational policy. We will cover new approaches to large-scale summative assessment that address most of the complaints educators have today about those instruments, building balanced assessment programs with the right mix of assessment FOR and assessment OF learning, professional development issues around the proper use of balanced assessment systems, and the increased interest in non-cognitive assessments, especially in light of international competition for a workforce with 21st century skills. Speaker: John Oswald, Senior VP & General Manager, ETS

2009

Location: Oasis 1 & 2

4:30 PM - 5:30 PM

ETS Platinum Sponsor Session

Location: Ambassador 6

Clinical Division 

42

All sponsors are invited to this debrief meeting to provide feedback and insight to shape future ATP conferences to offer the exposure that offers the greatest benefit for your organization. If you are a 2009 sponsor please join us! ____________________________________________________

The Next Big Thing Sponsored by: 5:30 PM - 7:00 PM Location: Masters Plaza

Tuesday night we will focus the spotlight on what the future holds for the testing industry. What is the next big thing? Join us to find out!


Wednesday Conference Program

Wednesday, February 25th Registration Open

7:30 AM -

12:30 PM

Celebrity Foyer

Continental Breakfast with Exhibitors

7:30 AM -

8:30 AM

Celebrity Ballroom

Closing Keynote Panel: Testing in a Global Marketplace: Key Learnings that Create Assessment’s Future Landscape

8:30 AM - 10:00 AM

Ambassador 1-4

Morning Break

10:00 AM -

10:15 AM

Ambassador Foyer

Breakout Sessions

10:15 AM -

11:15 AM

Breakout Sessions

11:30 AM -

12:30 PM

Registration Open 7:00 AM - 12:30 PM Location: Celebrity Foyer

Continental Breakfast with Exhibitors 7:30 AM - 8:30 AM

Location: Celebrity Ballroom __________________________________________________________________________________________________________

Closing Keynote Panel 8:30 AM - 9:30 AM Testing in a Global Marketplace: Key Learnings that Create Assessment’s Future Landscape Location: Ambassador 1-4

Type: Cross Divisional

The Conference’s Closing Keynote Panel includes five industry experts who represent the breadth and geographical diversity of the testing industry and ATP’s Cyndy Fitzgerald Barbara Nichols David Foster Thomas Abrams Ronald K. Hambleton Mark Anderson membership as well as a range of perspectives on testing and assessment. The panel will be moderated by Cyndy Fitzgerald, and will address a variety of issues ranging from what we’ve learned in the 10 years since ATP conference was established, to the impact/implications of globalization on testing and assessment. We envision a lively exchange with lots of audience interaction. See you there! Moderator: Cyndy Fitzgerald, Senior Director, Client Services, KRYTERION, Inc. Panelists:

Barbara Nichols, CEO, Commission on Graduates of Foreign Nursing Schools David Foster, CEO, KRYTERION, Inc. Mark Anderson, President, Pearson Language Tests Ronald K. Hambleton, Distinguished University Professor,University of Massachusetts Thomas Abrams, Attorney, Vedder Price P.C. 43


Wednesday Conference Program Morning Break 10:00 AM - 10:15 AM Location: Ambassador Foyer

____________________________________________________

Breakout Sessions

Location: Oasis 1 & 2 Level: Intermediate

Cognitively-Based Assessment of, For and As Learning (CBAL) Type: Education 

Accountability testing is a critical tool for improving today’s education systems. However, new systems are needed that pay attention to the ways in which accountability affects classroom learning and instruction. The presenters describe research aimed at developing a system intended to document student achievement, help teachers plan what instruction should occur next and be a worthwhile learning experience. The approach attempts to synergistically unify three components: accountability assessment, formative assessment and professional support. Key characteristics include: (a) accountability tests, formative assessment and professional support, which are derived from the same conceptual base that has been built upon cognitive research, state standard, and curricular considerations; (b) cognitively-based assessment of, for and as learning (CBAL) assessments that consist largely of engaging, extended, constructed-response tasks that are delivered by computer and automatically scored or scored online by teachers; and (c) accountability assessment that is distributed over several administrations throughout the school year. Results from each administration will provide information for more extensive formative or diagnostic assessment. Speakers: Chad Buckendahl, Alpine Testing Solutions Malcolm Bauer, Senior Scientist, ETS

____________________________________

Cost Per Item Estimates the Opposing Counsel Can Understand Location: Oasis 5 Type: Certification/Licensure Level: Beginner

When those in the testing business assert the value of their items when defending their intellectual property rights, they toss around figures like $1,000/item to as much as $4,000/ item, but few have offered references or real-life practical experience to back up these estimates. This presentation outlines the process used to systematically estimate the costs to develop an item from a raw form to viable scored item. Though the final total may seem preposterous to those outside of testing (judges, juries, etc.), the presenters offer insights as to the complexity of the item development process that non-psychometricans can understand. The session also illustrates how ARRT tracked the item costs from 2005 to 2007 and how the pool of 344 newly written raw items shrank to only 34 scoreable items through the rigorous process of item development. Participants learn how to adapt the ARRT costestimate model to determine the value of their intellectual property in a way that the courts can understand.

2009

____________________________________

Creating More Secure Exams through Performance-Based Testing

10:15 AM - 11:15 AM

Location: Celebrity F Level: Intermediate

Speakers: Lauren Wood, Director, Psychometrics, American Registry of Radiologic Technologists Wayne Patience, Assistant Vice President, ACT Higher Education Programs Tom Biester, Director, Psychometrics, American Board of Surgery

44

Type: Certification/Licensure

One key argument for supporting a performance-based testing (PBT) initiative is that PBT is more secure. Because individuals have to perform a task, it is reasoned, there is little wiggle room for a test taker to “fake� a performance. This session is designed to provide participants with how-to examples and results data of performance-based items that have been integrated into computerized certification testing programs. The presenters discuss their three PBT initiatives; their successes and pitfalls in the development and use of PBT, and how PBT is used as a test security strategy. Topics such as creating multiple performances, making frequent exam changes and the use of statistical analysis to identify exam irregularities are discussed. Speakers: Dennis Maynes, Chief Scientist, Caveon Test Security Sierra Hampton, Manager, Exam Development, Citrix Systems, Inc. Johnny Weissmuller, US Air Force Andrew Wiley, Manager, Certification Exam Design and Development, The College Board

____________________________________

Flawed Leadership: Is it Pressure or Personality? Location: Celebrity H Level: Beginner

Type: Cross Divisional 

With the shrinking talent pool, expanded responsibilities of leaders and the intense pressure being placed on leaders, the impact of flawed leadership attributes has become a central focus across organizations. Every talent manager is trained to recognize the positive attributes that identify a candidate with high potential. However, when the pressure is on, a whole new side of the candidate emerges. These derailer characteristics are what peers and subordinates encounter during day-to-day interactions. If talent management is to build a work force for the future, these derailers should be at the forefront of discussions. This session explores current trends in proactively identifying candidate derailers prior to placement in a highpotential program, best practices to minimize their effect on performance, and preventing derailment by focusing on correct placement and promotion strategy. Finally, this session explores three key talent programs at Dell and shares best practice strategies for implementing assessments in worldwide development programs. Speakers: Brandy Agnew, Senior Consultant, Dell, Inc. Ryan Ross, Director, Alliance & Partners, Hogan Assessment Systems


Wednesday Conference Program Innovative Items - Turning Ideas into Reality Location: Rancho Level: Intermediate

Type: Certification/Licensure

The options available to test sponsors for innovative item types is only limited by a person’s imagination. While fantastic ideas abound, there is little understanding of how to turn those ideas into actual examinations and how to calculate the practical cost/benefit of doing so. This session examines the practical side of transforming ideas into reality in innovative testing. The presentation provides a project management framework for the development and implementation of innovative item types that includes the systematic steps required for item design, development, administration and scoring. It uses case-based suggestions on effective methods to manage the development costs and timelines, provides a model for evaluating returnon-investment and proposes processes for maximizing the contribution to test utility. Stakeholder perspectives and considerations are addressed for item developers, test delivery channels, psychometrics, test takers and sponsors. Speakers: Clarence Chaffee, President, The Caviart Group David Cox, President, Professional Testing, Inc. John Baker, CEO, Zoomorphix Systems

processing requirements for all examinees, while ensuring that measurement of the construct of interest is preserved. In several studies, linguistic modification has shown potential for improving the performance of ELLs on academic content tests. Separately, for mathematics tests, linguistically complex word problems are generally more difficult for ELLs, even when matched with non-ELLs, on overall mathematics proficiency. In contrast, certain item features, such as the use of symbolic representation in mathematics items has been found to benefit ELLs. This presentation is informative for all individuals involved in testing and those who are interested in the performance of ELLs. Speakers: John Young, Senior Research Scientist, Educational Testing Service Maria Martiniello, Associate Research Scientist, Educational Testing Service

____________________________________

New Testing Methodologies, Technology Developments, and Test Security Threats to Validity Location: Oasis 3 Level: Intermediate

Type: Industrial/Organizational 

____________________________________

Less Adverse Alternatives: A Review of the Department of Justice’s Advocacy in Light of Certiorari Petition for Alternatives Testing Case Presently Before the Supreme Court Location: Oasis 4 Level: Intermediate

The session features predictions of the future of test security by individuals playing key roles in program management, test development, research and security enhancement. They address the following questions:

Type: Industrial/Organizational 

For years, the US Department of Justice has pushed public sector employers to adopt “less adverse alternatives” to cognitive ability tests. In October 2008, the US Supreme Court was asked to take up a testing case that highlights the following Constitutional issue: “Does the Equal Protection Clause prohibit a municipal employer from discarding examination results on UIFHSPVOETUIBUAUPPNBOZBQQMJDBOUTPGPOFSBDFSFDFJWFEIJHI scores and in the hope that a future test would yield more highscoring applicants of other races?” Dave Arnold and Jim Sharf review district court decisions dealing with the Department of Justice’s advocacy of less adverse alternatives and review in detail the test case before the Supreme Court. Speaker: Jim Sharf, Sharf and Associates, Employment Risk Advisors, Inc.

t

8IBUDIBOHFTJOUFTUEFMJWFSZTFFNXFMMPOUIF way?

t

8IBUBSFLFZDIBMMFOHFTUPQSPEVDUJWFDIBOHF

t

8IJDIBSFBTOFFEUIFNPTUDIBOHF

t

8IJDIBSFBTPGDVSSFOUSFTFBSDIBOE   development look to be unpromising and possibly candidates to be put on the shelf?

t

8IFSFTIPVMEXFCFMFBSOJOHGSPNFBDIPUIFSBOE taking ideas from other parts of our field and industry and applying them in new contexts?

t

)PXNBZXFFYQFDUDIFBUFSTBOEUIJFWFTUP expand their methods of attack?

t

8JMMXFCFNPSFFòFDUJWFZFBSTGSPNOPXJO  combating cheating and the theft of items, tests and pools, than we are now?



____________________________________

Linguistic Complexity of Test Items and the Impact on the Performance of English Language Learners Location: Celebrity G Type: Education  Level: Intermediate

This presentation describes research findings regarding the linguistic complexity of test items and the impact on the performance of English language learners (ELLs) on academic content tests, such as those required under No Child Left Behind legislation. The presentation provides an overview of different test development approaches that have been investigated for improving the validity and fairness of assessments for ELLs. For example, the linguistic modification of items is a testing accommodation that lessens the verbal

Speakers: Eugene Burke, Science & Innovation Director, SHL Group Limited John Fremer, President, Caveon Test Security Ronald K. Hambleton, Distinguished University Professor, University of Massachusetts

____________________________________

Online Testing: Expanding the Reach of Test Delivery Location: Mirage Level: Beginner

Type: Certification/Licensure

The Internet is expanding in both reach and capacity. Regardless of how good the content is, if you are dialing up on a 14.4k modem your Internet experience will be tainted by your ability to access the information. Similarly, your test taker experience may be influenced by your ability to conveniently 45


Wednesday Conference Program Breakout Sessions 10:15 AM - 11:15 AM Cont.

provide tests delivery options. For test takers with disabilities or who live in remote locations traditional test delivery can be a significant challenge. Learn how you can apply online test delivery to your existing delivery options to provide expanded reach and convenience without sacrificing security or quality. Speaker: David Foster, CEO, KRYTERION, Inc.

____________________________________

ATA Inc. Platinum Sponsor Session China: A Booming Certification Market Location: Ambassador 6

Type: Cross Divisional 

Testing has a long history in China. It is part of a Chinese culture to be tested and differentiated. Meanwhile, though, many are under the mistaken impression that hardware, literacy, bandwidth and technologies lag behind in a developing country such as China. But upon closer inspection, China reveals itself to be a qualification-driven society. China is catching up with the rest of the sophisticated world of testing and certifications; its people seek qualifications to find and secure jobs to improve their lives. This session gives an in-depth analysis of the driving forces that propel the growth of the Chinese certification market including factors such as economic environment, job competitions, talent shortages, technology infrastructures and government policies. It also gives a forecast on the future growth of the market based on an IDC research report. Sponsors who are interested in entering this market will gain more understanding and get better prepared for their possible steps into what is potentially the largest testing market in the world. Speaker: Alex Tong, VP, Business Development, ATA Inc.

____________________________________

Breakout Sessions 11:30 AM - 12:30 PM Adaptation and Management of Test Items Location: Oasis 6 & 7 Type: International  Level: Intermediate

Although globalization has removed geographic barriers and opened new economic opportunities, it also poses a significant emerging challenge for test publishers and awarding bodies, including: “How do we manage the adaptation of test items for multiple languages and cultures efficiently and effectively?” This session outlines tactics and strategies to address this challenge through a blend of processes, technology, and adherence to guidelines and best practices. This session explores some of the procedures and solutions for adaptation currently in use at EXIN, an awarding body based in the Netherlands, which delivers certification programs on many different subjects within information sciences and IT management. It also explores current international guidelines for the adaptation of tests and the relationship with quality assurance. The session includes examples that illustrate multi-lingual item management. In addition, particular attention is paid to the challenges posed by test item management, specifically: problems posed by having some item management organized outside the item bank; quality assurance issues connected with the logging of issue reports outside the database; and the difficulty of navigating item management and item content. Speakers: Eric Shepherd, CEO, Questionmark Stefanie Moerbeek, Coordinator, Exam Development, EXIN

____________________________________

PSI Platinum Sponsor Session

Development and Use of Testing Standards and Best Practices

Preserving Test Security While Maintaining Standards Location: Ambassador 7

Location: Celebrity H Level: Beginner

Type: Cross Divisional 

2009

This session examines key ingredients of a successful certification or licensure testing program, including a system for constructing unlimited parallel test forms and implementing them in a test administration program. The system, called FormCast TM, is designed to ensure the success of high stakes examination programs by addressing two critical concerns: maintaining security of examination content and preserving professional standards for competence. The session will describe this solution to these key concerns, validation of the solution, and its deployment in a testing program, including a case study. The session brings together multiple diverse perspectives of internal and external psychometric specialists, a certification and licensure program expert, and a test program administrator and user of the system. The session highlights PSI’s approach to psychometric design and development, technology delivery, managed testing environment, and complete support and service infrastructure. Speakers: Brian Kim, Professor, Occidental College Foye Longworth, Departmental Analyst, State of Michigan, Department of Labor & Economic Growth, Licensing Division, Testing & Education Services Unit Greg Becker, Senior VP, Strategy & Market Development, PSI John Weiner, Chief Science Officer, PSI 46

Type: Cross Divisional 

What are the key functions, uses, and current status of major testing standards and best practices documents? t

"&3""1"/$.&i4UBOEBSETGPS&EVDBUJPOBM and Psychological Testing”



t

"/4**40*&$1FSTPOOFM$FSUJöDBUJPO"DDSFEJUBUJPO

t

*5$(VJEFMJOFTPO5FTU6TF (VJEFMJOFTPO  Computer-based and Internet Delivered Testing and Guidelines on Adapting Tests

t

&'1"&"801&VSPQFBO4UBOEBSETPO5FTUTBOE  on Test Use

t

"51#FTU1SBDUJDFT%PDVNFOU

What aspects of the development of these standards is most challenging? What appear to be the most significant recent changes in the measurement field and industry that will impact testing practice? What do the testing professionals who develop and manage testing and assessment programs need to focus on to be sure that their programs reflect current standards and best practices? These questions will be addressed by measurement professionals who are very knowledgeable


Wednesday Conference Program about the different documents and about how they can be applied to evaluate programs. Speakers: John Fremer, President, Caveon Test Security Jon Twing, Executive VP, Pearson Roy Swift, Program Director, American National Standards Institute Wayne Camara, VP, Research & Psychometrics, The College Board

____________________________________

How China is Securing Testing Location: Polo Room Level: Beginner

Type: International 

After two decades of fast economic development, China’s certification and testing market has undergone a transformation and explosion far beyond the imagination of the West. For many test sponsors whose lifelines rely upon the security of their item banks, the market in China is something they both desire and fear. With concerns over item security and IP infringement, test sponsors are still dragging their feet into potentially the largest testing market in the world. Exam security and integrity is actually becoming an increasing focus of attention in the testing community in China. Test sponsors and government authorities as well as commercial test service providers are actively seeking solutions to fight exam cheating and protect the IPs of test owners. This presentation offers a comprehensive overview of the legislative and administrative measures taken and the methodologies/technologies applied in ensuring safe and reliable test delivery in China. Best practices, issues, challenges and the underlying social and cultural drivers also are examined. Speaker: Alex Tong, VP, Business Development, ATA Inc.

____________________________________

Innovative Vision: Live Lab Testing Location: Celebrity G Level: Intermediate

Type: Certification/Licensure

Innovative lab testing can be delivered in virtual or live lab environments. In this forum, speakers share the trials and tribulations of developing creative exam environments that have resulted in huge face validity and raving respect from candidates and consumers. Moving beyond the ability to assess individuals’ core knowledge, these exams allow candidates to demonstrate their skills and abilities in real-world environments. The design, development, deployment and measurement of innovative testing environments pose many challenges. The speakers share the requirements for specifying the goal of the exam(s); the challenges of training individuals to develop performance items with multiple scoring opportunities; the process for determining the best way to collect and analyze scoring opportunities; how to address validity realities; and the challenges of deployment of items and collection of data required for analysis and defensibility. Speakers: Erik Ullanderson, Manager, Global Certifications, Cisco Systems Gopal Krishnan, Product Manager, Pearson VUE Melissa Tuite, Senior Manager, Technical Certification, VMware, Inc. Robert Cromartie, VMware, Inc. Russell Smith, Senior Psychometrician, Alpine Testing Solutions

Pre-Equating and Its Impact on the Cut Score Under Different Scoring Models Location: Oasis 5 Level: Intermediate

Type: Certification/Licensure

When immediate score reporting is a required condition for a professional certification and/or licensure testing program and a standard setting has not been completed, pre-equating can be an appropriate approach to ensure comparability of test scores across forms on a common metric. This research identifies various conditions and constraints for pre-equating and explores the impact of pre-equating on cut scores in comparison to post-equating under different scoring models. Sharing specific examples with participants, this session presents the process of pre-equating given specific conditions by the certification and licensure board. Speakers: Larissa Smith, Psychometrician, Prometric Shungwon Ro, Psychometric Supervisor, Prometric

____________________________________

Test Taker Rights and Responsibilities Location: Celebrity F Level: Intermediate

Type: Education 

Testing is the primary means by which life decisions are made in business, education, the military, hospitals and the civil service. Tests do not inherently discriminate against those who take them, and, undoubtedly, scores derived from tests have been used to admit, advance and employ individuals. Occasionally, test results are used inappropriately. Testing incidents can affect a test taker’s privacy or the opportunity to gain admission or employment, and the occurrence of such events may prompt legal proceedings or even regulation. How can testing organizations protect test takers, test users and, ultimately, themselves from the negative consequences of such events? This session explores the establishment of a document of rights and responsibilities for test takers that covers test fairness, customer service, privacy, security, reporting results and advocacy. It also explores the implementation of an “Ethics & Compliance” panel, comprised of individuals external to the organization, to analyze incidents and recommend changes to testing policies and procedures in relation to these rights and responsibilities. Speakers: Ashok Sarathy, Director, GMAT Operations, GMAC Peg Jobst, VP, GMAT Program, GMAC

____________________________________

Troubled Times for Workplace Testing? Validation of Tests and Selection Procedures in Light of Revised Government Reviews Location: Oasis 4 Level: Intermediate

Type: Industrial/Organizational 

The use of tests and other assessments for workplace decision making is increasing. At the same time, regulatory agencies such as the EEOC and OFCCP are increasing their scrutiny on these procedures. In light of an increased focus on test validation on the part of the agencies that audit and litigate employment practices, testing specialists should be well versed on the recent changes to the methodologies and practices used by these agencies. This presentation reviews the changes on how selection processes are being reviewed and the implications for the implementation and validation 47


Wednesday Conference Program Breakout Sessions 11:30 AM - 12:30 PM Cont.

of assessments in the workplace context. Pros and cons of the various approaches are discussed in light of recent audit requirements. Examples from operational selection systems and recent audit results are shared and discussed in the session. Commentary on recent trends is provided by workplace assessment specialists and an employment lawyer. The session focuses specifically on government auditing practices and their implications for assessment practice. Speakers: Doug Reynolds, VP, Assessment Technology, Development Dimensions International (DDI) John Weiner, Chief Science Officer, PSI Keith Pyburn, Fisher & Phillips, LLP

____________________________________

Two for One: Leveraging Assessment Data for Employee Selection and Development Location: Oasis 3 Level: Intermediate

Type: Industrial/Organizational 

Organizations are increasingly using assessments in their applicant screening processes. Moreover, firms are more likely to use job analyses, validation studies and competency model linkages to show how assessments predict successful job performance. Firms expend many resources selecting assessments, conducting implementation research, and building profiles to help them identify high performers and make informed hiring decisions. Despite the investment in assessments, the resulting data is often cast aside once a job offer is accepted. However, the assessment data used to determine the candidate’s fit for the position could also be used for orientation and development. The data is available, so why do companies not use it with a development focus to increase the candidate’s success on the job? This session reviews strategies for leveraging assessment data for both employee selection and development purposes. The session includes issues to consider and specific methods for using this approach to help new hires get up to speed, socialize within the organization, prepare for their new role, and feel that their new firm values them and wants to help them achieve success. Speakers: Craig Haas, Consultant, Hogan Assessment Systems Jocelyn Courtney-Hays, Consultant, Human Capital Solutions, pan - A TALX Company

Cito Platinum Sponsor Session Hit the road with Cito! Digital Assessment Instruments to Monitor Driver Behavior and Improve Driver Skills Location: Ambassador 6

Type: Cross Divisional 

CitoDrive is an E-platform for multimedia assessments of driver competence: It supports driver training and it monitors driver behavior and effects. There is a life-long learning approach for different age and target groups. The stakes are: successful exams, cost-reduction, safety and a healthy environment. Against a background of increased emphasis on driver safety it is important to assess critical driver competencies and to support driver training for a wide range of competence levels. Digital testing is very efficient: it gives you strong evidence at low costs, a minimal investment of time, resources and people and you minimize the risks for the candidate or other drivers. During the session you will experience some examples of this multimedia test; different ways of testing perception and danger detection, and also the unique filtering system with which it is possible to offer drivers the exact amount of training to improve their driving skills. Speaker: Aranka Krechting, Account Manager, Cito

____________________________________

Alpine Testing Solutions, Inc. Platinum Sponsor Session Real Tools for Real Testing Programs Location: Ambassador 5

Type: Cross Divisional 

Technology continues to be of increasing importance in supporting testing programs. However, technology can easily become a financial and resource drain with little recognizable benefit! Value proposition, ease of access, security, collaboration, customization and support are essential criteria to consider when evaluating tools and technologies that support the test development and data management process. These essential criteria will be discussed and applied against ItemMetrics™ and CertMetrics™, item development and data management tools developed with meticulous attention to accuracy and functionality by Alpine Testing Solutions’ psychometric and software development teams. Speakers: Blair Harris, Test Developer, Alpine Testing Solutions Russell Smith, Senior Psychometrician, Alpine Testing Solutions Brian Adams, President and CEO, Alpine Testing Solutions

ProExam

Thank you 2009

for your contribution to the success of Innovations in Testing 2009! 48


Speaker Index

2009

Speaker

Page

Abrams, Thomas . . . . . . . . . . . . . . . . . . . . . 43 Ackermann, Richard . . . . . . . . . . . . . . . . . . 39 Adams, Brian. . . . . . . . . . . . . . . . . . . . . . . . . 48 Agnew, Brandy. . . . . . . . . . . . . . . . . . . . . . . 44 Altman, Jason . . . . . . . . . . . . . . . . . . . . . . . . 25 Anastasio, Ernie . . . . . . . . . . . . . . . . . . . . . . 23 Anderson, David . . . . . . . . . . . . . . . . . . . . . 15 Anderson, Mark . . . . . . . . . . . . . . . . . . . . . . 43 Appelgren, Lise-Lotte . . . . . . . . . . . . . . . . 18 Arbet, Scott . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Arnold, David . . . . . . . . . . . . . . . . . . . . . . . . 36 Baker, John . . . . . . . . . . . . . . . . . . . .18, 32, 45 Barry, Carol . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Bauer, Malcolm . . . . . . . . . . . . . . . . . . . . . . 44 Becker, Greg . . . . . . . . . . . . . . . . . . . . . . . . . 46 Becker, Kirk . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Biester, Tom . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Bomer, Jamie . . . . . . . . . . . . . . 26, 30, 33, 40 Bontempo, Brian . . . . . . . . . . . . . . . . . . . . . 35 Bovell, Dwan . . . . . . . . . . . . . . . . . . . . . . . . . 40 Brittingham, Peter . . . . . . . . . . . . . . . . . . . 26 Brooks, Amy . . . . . . . . . . . . . . . . . . . . . . . . . 16 Bruce, Wes . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Buckendahl, Chad . . . . . . . . . . . . . . . . . . . . 44 Burgoyne, Robert . . . . . . . . . . . . . . . . . . . . 30 Burke, Eugene . . . . . . . . . .15, 32, 35, 40, 45 Burns, Liz . . . . . . . . . . . . . . . . . . . . . .14, 30, 39 Burroughs, Jill . . . . . . . . . . . . . . . . . . . . . . . . 30 Camara, Wayne . . . . . . . . . . . . 14, 23, 38, 47 Cannata, Joe . . . . . . . . . . . . . . . . . . . . . . . . . 40 Carswell, Julie . . . . . . . . . . . . . . . . . . . . . . . . 35 Chaffee, Clarence . . . . . . . . . . . . . . . . . 32, 45 Christensen, Jerry . . . . . . . . . . . . . . . . . . . . 12 Clark, Gary . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Cline, Jay . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Collins, Fiona . . . . . . . . . . . . . . . . . . . . . . . . 16 Cook, Linda . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Cooney, Gavin . . . . . . . . . . . . . . . . . . . . . . . 25 Courtney-Hays, Jocelyn . . . . . . . . . . . . . . 48 Cox, David . . . . . . . . . . . . . . . . . . . . . . . . 32, 45 Cromartie, Robert . . . . . . . . . . . . . . . . . 36, 47 Cummings, Oliver W . . . . . . . . . . . . . . . . . 42 Cyr, Dale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Davis, Laurie . . . . . . . . . . . . . . . . . . . . . . . . . 19 Davis, Susan . . . . . . . . . . . . . . . . . . . . . . 19, 26 De Jong, John H.A.L. . . . . . . . . . . . . . . . . . 23 Devine, Julie . . . . . . . . . . . . . . . . . . . . . . . . . 19 DeVore, Richard . . . . . . . . . . . . . . . . . . . . . . 26 Dille, Stephanie . . . . . . . . . . . . . . . . . . . . . . 12 Dunbar, Stephen . . . . . . . . . . . . . . . . . . . . . 29 Duvin, Helene. . . . . . . . . . . . . . . . . . . . . . . . 23 Ehrlich, Lisa . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Elliot, Scott . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Eter, Dania . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Evans, Amanda . . . . . . . . . . . . . . . . . . . . . . 21 Eyob, Daniel . . . . . . . . . . . . . . . . . . .14, 28, 40 Megan, France . . . . . . . . . . . . . . . . . . . . . . . 16 Fetzer, Mike . . . . . . . . . . . . . . . . . . . . . . . . . . .## Fitzgerald, Cyndy . . . . . . . . . . . . . . . . . 28, 43 Foster, David . . . . . . . . . . . . . . . 12, 35, 43, 46 Fremer, John . . . . . . . . . . . . . . . . . . . . . . 45, 47 Garcia, Alejandra . . . . . . . . . . . . . . . . . . . . . 30

Speaker

Page

Garcia, Ruben . . . . . . . . . . . . . . . . . . . . . 19, 40 George-Ezzelle, Carol. . . . . . . . . . . . . . . . . 19 Geranpayeh, Ardeshir . . . . . . . . . . . . . 36, 39 Griffin, Stephen . . . . . . . . . . . . . . . . . . . . . . 21 Guo, Fanmin . . . . . . . . . . . . . . . . . . . . . . . . . 30 Gutierrez, Sara . . . . . . . . . . . . . . . . . . . . . . . 21 Haas, Craig. . . . . . . . . . . . . . . . . . . . . . . . 40, 48 Hambleton, Ronald K. . . . . . . . . . .40, 43, 45 Hampton, Sierra . . . . . . . . . . . . . . . . . . 26, 44 Hardy, Dean . . . . . . . . . . . . . . . . . . . . . . . . . 36 Harmes, J. Christine . . . . . . . . . . . . . . . 15, 16 Harris, Blair . . . . . . . . . . . . . . . . . . . . . . . . . . 48 Harris, Deborah . . . . . . . . . . . . . . . . . . . . . . 38 Harris, William G. . . . . . . . . . . . . . . .14, 23, 39 Henderson, Gwen . . . . . . . . . . . . . . . . . . . . 15 Herzberg, David. . . . . . . . . . . . . . . . . . . . . . 36 Hinshaw, Kathy . . . . . . . . . . . . . . . . . . . . . . 38 Jobst, Peg. . . . . . . . . . . . . . . . . . . . . .36, 38, 47 Jones, John . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Jung, Cordell . . . . . . . . . . . . . . . . . . . . . . . . . 40 Kaliski, Pamela . . . . . . . . . . . . . . . . . . . . . . . 16 Kantrowitz, Tracy . . . . . . . . . . . . . . . . . . . . . 25 Karges, Kaiya. . . . . . . . . . . . . . . . . . . . . . . . . 19 Kim, Brian. . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Kingsbury, Gage . . . . . . . . . . . . . . . . . . . . . 38 Klion, Reid . . . . . . . . . . . . . . . . . . . . . . . . 25, 35 Krechting, Aranka . . . . . . . . . . . . . . . . . . . . 48 Krishna, Vijay. . . . . . . . . . . . . . . . . . . . . . . . . 33 Krishnan, Gopal . . . . . . . . . . . . . . . . . . . . . . 47 Kroecker, Tim . . . . . . . . . . . . . . . . . . . . . . . . 25 Kubach, Doug . . . . . . . . . . . . . . . . . . . . . . . 39 Kunce, Chuck . . . . . . . . . . . . . . . . . . . . . . . . 14 Lee, Sung . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Lipner, Rebecca . . . . . . . . . . . . . . . . . . . . . . 32 Longworth, Foye . . . . . . . . . . . . . . . . . . . . . 46 Majeau, Renate . . . . . . . . . . . . . . . . . . . . . . 12 Mannes, Benjamin . . . . . . . . . . . . . . . . . . . 40 Marks, Casey . . . . . . . . . . . . . . . . . . . . . . . . . 23 Martiniello, Maria . . . . . . . . . . . . . . . . . . . . 45 Mattern, Krista . . . . . . . . . . . . . . . . . . . . . . . 38 Maynes, Dennis . . . . . . . . . . . . . . . .30, 33, 44 Meyer, Kevin . . . . . . . . . . . . . . . . . . . . . . 30, 33 Meyers, Jason . . . . . . . . . . . . . . . . . . . . . . . . 19 McClean, Sean . . . . . . . . . . . . . . . . . . . . . . . .## McClintock, Joe . . . . . . . . . . . . . . . . . . . . . . 40 Melican, Jerry . . . . . . . . . . . . . . . . . . . . . . . . 14 Melley, Chuck . . . . . . . . . . . . . . . . . . . . . . . . 39 Moerbeek, Stefanie . . . . . . . . . . . . . . . . . . 46 Moore, Steven . . . . . . . . . . . . . . . . . . . . . . . 36 Mulkey, Jamie . . . . . . . . . . . . . . . . . . . . . . . . 16 Munson, Liberty . . . . . . . . . . . . . . . . . . . . . 26 Nenshi, Nazir. . . . . . . . . . . . . . . . . . . . . . . . . 18 Nichols, Barbara. . . . . . . . . . . . . . . . . . . . . . 43 Nichols, Paul . . . . . . . . . . . . . . . . . . . . . . . . . 19 Niero, Christine . . . . . . . . . . . . . . . . . . . 15, 16 Olsen, James . . . . . . . . . . . . . . . . . . . . . . 18, 39 Oswald, John . . . . . . . . . . . . . . . . . . . . . . . . 42 Parshall, Cynthia . . . . . . . . . . . . . . . . . . . . . 15 Patience, Wayne . . . . . . . . . . . . . . . . . . 33, 44 Pethick, Layne . . . . . . . . . . . . . . . . . . . . 19, 28 Pia, Gerald . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Place, Jeffery . . . . . . . . . . . . . . . . . . . . . . . . . 28 50

Speaker

Page

Poole, Mark . . . . . . . . . . . . . . . . . . . . . . . 33, 40 Pope Greg . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Pyburn, Keith . . . . . . . . . . . . . . . . . . . . . 30, 48 Radwin, Gene . . . . . . . . . . . . . . . . . . . . . . . . 14 Ramaswamy, Rohit . . . . . . . . . . . . . . . . . . . 15 Rebello, Carly . . . . . . . . . . . . . . . . . . . . . . . . 15 Reynolds, Doug . . . . . . . . . . . . . . . . . . . 35, 48 Rickard, Patricia . . . . . . . . . . . . . . . . . . . . . . 39 Ro, Shungwon . . . . . . . . . . . . . . . . . . . . . . . 47 Roche, Brian . . . . . . . . . . . . . . . . . . . . . . . . . 26 Ross, Ryan . . . . . . . . . . . . . . . . . . . . .35, 42, 44 Rudner, Lawrence . . . . . . . . . . . . . . . . . . . . 12 Sale, Chris . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Sammet, Michael . . . . . . . . . . . . . . . . . . . . 35 Sarathy, Ashok . . . . . . . . . . . . . . . . . . . . 14, 47 Semko, Jennifer Ancona . . . . . . . . . . . 30, 39 Sharf, Jim . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Shepherd, Eric . . . . . . . . . . . . . . . . .12, 32, 46 Sitarenios, Gill. . . . . . . . . . . . . . . . . . . . . . . . 36 Sluijter, Cor . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Smith, Larissa . . . . . . . . . . . . . . . . . . . . . . . . 47 Smith, Russell . . . . . . . . . . . . . . 19, 26, 47, 48 Snead, Charles . . . . . . . . . . . . . . . . . . . . . . . 32 Snowhite, Larry . . . . . . . . . . . . . . . . . . . . . . 39 Sorensen, Don . . . . . . . . . . . . . . . . .12, 18, 36 Steinkamp, Susan . . . . . . . . . . . . . . . . . . . . 29 Strongin, Matthew . . . . . . . . . . . . . . . . . . . 36 Sullivan Warias, Renee . . . . . . . . . . . . . . . . 14 Sutton, Quinn . . . . . . . . . . . . . . . . . . . . . 12, 35 Swartz, Richard . . . . . . . . . . . . . . . . . . . . . . 29 Swift, Roy . . . . . . . . . . . . . . . . . . . . . . . . . 33, 47 Talento-Miller, Eileen . . . . . . . . . . . . . . . . . 30 Taylor, Linda . . . . . . . . . . . . . . . . . . . . . . . . . 39 Thayn, Kim. . . . . . . . . . . . . . . . . . . . . . . . 12, 32 Thayn, Scott . . . . . . . . . . . . . . . . . . . . . . 12, 32 Thiemann, Alan . . . . . . . . . . . . . . . . . . . 30, 29 Thieme, Michael . . . . . . . . . . . . . . . . . . . . . 40 Thomas, Leslie . . . . . . . . . . . . . . . . . . . . . . . 28 Thurlow, Martha . . . . . . . . . . . . . . . . . . . . . 25 Tipton, Amy. . . . . . . . . . . . . . . . . . . . . . . . . . 19 Tong, Alex . . . . . . . . . . . . . . . . . . . . . . . . 46, 47 Tuite, Melissa . . . . . . . . . . . . . . . . . . . . . . . . 47 Turner, Tina . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Twing, Jon . . . . . . . . . . . . . . . . . . . . . . . . 23, 47 Ullanderson, Erik . . . . . . . . . . . . . . . . . . . . . 47 van der Linden, Wim J. . . . . . . . . . . . . . . . 35 Wagner, Donald . . . . . . . . . . . . . . . . . . . . . . 28 Wang, Xiang Bo . . . . . . . . . . . . . . . . . . . . . . 29 Ward, Patrick . . . . . . . . . . . . . . . . . . . . . . 12, 38 Weiner, John . . . . . . . . . . . . . . . 30, 35, 46, 48 Weissmuller, Johnny . . . . . . . . . . . . . . 12, 44 Welch, Catherine . . . . . . . . . . . . . . . . . . . . . 29 West, Bill . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Wheldon, Hazel . . . . . . . . . . . . . . . . . . . 11, 36 Whorton, Bob . . . . . . . . . . . . . . . . . . . . . . . . 30 Wild, Cheryl . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Wiley, Andrew . . . . . . . . . . . . . . . . .14, 38, 44 Williams, Kevin . . . . . . . . . . . . . . . . . . . . 25, 36 Wood, Lauren . . . . . . . . . . . . . . . . . . . . . 16, 44 Woodley, Cynthia . . . . . . . . . . . . . . . . . . . . 15 Young, John . . . . . . . . . . . . . . . . . . . . . . . . . 45 Young, Phil . . . . . . . . . . . . . . . . . . . . . . . . . . 39


Your Voice in The Testing Community Leading publishers and assessment service providers in today’s testing industry.

Train on best practices and intelligent test use Educate policy makers and the public on merits of testing Safeguard the intellectual property of test publishers Trade insights and share experiences within the testing community

Find your voice today!

For membership information call 1-866-240-7909 or visit: TestPublishers.org


2009

ATP Members Academic Therapy Publications A&D Resource AICPA Alpine Testing Solutions American Institutes for Research (AIR) American Printing House for the Blind ATA Inc. Assessment Distribution Services Assessment Systems Assessment Systems Corporation Autodesk, Inc. Bay State Psychological Associates, Inc. Board of Certification for Emergency Nursing Cambridge Assessment The Cambridge Don CASTLE Worldwide Caveon Test Security CASAS Center for Credentialing & Education, Inc. Center on Education and Training for Employment Certification Management Services Cisco Systems, Inc. Criteria Corporation Cito B.V. The Clark Wilson Group The College Board Consulting Psychologists Press, Inc. Craft Systems, Inc. Criteria Corporation CTB/McGraw Hill Data Recognition Corporation Directional Insight International The Donath Group Educational & Industrial Testing Service Educational Testing Service (ETS) EMC Corporation Employment Research & Development Institute Envisia Learning ETS Global Executive Development Assessment Centre Express Evaluations First Advantage Background Services Corp. Galileo Systems, LLC Genuine Genius Solutions Global Psychometric Services, Inc. Graduate Management Admission Council Harcourt Assessment Inc. Hester Evaluation Systems, Inc. Hewlett Packard Hogan Assessment Systems Hogrefe & Huber Publishers IDS Publishing Corporation Industrial Psychology International, Ltd. Inscopia Software Institute for Personality and Ability Testing Internet Testing Systems Integral7, Inc. Integriview KRYTERION, Inc. Knapp & Associates International, Inc. Lamark B.V.

LaserGrade, L.P. LIMRA International Lotus Development Corporation Marica Andberg Associates Measured Progress, Inc. Measurement Incorporated Meta Metrics, Inc. MetriTech, Inc. Metro Institute Microsoft Corporation MindData Systems, Ltd. Morris & McDaniel, Inc. Mountain Measurement, Inc. Multi-Health Systems, Inc. MySQL AB National Council of State Boards of Nursing NetApp Nichols & Molinder Assessments Organization Analysis & Design Pearson Assessments Pearson Educational Measurement Pearson VUE PEN People Focus, Inc. Performance Assessment Network Performance Testing Council (PTC) Personnel Systems Corporation Professional Credential Services, Inc. Professional Examination Service Profiles International, Inc. Prometric Psychological Assessment Resources, Inc. PSI Services, LLC Psychological Services Bureau, Inc. Psychometrics Canada, Ltd. Quality Metric, Inc. Questar Assessment, Inc. Questionmark Ramsay Corporation Research in Motion, Ltd. The Riverside Publishing Company SAP AG The SASSI Institute Scantron Schroeder Measurement Technologies, Inc. ScientificSelection.com, LLC SHL Sigma Assessment Systems, Inc. Storage Networking Industry Associates Summit Consulting Group Targeted Testing, Inc. Tests for Higher Standards Vangent, Inc. Vantage Learning, Inc. View Assessments, Inc. Western Psychological Services Wild & Associates Winslow Research Institute Wonderlic, Inc.

54


Exhibit Hall Map & Exhibitors

Celebrity Ballroom 309 104

102

100

Cyber Cafe

411

406

410

107 206

207 306

105 204

205 304

305 404

407

203 302

303 402

405

201 300

301 400

101 200

ENTRANCE

ACT, Inc. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400

Hogan Assessment Systems . . . . . . . . . . . . . . 302

Alpine Testing Solutions . . . . . . . . . . . . . . . . . . 206

Iscopia Software . . . . . . . . . . . . . . . . . . . . . . . . . 404

American National Standards Institute. . . . 305

Integral7, Inc. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304

Applied Measurement Professionals, Inc. . . . . 402

Internet Testing Systems . . . . . . . . . . . . . . . . . 300

ATA Inc. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

KRYTERION, Inc. . . . . . . . . . . . . . . . . . . . . . . . . . . 101

ATP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .102

McCann Associates . . . . . . . . . . . . . . . . . . . . . . . 411

CASTLE Worldwide . . . . . . . . . . . . . . . . . . . . . . . 205

Measurement Incorporated . . . . . . . . . . . . . . 410

Caveon Test Security. . . . . . . . . . . . . . . . . . . . . 403

Mensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203

Certification Management Services. . . . . . . 309

NPC, Inc. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .407

Cito. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306

pan - A TALX Company. . . . . . . . . . . . . . . . . . . 303

The College Board. . . . . . . . . . . . . . . . . . . . . . . . 301

Pearson VUE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201

Comira. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204

Professional Examination Service . . . . . . . . . 405

Comms Multilingual Ltd, UK . . . . . . . . . . . . . . 105

PSI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

ETS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200

Questionmark . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

GMAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 406

Summit Consulting Group. . . . . . . . . . . . . . . . 207 55

403


2009

Notes

56


Notes

57


2009 Platinum Sponsors

10% Cert no. BV-COC-953237


ATP 2009