Conference Space
CHA
F
BREAKFAST KEYNOTES AND PLENARY SESSIONS EXHIBITS BREAKOUTS AND DISCUSSIONS DEMO COMPETITION
EO
VEREMA RESTAURANT
TRELL
GARNACHA BALL ROOM SITGES I
MPHITHEATER THÉÂTRE GINESTA
GARRAF
BREAKFAST VEREMA GARDEN/JARDIN
MACABEO
BALL ROOM SITGES II
PARELLADA MONASTRELL SYRAH AMPHITHEATER AMPHITHÉÂTRE GINESTA
SAMSÓ MOSCATELL RIBERA I-II FRAGATA I-II ESTANYOL I-II CÒDOLS I-II-III
2
BREAKFAS KEYNOTES EXHIBITS BREAKOUT DEMO COM
Table of Contents
Wednesday Conference Programme. . . . . 9 - 15 Thursday Conference Programme. . . . . 17 - 32 Friday Conference Programme . . . . . . . . 34 - 40 Special Thanks . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Conference Space. . . . . . . . . . . . . . . . . . . . . . . . . 2 Schedule-at-a-Glance. . . . . . . . . . . . . . . . . . . . . . 3 Welcome Letter. . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Keynote Speaker Profiles . . . . . . . . . . . . . . . . 5 - 6 2010 Conference Sponsors. . . . . . . . . . . . . . . 7 - 8
Schedule-at-a-Glance Wednesday, 29 September 2010
10:00 – 17:15 13:00 – 14:15 14:15 – 14:30 14:30 – 15:00 15:00 – 15:30 15:30 – 16:30 16:30 – 17:15
Registration Open Opening Keynote Address Sitges Ballroom I Break Sponsor Sessions Exhibits and Coffee Break Sitges Ballroom II Breakout and Discussion Sessions Exhibits and Networking Sitges Ballroom II
Thursday, 30 September 2010
07:00 – 17:15 07:00 – 08:30 08:30 – 09:30 09:30 – 09:45 09:45 – 10:30 10:30 – 10:45 10:45 – 11:45 11:45 – 12:30 12:30 – 13:30 13:30 – 14:30 14:30 – 14:45 14:45 – 15:15 15:15 – 15:30 15:30 – 16:30 16:30 – 17:15 Evening
Registration Open Breakfast Verema Restaurant Keynote Address Sitges Ballroom I Break CAT meets DOG Sitges Ballroom I Break Breakout and Discussion Sessions Demo Competition and Exhibits Sitges Ballroom II & Parellada Lunch with Exhibitors Sitges Ballroom II CEO's Perspective Sitges I Break Sponsor Sessions Break Breakout and Discussion Sessions Exhibits and Networking Sitges Ballroom II Prometric Fiesta Village of Sitges
Friday, 1 October 2010
07:00 – 13:00 07:00 – 09:00 09:00 – 10:00 10:00 – 10:15 10:15 – 11:15 11:15 – 11:30 11:30 – 12:30 12:40 – 13:00
Registration Open Breakfast Verema Restaurant Closing Keynote Sitges Ballroom I Break Breakout and Discussion Sessions Break Breakout and Discussion Sessions Final Consideration Sitges Ballroom I
Registration Hours Wednesday, 29 September 10:00 – 17:15
Thursday, 30 September 07:00 – 17:15 3
Friday, 1 October 07:00 – 13:00
Dear Conference Delegates, It gives us great pleasure to welcome you all to Barcelona and to the beautiful Dolce Sitges for our second annual Growing Talent in Europe and Gaining Advantage through Assessment conference. The Association of Test Publishers (ATP) established the European Division (E-ATP) in 2006 to provide test publishers and providers of assessment tools and services in Europe with a collective voice. Our mission is to educate and engage the various communities of stakeholders in the testing industry, as well as those who set policy and guidelines. We are pleased to be here in Barcelona with the foremost leaders in the European testing industry, along with leading educators, developers and experts in the field of assessment. Our conference committee has worked hard to provide a rich and diverse program. The purpose of this conference is to facilitate networks, encourage delegates to identify new business opportunities, and inform industry professionals about the latest developments and research projects in the field of educational and performance assessment. Be sure to join us each day to hear our invited keynote speakers: David Bearfield, Director of the European Personnel Selection Office, Sara de Freitas, Director of Research at the Serious Games Institute and Olaf Kรถller, Director at the Leibniz Institute for Science Education. These speakers will kick off each day of our conference with their intriguing experiences and ideas. We also hope you will take time at this conference to pursue networking opportunities which will serve to enrich both your personal and professional lives as you make friends and contacts from the different areas of the assessment community. And finally we would like to acknowledge that this conference would not be possible without the financial support of our valued sponsors. We urge all delegates to visit their booths and attend their sponsor sessions.
We wish you a productive conference as well as a relaxing, educational experience! Sincerely,
Eugene Burke Chair, E-ATP Science and Innovation Director, SHL Group Ltd.
William G. Harris Chief Executive Officer, Association of Test Publishers 4
Keynote Speaker Profiles David Bearfield
Director European Personnel Selection Office (EPSO) David graduated in 1989 from the University of Salford, England, with a 1st Class BA Hons degree in Modern Languages. He speaks fluent English, French, German and Swedish. From there, he went on to enter the UK civil service European Fast Stream, firstly in the Transport Ministry before joining the Diplomatic Service. After succeeding in a graduate entry open competition to work for the European Commission, David began his EU career in the selection and recruitment department. Over the course of the past 16 years, he has held a variety of posts in the fields of HR, Communication and Change Management, including working in the Private Office of its Vice-Presidents Neil Kinnock and Siim Kallas, and being Head of Internal Communication. In total, David has 20 years of experience in the public sector. Since 2007, David has been Director of the European Personnel Selection Office and has, amongst other achievements, successfully secured a mandate for a far-reaching overhaul of selection procedures for civil servant entry to the EU Institutions to bring them in line with global best practice.
Sara de Freitas
Director of Research Serious Games Institute at the University of Coventry Sara is Director of Research at the Serious Games Institute at the University of Coventry where she leads an applied research team working closely with industry. Recently appointed Professor of Virtual Environments she holds a visiting senior research fellowship at the London Knowledge Lab, London, and is a Fellow of the Royal Society of Arts. In 2003 Sara founded (and continues to Chair) the UK Lab Group, which brings the research and development community together to create stronger links between industrial and academic research through supporting collaborative programmes and for showcasing innovative R&D solutions for the knowledge economy. Voted the Most Influential Woman in Technology 2009 by US Fast Company, Sara chaired the IEEE Serious Games and Virtual Worlds conference in 2009, and is a regular speaker at international conferences. Sara currently holds funding from the Advantage West Midlands, Erasmus Scheme, European Regional Development Fund, European Union and the Technology Strategy Board. Her current lines of research are Examining Multimodal Interfaces, Experience Design and Perceptual Modeling in Games and Virtual Worlds. Sara publishes widely in the areas of: Pedagogy and e-Learning; Change Management and Strategy Development for Implementing e-Learning Systems and Serious Games and Virtual Worlds for Supporting Training and Learning.
5
Keynote Speaker Profiles Olaf Kรถller
Director, Department of Education Leibniz Institute for Science and Math Education, Kiel, Germany Olaf Kรถller is currently Director of the Department of Education at the Leibniz Institute for Science Education (IPN) and full professor of educational research at the University of Kiel. Till September 2009, he was the founding Director of the German Institute for Educational Progress (IQB). He started his scientific career at the Leibniz Institute for Science Education (IPN) at the University of Kiel, Germany after graduating with a degree in Psychology in 1991. In 1996 he changed to the Max Planck Institute for Human Development, where he finished his dissertation (Ph. D.) in 1997. In 2002 he accepted the offer of a full professorship at the University of Erlangen-Nuremberg. In 2004 he changed to Humboldt University, Berlin. As the founding director of IQB, Olaf Kรถller was responsible for the national assessment program in Germany, which means that IQB conducts large-scale studies to test whether students in Germany reach national educational standards. Aside from his activities in academic assessment, his major research interests are Reciprocal Effects of Academic Self-Concepts and Achievement, the Development of Academic Interests and their Effects on Achievement, Educational and Occupational Choices. Olaf Kรถller has published approximately 150 national and international journal articles, book chapters and monographs.
6
Gold
Thank You to Our 2010 Sponsors www.cito.com Worldwide, Cito provides and evaluates good, fair tests. Cito’s measurement and monitoring method enables our clients to get an objective analysis of knowledge, skills and competences. This enables sound choices in the area of personal and professional development. Not only do we utilise our expertise to produce our own work, but also to offer advice, support and research to others. www.gmac.com or www.mba.com The Graduate Management Admission Council is a nonprofit organization of leading graduate business schools and is dedicated to creating access to and disseminating information about graduate management education. The Council owns the GMAT® exam, which nearly 5,000 graduate management programs at approximately 1,900 business schools worldwide use to assess their applicants. www.gmac.com or www.mba.com The Graduate Management Admission Test® (GMAT®) is the only admissions test developed for business schools, by business schools, and used by nearly 5,000 graduate business programs worldwide. Backed by more than 50 years of research, the GMAT is a valid predictor of academic performance in the first year of graduate management study.
LPLUS E X A M I N AT I O N AND TRAINING MANAGEMENT
www.lplus.de LPLUS GmbH has an experience of over 20 years as niche provider for the organisation and implementation of computer-based testing. The LPLUS-System® is one of European leading integrated software for extent automatized assessment management. It supports effectively the entire process of the administrative preparation, the examination procedures and its completion and statistics. Particular strengths of the system are the high technical and legal safety as well as the great flexibility. www.opp.eu.com OPP Ltd distributes throughout Europe the Myers Briggs Type Indicator©, the 16PF©, the FIRO-B™ and other world class products designed for the selection and development of talent. Its headquarters are in Oxford, UK and it has offices in Amsterdam, Copenhagen, Dublin and Paris as well as in Chicago, IL. www.pearsonvue.co.uk Pearson VUE is the global leader in computer-based testing; providing a full suite of test services, delivering exams through the world's most comprehensive and secure network of test centers. Pearson VUE is a member of Pearson plc; the FTSE 100 listed international education and media company. www.prometric.com Prometric is the recognized global leader in technology-enabled testing and assessment, providing an innovative suite of customer-inspired test development and delivery services to a diverse portfolio of 450 clients worldwide. More than nine million exams are delivered annually through a network of 10,000 testing locations across more than 160 countries. 7
Bronze
Silver
Thank You to Our 2010 Sponsors
8
LPLUS Sponsor Session:
LPLUS
Programme - Wednesday, 29 September 2010 Pearson Sponsor Session:
10:00 – 17:15 13:00 – 14:15 14:15 – 14:30 14:30 – 15:00 15:00 – 15:30 15:30 – 16:30 16:30 – 17:15
Registration Open Opening Keynote Address Sitges Ballroom I Break Sponsor Sessions Prometric Sponsor Session: Break Exhibits and Coffee Sitges Ballroom II Breakout and Discussion Sessions Exhibits and Networking Sitges Ballroom II
Welcome and Opening Keynote: 13:00 – 14:15
EPSO Development Programme and the progress that has been made in transforming the Office.
Pearson Sponsor Session:
The audience will gain an invaluable insight into EPSO's journey on the road of major transformation from a selection system that was stuck in the past and which had been described by observers as something of a 'sacred cow'. This presentation will include the many political, legal and practical challenges that have inevitably been thrown up along the way. David will also explain how he encouraged his staff to become actively engaged and involved in designing and implementing the process of change.
Meeting the Challenges of Competency-Based Assessment and Development for the European Union Institutions Room: Sitges Ballroom I EPSO is responsible for selecting staff to work for the Institutions and Agencies of the European Union including the European Parliament, the Council, the European Commission, the Court of Justice, the Court of Auditors, the Economic and Social Committee, the Committee of the Regions and the European Ombudsman. Each Institution is then able to recruit staff from among the pool of candidates selected by EPSO.
The audience will be taken through the three pillars for change laid out in the EPSO Development Programme (inspired by global best practice) namely a new HR Planning Strategy; new Testing and Professionalism methods, notably the shift from knowledge to competency-based testing; and tackling the issue of Attractiveness and Diversity which has resulted in a rebranding of the EU as a prospective employer. With the prospect of over one-third of staff retiring over the next ten years, the EPSO Development Programme aims to equip the EU Institutions with a modern and effective recruitment structure that will enable them to compete successfully in the war for talent and present a unique offering to Europe's brightest and best.
David's presentation will tell the story behind the biggest overhaul of selection procedures for EU civil servants in over 50 years. It is taking place in the context of an enlarged European Union of 27 Member States and 23 official languages. He will explain the reasons for change, how he secured a mandate for major change from a highly diverse and complex stakeholder group, how the process is being implemented and how the change was managed – internally and externally – against a backdrop of challenge and change across the public sector more generally. He will present the
Speaker: David Bearfield, Director, European Personnel Selection Office (EPSO)
9
Wednesday, 29 September 2010 ∙ 14:30 - 15:30 Sponsor Sessions: 14:30 – 15:00 LPLUS Sponsor Session:
• World-wide Language Proficiency Tests (ICAO Test Alpha-Tango) • Collaborative question design and question management (LPLUS TM-Editor) The presentation will demonstrate the size and flexibility of the LPLUS-System. It will also explain the settings of the roles & rights of the system, as well as the multiplex settings to show specific test terms and rules.
LPLUS
You Always Get What you Want: Custom-made Solutions Realized by the LPLUS-System
Participants should also be aware that the system must have a very high security level LPLUS Sponsor Session: LPLUS of operation to be accepted by governmental examination bodies and authorities.
Room: Garraf
Pearson Sponsor Session:
The session will show major projects of online testing systems which have been developed over the years by LPLUS. The presentation will show examples of different individual customer requirements and how LPLUS was able to customise the system to fulfil each of the customer requests. The main emphasis concerns the following: special question types, Prometric special examination settings, special methods Sponsor Session: of valuation and examination workflows. In addition how LPLUS developed various customer concepts of online examinations, will be made clear by showing you the following examples:
Speaker: Lutz Pleines, LPLUS GmbH
Pearson Sponsor Session:
Mobile Trends in Assessment Room: Syrah Mobile devices are rapidly changing the way in which people access content, use information Prometric and communicate. With the advancement of Sponsor Session: mobile technology, what does that mean for assessment providers and test vendors?
• Levelling mass examinations (University
of Bremen) Pearson Sponsor Session:
• Mobile Test Centre concept (University of Frankfurt/Main) • Europe wide Testing Network (EASA, Cologne)
Speaker: Michael Kayser, Pearson VUE
• Innovative question types for vocational education (DIHK GmbH, Bonn / ZWH, Düsseldorf)- Multi-Media elements, free drawing, free text, formula editor, simulations
Pearson Sponsor Exhibits and Session: Coffee Break: 15:00 – 15:30
• Nationwide online Abitur in public schools Exams (Slovakia)
Room: Sitges Ballroom II
• Using students notebooks for high secure examination (Free University of Berlin) • 7/24 availability examination server for aviation personnel (Sweden) • Self-assessment and training solutions (Lufthansa Flight Training, Bremen) 10
Wednesday, 29 September 2010 ∙ 15:30 - 16:30 Breakout and Discussion Session 1: 15:30 – 16:30
the two organisations have worked together on some exciting new end-user computer skills certification courses: Computer Wings and Spreadsheet Safe. Computer Wings are ‘next step’ computer skills in Web, Communications, Projects and IT and consists of ten stand alone modules. Spreadsheet Safe is a spreadsheets risks computer skills training and certification course and is a recognised certification standard for spreadsheet end-users who work with key or business critical spreadsheets in any industry. The presenters will share the key process steps and some of the pitfalls in preparing new programmes from the concept stage through to syllabus definition, as well as mapping and crediting to pertinent national standards. This session will also provide some insights into the end-to-end process of programme development sharing a number of illustrative samples of materials and demonstrations.
Breakout 1-1: Computer-based Testing – Regulation or Innovation? Room: Parellada Subtitle: The Regulatory Context: How an Awarding Organisation can use the Presence of a Regulator to Add Value and Bolster Quality This presentation will introduce City & Guilds, the leading organisation for vocational awards in England, placing it in the context of the relevant aspects of the English education and training system. It will then introduce the concept of regulation that profoundly influences the work of awarding organisations and the nature of qualifications. Flowing from how political trends and the increasing use of technology impacts on regulatiosa, this session will show how a high quality awarding organisation such as City & Guilds does and should use the presence of regulation to bolster its own commitment to high quality in assessment and education.
Speaker 1: Gary N. Cleere, Q-Validus Ltd. Speaker 2: Joe Wilson, Scottish Qualifications Authority
Breakout 1-2: Case Study: Detecting and Preventing Fraud in High-Stakes Testing
Speaker: Vikki Smith, City & Guilds
Room: Syrah High stakes assessment programmes are evolving their investigative and preventive capabilities to anticipate and respond to increasingly sophisticated attempts at exam fraud. On the other hand, cheaters, brain dumpers and proxy testers are also taking advantage of technology and globalization to achieve their own goals. This session will discuss the innovative steps taken by the Graduate Management Admission Council® (GMAC) and its testing services provider Pearson VUE as they identified ways to step up to this challenge. This informative look at a real-world case study will illustrate how data forensics, biometrics, data
Subtitle: Process, Patience…Lessons in Endorsing and Delivering Some New EndUser Computer Skills The Scottish Qualifications Authority (SQA) is the national accreditation and awarding body in Scotland. SQA works in partnership with schools, colleges, universities and industry to provide high quality, flexible and relevant qualifications and assessments. Q-Validus, is a leading enduser computer skills certifications programme development house, and is recognized as an innovative player in the end-user computer skills certifications space. Over the last two years, 12
Wednesday, 29 September 2010 ∙ 15:30 - 16:30 mining and information sharing were successfully used to 'crack' a series of important exam fraud cases. The presenters will also discuss their experiences working with internal and external stakeholders to ensure that investigative results lead to positive enforcement outcomes.
advantages and challenges of developing and using adaptive tests, the background of the Danish testing scheme, and the future plans. The national tests in Denmark are mandatory for pupils from 2nd to 8th grade (8 to 14 years). The scheme includes tests in reading, maths, English, biology, physics/chemistry and geography.
Speaker 1: Daniel Eyob, GMAC Speaker 2: Mark Poole, Pearson VUE
Speaker: Tue Halgreen, Ministry of Education, Denmark (Skolestyrelsen)
Breakout 1-3: Computer-based National Exams and Tests in the Netherlands and Denmark
Subtitle: Meeting the Challenges of Change toward CBT Exams Cito, the Dutch National Institute for Educational Measurement, develops the national examinations in the Netherlands for preparatory intermediate vocational education, higher general secondary education and pre university education. As an expertise centre, Cito also does research in and offers advice for modernizing national examinations. Computers play an increasingly important role in education. And outside the schools as well, students are making use of ICT and computers in a growing number of situations. The use of ICT in examinations could add surplus value in many areas. This includes the area of test content, logistics and psychometrics. Cito developed a variety of computer based national examinations for Dutch secondary education, in cooperation with the National Examination Board and the schools. Various pilot projects have been carried out in the Netherlands in recent years to gain experience with administering examinations by computer, rather than on paper. In the coming years, the number of computer-based exams will grow (in 2009 about 17% of all examinations growing to about 30% in 2011). Essential part of the introduction is the acceptance of the innovation by students, teachers and principals. This session will focus on the use of computer based testing (CBT) in the national examinations. Some results of pilot projects and examples of the use of CBT will be presented.
Room: Macabeo Subtitle: A Nationwide Adaptive Testing Scheme: The Advantages and Challenges of Developing and Using Adaptive Tests in Denmark As one of the first countries in the world, Denmark has introduced a national testing scheme based on adaptive testing. The tests are computer-based, and each pupil gets an individual test. If the pupil gives correct answers to the questions, the questions become more difficult – wrong answers lead to easier questions. The advantage is that all pupils can use the same test system. Each pupil is tested at an appropriate level of difficulty. The less skilled pupils are given a fair chance of getting as many correct answers as the more skilled pupils. Adaptive tests are more effective and provide a more precise assessment than an ordinary test. Moreover, the tests can be used several times by each pupil to monitor progress. Introducing adaptive tests is also a challenge and a considerable task. A group of specialised teachers have written approximately 10,000 items for the twelve adaptive tests. Each item has been tried out on 700 pupils and statistically analysed using the Rasch model. Moreover, teachers and pupils need to become accustom to a new way of testing. Especially the skilled pupils will have to get used to not being able to answer all questions correctly. This session will focus on the
Speaker: Nynke de Boer, Ministry of Education, Netherlands 14
Wednesday, 29 September 2010 ∙ 15:30 - 17:15 Discussion 1-4: Emerging Social and Technology Trends Impacting Talent Assessments
Discussion 1-5: Regulatory and Legal Issues Regarding Pre-Employment Selection Testing: Things you Need to Know if you are Working with an American Client
Room: Monastrell Barely a decade old, the 21st century has already been witness to dramatic social, economic and technological change: we have seen globalization and global economic crisis, the “socialization” of the Web and “mobilization” of Internet applications. What are the implications for European test publishers? This presentation will explore current trends and innovations that are impacting talent assessment and discuss the opportunities they bring to European test publishers. Discussion topics will include:
Room: Garraf Many nations have some degree of regulation governing the use of pre-employment selection tests. However, the legal environment in the United States is likely the most complex given its extensive case law and guidelines relevant to the issue. Since it provides a large potential market, this topic is of likely interest to international test publishers and consultants who are considering offering selection-related products and services in the United States. The discussion group will provide a forum for individuals to increase their understanding of these issues. Depending upon audience interests, possible discussion topics will include: US Civil Rights Laws (adverse impact, gender and race norming); Americans with Disabilities Amendment Act; Uniform Guidelines on Employee Selection Procedures; Office of Federal Contract Compliance Programs (OFCCP) ; Role of test validity and other psychometric and fairness evidence in defending assessment processes; Compliance with data protection and privacy laws (e.g., Safe Harbor, Patriot Act); Recent court cases and challenges. The facilitators will be individuals with several years of experience in both the applied and legal arenas of pre-employment assessment in the United States.
• Crowd-sourcing for cost-effective item harvesting • Delivering and managing multilingual tests • Ensuring accessibility for test takers with disabilities • Leveraging the “Fraud Triangle” to assess and mitigate threats to test security • Embedding assessments in web content and social media Speaker 1: Susan Martin, SAP Speaker 2: Eric Shepherd, Questionmark Speaker 3: Kasper Spiro, STOAS
Speaker 1: Jack W. Jones, Vangent, Inc. Speaker 2: Reid Klion, pan – A TALX Company Speaker 3: James Sharf, Sharf and Associates Speaker 4: John Weiner, PSI Services LLC
Exhibits and Networking: 16:30 – 17:15 Room: Sitges Ballroom II 15
LPLUS Sponsor Session:
LPLUS
Programme - Thursday, 30 September 2010
07:00 – 17:15 07:00 – 08:30 08:30 – 09:30 09:30 – 09:45 09:45 – 10:30 10:30 – 10:45 10:45 – 11:45 11:45 – 12:30 12:30 – 13:30 13:30 – 14:30 14:30 – 14:45 14:45 – 15:15 15:15 – 15:30 15:30 – 16:30 16:30 – 17:15
Registration Open Pearson Sponsor Session: Breakfast Verema Restaurant Keynote Address Sitges Ballroom I Break CAT meets DOG Sitges Ballroom I Break LPLUS Sponsor Session: LPLUS Breakout and Discussion Sessions Prometric Sponsor Session: Demo Competition and Exhibits Sitges Ballroom II & Parellada Lunch with Exhibitors Sitges Ballroom II CEO’s Perspective Sitges Ballroom I Break Pearson Sponsor Session: Sponsor Sessions Pearson Sponsor Session: Break Breakout and Discussion Sessions Exhibits and Networking Sitges Ballroom II
Keynote Address: 08:30 – 09:30
changing the landscape of education Prometric to one Sponsor Session: where learners are more actively engaged at every stage of the learning process.
Serious Gaming: A New Vision for e-Assessment through Immersion in the Digital Age
Speaker: Sara de Freitas, Director of Pearson Sponsor Session: Research, Serious Games Institute at the University of Coventry
Room: Sitges Ballroom I
CAT meets DOG: 09:45 – 10:30
Serious games utilise games technologies for supporting education and training that offers more immersive and engaging learning experiences. This keynote sets out a vision for future learning where games elements and different forms of assessment are built into the learning through increased immersion and feedback between the learner and the computer interface. In the future, all learning will be designed to support these more immersive experiences, but what does this really mean for future learning and e-assessment? The new vision for learning has implications not only for how learning is assessed, but also for how it will be designed and delivered. This paradigm shift in learning is providing real challenge and opportunity for assessment as a whole and
Advantages and Disadvantages of Adaptive Testing Room: Sitges Ballroom I Computer Adaptive Testing (CAT) came of age with the recent formation of the International Association for Computer Adaptive Testing. But is CAT the answer to all testing needs and does CAT truly deliver all that proponents enthuse about? Two senior figures in the testing industry debate the value of CAT. Larry Rudner will act as the advocate for CAT drawing on his long experience in developing and deploying CAT 17
Thursday, 30 September 2010 ∙ 09:45 - 11:45 Breakout and Discussion Session 2: 10:45 – 11:45
solutions. Eugene Burke will act as the contrarian putting forward the argument that CAT suffers from a number of significant limitations which can be addressed by alternative models. Each will speak for and against CAT with the opportunity for rebuttal before engaging the audience to share their views of the arguments put forward. Are you a proponent for CAT or a sceptic?
Breakout 2-1: Occupational Knowledge and Skills Testing in Vocational Education and Training: Policy and Practice in the United States of America
This session will offer you the chance to test your views against the arguments of two experienced designers and programme managers from the educational and employment sectors. Come and find out whether the CAT prevails or whether the DOG emerges as the victor?
Room: Syrah The assessment of occupational knowledge and skills in vocational education and training (VET) has received enormous attention in United States over the last few years. In the past, measurement of VET programs focused primarily on educational outputs such as the number and types of students served, number graduating, and number of graduates obtaining employment. The most recent U.S. legislation for VET, known as Perkins IV, brought specific attention to measuring educational outcomes. Specifically, as a condition for receipt of federal funding, states are required to assess students’ attainment of technical skill proficiencies aligned with industry-recognized standards. Although a number have created statewide exams or have purchased third-party assessments, many states lack standardized testing systems and the resources to create or purchase them. This has led to exploration of alternative approaches to assessing VET technical knowledge and skills. Topics covered in this presentation include the impact of federal and state testing policies, issues and obstacles encountered, and approaches to these issues. Participants will interact through polling via clicker technology, comparing and contrasting experiences with VET testing in their respective countries in real-time.
Larry Rudner is Vice President for Research & Development for the Graduate Management Admissions Council where he leads the team of psychometricians, statisticians, and survey experts that oversees the development of all aspects of the Graduate Management Admissions Test. Larry received the Association of Test Publishers Career Achievement Award in 2009. Eugene Burke heads Science and Innovation for SHL Group Ltd. and is the current Chair of the European Association of Test Publishers, and past Chair of the British Psychological Society’s Steering Committee for Test Standards and past Chair of the BPS Division of Occupational Psychology, as well as a past Council Member of the International Test Commission. Speaker 1: Eugene Burke, SHL Group Ltd. Speaker 2: Lawrence Rudner, Graduate Management Admission Council
Speaker 1: John Foster, National Occupational Competency Testing Institute Speaker 2: Robert Mahlman, The Ohio State University, Center of Education and Training for Employment 18
Thursday, 30 September 2010 ∙ 10:45 - 11:45 Breakout 2-2: How to Measure Skills, Ability and Talent!
being transformed by technology, and this transformation requires new skills to succeed in the new information age. Given this, it will be argued, new ways of assessing such skills will be required. Despite the ever increasing adoption of e-assessment many of our computer-based examinations consist of computer marked Multiple Choice Question’s little different from their paper-and-pencil forebears of the previous century. This paper proposes that whilst MCQ’s still have a role to play in current examinations their utility in measuring complex, cognitive skills is limited and there is a need to move toward more innovative item types. Such item types involve interactivity, allowing a candidate realistic ways to interact with the question, alongside rich multi-media content which provides realistic, stimulating content for the candidate. A number of examples of such innovative items will be presented to show how such items can allow us to measure complex, cognitive skills in a way which would be either very difficult or impossible with the use of MCQ’s.
Room: Garnacha Subtitle: O*Net Anchors Validity Generalization Defense of Employment Ability Testing Validity generalization (VG) is industrial psychology’s science of the general case: reliable measures of cognitive ability are valid predictors of both training performance and job performance for virtually all jobs. Assessing cognitive ability triggers a prima facie charge of employment discrimination in the U.S., however, because of average group differences on such measures. The legal rebuttal is to show that the measure of cognitive ability is “job related for the position in question and consistent with business necessity. Validity can be shown under current professional standards (without the need for a local validity study) based on meta-analytic validity generalization research anchored using O*NET research to document verbal, quantitative and technical / problem solving abilities (i.e. measuring General Mental Ability or GMA) required on the job. The cumulative knowledge of VG, the legal context, and the evolution of professional standards supporting VG will all be discussed. Building on O*NET taxonomies, examples using published employment tests and certification exams will be illustrated and discussed by the former Chief Psychologist at the U.S. Equal Employment Opportunity Commission.
Speaker: Belinda Brunner, Pearson VUE
Breakout 2-3: Get Serious! Serious Games as Beneficial Add-ons to the Testing Process Room: Garraf The testing industry needs serious games to stay relevant. Why? Because today’s pupils are drastically different from the pupils that existed when the majority of our current methodology was developed. How are they different? Multi-tasking is their second nature, constant interaction with fellow students a given and knowledge is a readily accessible commodity. These developments, in concert with the technological breakthroughs of recent years, open up great possibilities for our industry to reinvent itself and remain relevant. This session will present a case based on the
Speaker: Jim Sharf, Employment Risk Advisors, Inc.
Subtitle: Innovative Items – Measuring 21st Century Skills A fundamental shift is occurring in the nature of the skills required in the 21st century. Both educational and work environments are 20
Thursday, 30 September 2010 ∙ 10:45 - 11:45 Squla learning platform developed by Futurewhiz in a strategic partnership with Cito, and will demonstrate how the application of technology from MMO (Massive Multiplayer Online) games and social networking offers a highly motivational and effective low-stakes testing platform with considerable potential for developing into a highstakes platform. Squla was tested extensively during 2009 and has now been released for grade 7 (ages 10-11) pupils. In September 2010 it was expanded to cover grades 5 to 8 (ages 8-12).
an open forum through which the audience will be encouraged to interact through questions, comments and observations. Speaker 1: Eugene Burke, SHL Group Ltd. Speaker 2: Achim Preuss, Cut-e Speaker 3: Geoff Trickey, PCL
Discussion 2-5: The Role of Assessment in Improving Performance in Customer Service Roles
Speaker 1: Sara de Freitas, Serious Gaming Institute at the University of Coventry Speaker 2: AndrĂŠ Haardt, Futurewhiz
Room: Monastrell The recent marketing literature states that a good customer experience will result in eight personal recommendations to other possible customers, while a poor customer experience will result in twenty-two recommendations not to purchase a good or service. In these difficult economic times, the customer experience is a critical factor in public and private sector strategies to maintain and build brand value, and a critical factor in the success of these strategies will be the acquisition of the right talent to staff customer service roles. This session explores the work of two major test publishers in developing assessment solutions to meet this need. Cary Vaughan will describe how a simple theory of the psychology of customer service behaviours has been translated into an efficient self-report questionnaire as well as more bespoke solutions for clients across eight countries and seven languages to date. Claire Fix will describe the recent development of a generic assessment tool designed to identify fit to customer facing roles. She will also share the design process supporting the tool as well as the evidence showing the value it offers clients seeking to improve customer service quality. Both presenters will provide the audience with opportunities to experience the assessment tools described through short demonstrations.
Discussion 2-4: Innovations in Personality Assessment Room: Sitges Ballroom I This session looks at recent developments in the application of personality assessment in the acquisition and development of talent. The three speakers are experienced practitioners who are also involved in the development of assessment solutions for client organisations. The session will cover three themes: pressure to improve the candidate experience driving the need for more efficient assessments; addressing the issue of faking good in high-stakes personality testing; how the use of personality assessment can address the risk of managerial failure. Achim Preuss will demonstrate an adaptive technology for the online administration of personality testing that is a response to the need for efficient but valid assessments. Eugene Burke will describe a recent innovation in personality assessment that brings together improved precision in measurement with forced choice formats to control for faking in personality scores. Geoff Trickey will describe applications of the Hogan Diagnostic Survey (HDS) to identifying and addressing potential career stallers and derailers among manager and executive candidates. The session will begin with short presentations by each speaker leading to
Speaker 1: Paul Smith, Pearson Assessment Speaker 2: Carly Vaughan, SHL Group Ltd. 21
Thursday, 30 September 2010 ∙ 10:45 - 12:30 Discussion 2-6: Can Technology help us Stop Exam Cheating?
solutions for selling tests to individuals and organizations; Centralized management and controlled access to tests and results; Tools for creating, delivering and managing multilingual items and tests; Accessibility options to accommodate test takers with disabilities; and Delivery to more people throughout the world via high-speed, low-speed and intermittently connected electronic devices.
Room: Macabeo Two of the greatest challenges for delivering exams globally are security and costs. This discussion forum helps the audience to understand what the current options are for deploying technology to help reduce the risk of cheating and the exposure of exam questions.
Speaker: Bart Hendrickx, Questionmark
Speaker 1: William Dorman, Kryterion Speaker 2: Ruben Garcia, Innovative Exams Speaker 3: Mark Poole, Pearson VUE
Demo 2: Taking a Fresh Approach to a 360 Degree and Multi-Rater Feedback: Introduction of Innovative new “Forced-Choice” Assessment Format
Demo Competition and Exhibits: 11:45 – 12:30
In this presentation, the speakers will discuss the advantages and the suitability of the different approaches to 360 / Multi-Rater Feedback, explore the different situations in which 360 / Multi-Rater assessments can be used to support development planning, and provide a demonstration of a forced-choice 360 degree assessment together with examples of the accompanying individual feedback reports and development guides, complete with leveraging tips and development highlights.
Join us for the demo competition and vote on the most innovative and industry changing demos shown. More details will be provided as the doors open to the competition. Don’t miss it! Room: Parellada
Demo 1: New Technologies for Test Publishing in a Flatter World Test publishers will thrive in a flatter world by making tests readily accessible to an increasingly mobile and diverse workforce, thereby broadening their reach and maximizing business opportunities. This session will show how to utilize today’s cost-effective means for global delivery and real-time reporting while maintaining firm control over test management, content, quality and security. It will show how six new Questionmark technologies can help test publishers succeed in a super-connected environment: Easy-to use Web-based authoring for Subject Matter Experts, enabling the collaborative creation of items and tests; Online
Speaker 1: Robert Feltham, Cubiks Group Limited Speaker 2: Dimmy van Zanten, Cubiks Group Limited
Demo 3: MBTI Thinkbox While traditional instructor-led training continues to provide the vast majority of corporate training hours, research shows that learning on-demand has become a user-requirement for any organisation. However, the systems and processes required to put learning on-demand into action are just emerging. In 23
Thursday, 30 September 2010 ∙ 11:45 - 12:30 a learning on-demand solution, courses, references, help files, documents, webcasts, audios, videos, book references and presentations are all made available - just as a worker needs them. Formal training and performance support are made available in an integrated environment. The learner decides which to use, enabling selfdirected learning. MBTI ThinkBox is an exciting online development tool offering a world of ondemand learning resources including expert tips, eCoaching, short videos, book summaries and more - all based on a personality assessment. The tool provides a sustainable learning environment that can improve teamwork, develop leaders and increase employee engagement.
EDUMATIC lets you export your exercises and tests as SCORM packages, that can be imported in any SCORM compatible LMS. Speaker: Bert Wylin, Televic Education
Demo 5: Marketing Your Test Security/ Integrity Message Effective stakeholder communication strategies are key to ensuring that test takers understand their responsibilities and test users maintain their confidence in your test's validity. You can also use security and test integrity as a differentiator for your program or brand. The question is: How to grab your stakeholders' attention long enough to deliver the message? This demonstration will highlight key security messages you may wish to communicate, discuss the pros and cons of different approaches, and illustrate some innovative methods utilizing both traditional and new media that are being used by testing programmes to successfully communicate with their intended audiences.
Speaker 1: Robert McHenry, IPAT Speaker 2: Penny Moyle, OPP Ltd.
Demo 4: Edumatic Exam: Online Assessments and Tests Edumatic Exam is a full online e-assessment environment. Create, publish and report. More than 25 question types, all with various feedback options, simple or complex scoring mechanisms and everything you need to conduct secure online tests and exams. Edumatic Exam is highly scalable and can be used by thousands of users synchronously. Edumatic Exam can be used as a training tool, as an online tool for assessments and placement tests. Edumatic can be used independently but it is also developed to connect easily with your existing (administrative or e-learning) platforms to facilitate the user and session management. It has a user friendly, graphical user interface, making it usable by everyone in your organization, both for creating exams and for viewing reports and results. Tracking and logging is very extensive and gives both general and very specific, detailed reports (that you can export to Excel). Media files (images, audio, video, animations) are easy through the Media Library. The EDUMATIC Exam technology supports open and half-open items. Content created within EDUMATIC can be distributed online and offline.
Speaker: Mark Poole, Pearson VUE
Demo 6: Optimising Knowledge Gain - Minimising Resources - a Pioneering Concept, Based on Job Role and Competency Requirements Many organisations make the mistake of testing and training the entire workforce on all aspects, instead of focusing on individual’s specific role and competency requirements. This incurs unnecessary cost and wasted time. We will present a pioneering concept combining best practise in assessment and e-learning in a single solution. By integrating role-specific diagnostic tests with tailored e-learning courses, staff need only undertake training on individual 24
Thursday, 30 September 2010 ∙ 11:45 - 12:30 Demo 8: Using Voice Recognition Technology to ID Candidates for Remote, Proctor-less Test Administrators
knowledge gaps. This generates significant cost and time savings and delivers a more efficient and effective workforce. We will explain how the concept was successfully applied to a case involving a system migration from Microsoft Office 2003 to 2007. We will demonstrate how the client used our new “In-Application” question format for the interactive testing of computer skills in an authentic environment. Furthermore, the demonstration will show how the organisation’s specific role needs and competence requirements for Office 2007 were used as a basis for targeted training programs.
Technology will be instrumental in providing innovative and reliable assessment solutions for remote, proctor-less testing of candidates. Corporations and government agencies have the need to quickly and cost effectively evaluate candidates who are not located near a corporate office or test center. Language Testing International is utilizing voice recognition software as a method to confirm the identification of a test candidate doing a remote telephone or computer delivered speaking proficiency test. Through a short, simple “voice print” procedure administered by the recruiter or HR personnel, LTI maintains in its secure archive a voice sample that can be used to confirm the identity of a candidate taking a remote telephone or computer-delivered speaking proficiency test anywhere in the world at any time in 60+ languages. LTI Vice President, Helen Hamlyn, will demonstrate a procedure used by LTI clients to register voice samples of potential test candidates. Ms. Hamlyn will also demonstrate how the registered voice sample is utilized at the time of administering a speaking proficiency test to verify the identification of the remote test candidate.
Speaker 1: Lena Bieblad, EnlightKS Speaker 2: Anna Zolland, EnlightKS
Demo 7: Latest Developments in Question Design Behind the European market leading position of the LPLUS-System® in aviation sector more and more of the major universities in Germany use the LPLUS-System® for their online examinations. Flexibility and Security are the distinguishing marks of the LPLUS-System®. The hands-onworkshop shows the latest development of sophisticated question design such as drawings in eExamination, the use of a formula editor and simulations. Even the classical questions types like MC, Fill In and drag & drop question types of the LPLUS-System® offer new opportunities for authors. Participants of the workshop learn how easy it is to create questions, to set up examinations and to run examinations. In addition, the participants learn something about the security features of LPLUS-System® and how to handle mass assessment while keeping quality (as used by the University of Bremen).
Speaker: Robert I. Katz, Language Testing International
Demo 9: When Chance is Not Acceptable: Using the Adjective Check List to Facilitate Career Selection, Development and Performance Career theorists recognize the importance of personality in career choice, competence, and satisfaction. BeMIS (Behavioral Management Information System) is a powerful tool that identifies relevant character and personality attributes and
Speaker: Lutz Pleines, LPLUS GmbH
25
Thursday, 30 September 2010 ∙ 11:45 - 14:30 Demo 10: Expanding Your Test Delivery Channel through Innovative Secure Technology
assists in developing career facilitating behavior and selection for academic and other purposes. Easy to use, BeMIS provides personalized reports and development plans to foster career success. Attendees will be shown a live demonstration of BeMIS, with a specific emphasis on how BeMIS may be utilized to facilitate services for effective career selection, development, and performance.
There are various ways to securely deliver tests. However, until now delivery options have been limited to unsecured low-stake or highstake test center delivery models. New delivery models allow you to extend the distribution of your existing bricks-and-mortar testing center channel to new locations. The distribution of high-stake exams has always been limited to the channel of a bricks-and-mortar testing service provider. Many candidates are under-served because of the lack of testing coverage and the costs that are associated with taking a test at a traditional bricks-and-mortar test center. Although this model has been the only option to delivery high-stake exams, it can also limit your program's distribution. This session will demonstrate how sponsors can expand their coverage, while reducing their costs and maintaining security by using a new delivery method. This session will also address the ways in which innovative technology will assure candidate identification and content protection while conveniently taking the test closer to your test candidate.
The focus of this session is on knowledge based decision making. The selection of an academic major, a career, students admitted to an academic program, personnel for employment, the focus of counseling, or any other decision in which personality traits are influential are too important to be made without the availability and consideration of critical information BeMIS administers, scores, and interprets the ACL. Data from the web administration or the BeMIS PC administration of the ACL is scored and data-based in less than a second. Reports ranging from a single graphical and/or statistical profile to an interpreted 20-30+ page report may be produced in just a few seconds. BeMIS reduces hours of tedious effort to seconds of button presses and mouse clicks. Further, BeMIS provides a virtually unlimited range of statistical, graphical, and narrative reporting options. The focus of a report may be an individual, couple/relationship, 360 degree analysis or a group and a Target (a specified pattern of behavior) may be included for comparative analysis and/or selection purposes.
Speaker 1: Ruben A. Garcia, Innovative Exams LLC Speaker 2: Kate O’Connor, Innovative Exams, LLC
BeMIS expands the data provided by the ACL with: clinical implications from the measurement of the DSM IV-TR personality disorders; Character Strengths and Virtues (CSV); and Emotional Intelligence (EQ). Plus BeMIS is an open system in that the user may create new scales and charts, and reports, as well as modifying the default reports. In summary, the capability and application of BeMIS is just like that of the ACL: It is limited by the need and creativity of the user.
Lunch with Exhibitors: 12:30 – 13:30 Room: Sitges Ballroom II
Speaker: Daniel Collins, Measurement and Planned Development, Inc. 27
Pearson Sponsor Session:
Thursday, 30 September 2010 ∙ 13:30 - 14:30 CEO’s Perspective: 13:30 – 14:30 Room: Sitges I Following on from the success of last year’s CEO panel, we have engaged CEO’s and Chairs of leading companies in the testing and assessment industry to share their view and visions for where they see the industry going. Times have changed dramatically in the global economy and particularly in Europe where there has been a significant shift in the policy of governments as well as dramatic changes to the dynamic of labour markets. In line with the main theme of the conference, the panel will be invited to comment on the how they see these changes impacting the industry both in terms of the challenges they pose but also in the opportunities that they offer. A key factor in the future of testing will be, as evidenced in the keynote session on serious gaming, the role of technology. But how are companies looking at the potential value that technologies offer and where might we see the applications of new technologies emerge in the next few years? These are a few of the topics that will be discussed and debated in this session which will give you the chance to put our CEO’s to the test. Panel:
Bob Gomersall, Chairman, BTL William G. Harris, CEO, Association of Test Publishers Robert McHenry, CEO and Chairman, Oxford Psychologists Press Madan Padaki, Co-Founder and CEO, MeritTrac Eric Shepherd, CEO, Questionmark
Moderator: Eugene Burke, Science and Innovation Director, SHL Group Ltd.
Fiesta Join Prometric on Thursday night for an evening of food, cocktails, music, and dancing in the seaside village of Sitges.
29
Thursday, 30Session: September 2010 ∙ 14:45 - 16:30 LPLUS Sponsor LPLUS
Sponsor Sessions: 14:45 – 15:15
of today’s most successful testing programs are shaping our future, one new innovation at a time. The theoretical and the practical are coming together to address the new challenges faced by our industry, including; transitioning from knowledge and recall-based exams to performance-based and higher-order cognitive skills assessments, effectively addressing fairness, validity and security in an environment of instantaneous communications sometimes conflicting cultural norms and finally, leveraging new technologies to drive speed, efficiency and scale into the development and delivery of world-class assessments.
LPLUS PearsonSponsor SponsorSession: Session:LPLUS
Assessment Data – How to Create Meaningful Feedback Room: Syrah An overview of how assessment data can be Pearson Sponsor Session: Prometric utilized effectively for the benefit of organizations Sponsor Session: and candidates.
Speaker: Steve Williams, Prometric
Speaker: Betty A. Bergstrom, Pearson VUE
Breakout and Discussion Session 3: 15:30 – 16:30
Prometric Pearson Sponsor Session: Sponsor Session:
The Evolution of Computer-Based Testing: New Opportunities, New Challenges
Breakout 3-1: The Global Career Development Facilitator (GCDF): A Global Initiative to Assist People with their Career Goals
Room: Garraf
Room: Macabeo
The art and science of traditional testing methods are very well established. After more than a century and a half of experience in the development and delivery of multiple choice exams and hundreds of years of essay-based assessments our industry is well-equipped to maintain the status quo. But are the methodologies of the past sufficient to meet the needs of an increasingly complex and technologically advanced 21st century global society?
The GCDF and its country specific variants are helpful in identifying which career development facilitation providers have demonstrated command of core skills to help persons make informed career choices. In light of world economies changing because of globalization, competent career counseling-related services have become even more critical. GCDF certification mechanisms help governments, professional associations, and consumers feel assured their career development facilitation professionals are well trained and experienced. The GCDF program has been adapted for use across a wide range of countries in Asia, Europe and North America. In the next years,
Pearson Sponsor Session:
This session will explore the boundaries of the future by analyzing emerging trends in the field of assessment and illustrating how some 30
Thursday, 30 September 2010 ∙ 15:30 - 16:30 Subtitle: Advancing Psychometrics to Deliver Adaptive Testing Speaker: Theo Eggen, Cito
the demand for career counseling services will be higher and higher, as the traditional idea of career will dramatically change, placing more responsibility on the individuals to manage their own career path. The first career counseling certification in Europe is presented, from the historical aspects of the project to the administrative educational details, in accordance with the initial American certification standards. In the European and global labor market, the GCDF program could be one solution for offering clients high quality career services, in a very dynamic world of work, which encourages people to bring value to their workplace and to sustain a constant self-development effort, throughout their life.
Discussion 3-3: Assessment Workflow: Processes and Technologies for Managing Assessments Room: Garraf The impact of technology on testing and assessment is perhaps most clearly evidenced in the development of assessment workflows that manage critical steps from configuring an assessment project, to assigning candidates to assessments, through data management, scoring and profiling, to the distribution of reports to users and candidates themselves. This session brings together three experts in this field from companies that have led the use of assessment workflows in both Europe and the US. Achim Preuss will describe applications through which client requirements are captures and used to configure solutions to recruitment and selection needs, including novel displays for ability and personality tests. Geoff Trickey will describe an application of assessment workflow to the area of personal development including the distribution of development planners driven by the results of assessments aligned with client talent needs. Reid Klion will describe how Pan, one of the major providers of online assessment tools in the US, approaches the process flows required to execute a test/assessment project. All three contributors bring a wealth of experience from major private and public sector clients, and the session will include brief demonstrations of applications in use with client organisations.
Speaker: Andreea Szilagyi, EBCC
Breakout 3-2: The Evolution of Adaptive Testing: Advancing Psychometrics & Technology Room: Garnacha Building on a wide range of experience in developing and deploying Computer Adaptive Testing (CAT) solutions from early assessments of mathematics through to teaching skills, this session will being by providing an overview of the different types of algorithms and configurations developed for these distinct scenarios such as in developing a test for signaling dyslexia with youngsters. In the second part of this interactive session, the challenges and considerations of implementing CATs will be addressed by turning to questions such as how do we cope with advancing technology and how can we use it to our advantage?
Speaker 1: Reid Klion, pan – A TALX Company Speaker 2: Achim Preuss, Cut-e Speaker 3: Geoff Trickey, PCL
Subtitle: Advancing Technology to Deliver Adaptive Testing Speaker: Mark Molenaar, Cito
31
Thursday, 30 September 2010 ∙ 15:30 - 17:15 Discussion 3-4: Taking a Global View of Talent Assessment
Discussion 3-5: The Future of the Employment Testing Industry
Room: Syrah
Room: Monastrell
This session brings together an experienced panel across the testing industry to share what they see as trends in the assessment of talent in Asia, Europe, India and the US, as well as the increasingly global nature of talent assessment. The speakers will share their experiences in how assessment can be effectively deployed to address organisation’s talent needs, as well as how communication and networking can be used effectively to raise awareness of the value of assessment at national and international levels. The style of the session will be highly interactive with each participant sharing their views on the opportunities for the testing industry, the challenges they see in getting the value message across, as well as where they see the next big opportunities locally and globally. The audience will be encouraged to engage with the panel and to share their own views on how the EATP’s conference theme, Growing Talent: Gaining The Advantage Through Assessment, can be turned into a sustainable reality.
There are many pressures and dangers facing the traditional employment testing industry and, like the recorded music industry, it may have to adopt a new business model in order to survive. Examples of threats range from the increasing availability of free tests to the fact that most of the tests that are currently popular do not take advantage of the possibilities of the electronic age. The speakers will review some of the recent innovations in employment testing, sketch some future scenarios for the construction and delivery of tests and suggest a business model for test publishing in the future. Speaker 1: Robert McHenry, IPAT Speaker 2: Penny Moyle, Opp Ltd.
Exhibits and Networking: 16:30 – 17:15 Room: Sitges Ballroom II
Speaker 1: Eugene Burke, SHL Group Ltd. Speaker 2: Madan Padaki, MeritTrac Speaker 3: Nadine Schuchart, Hogrefe Speaker 4: Alex Tong, ATA Inc.
32
LPLUS Sponsor Session:
Pearson Sponsor Session:
Programme - Friday, 1 October 2010
07:00 – 13:00 07:00 – 09:00 09:00 – 10:00 10:00 – 10:15 10:15 – 11:15 11:15 – 11:30 11:30 – 12:30 12:40 – 13:00
Prometric
Registration Open Sponsor Session: Breakfast Verema Restaurant Closing Keynote Sitges Ballroom I Break Breakout and Discussion Sessions Pearson Sponsor Session: Break Breakout and Discussion Sessions Final Consideration Sitges Ballroom I
Closing Keynote: 09:00 – 10:00
to reduce the proportion of failing students. Sample-based national assessments as well as state-wide-assessments, covering the whole population of students, on the other side, are necessary measures to monitor whether school reforms and professional development really improve learning in school. Side effects of such an assessment system, e.g., gain score inflation, will be addressed.
Meeting the Challenges of Change in Assessment Systems - How can we link Educational Standards, Educational Assessment, and Instruction? Room: Sitges Ballroom I
Speaker: Olaf Köller, Leibniz Institute for Science and Math Education, Kiel, Germany
Many western countries suffer from large proportions of poor achieving students, even at the end of secondary school. These students often fail in the school-to-work transition and form a risk group for a modern society. Therefore, educational (performance) standards have been developed in several countries to point out common goals for several subjects. These standards contain domain-specific competencies (skills) that students should reach in primary and secondary school. In the case of minimum standards it is expected that more or less 100 percent of an age cohort meet the standards. Such a system of educational standards, however, has two implications, one for educational assessment and one for improving quantity and quality of instruction. The presentation will thus provide some ideas on how to combine standards, assessment, and instruction. It will strongly be argued, that improving teaching by means of professional development of teachers is the key strategy
Breakout and Discussion Session 4: 10:15 – 11:15 Breakout 4-1: Current Issues in Personality Assessments Room: Parellada Subtitle: Faking and Personality Assessment: Where We’ve Been and Where We’re Headed? The concept of “faking” on self-report personality and psychological inventories has occupied psychologists for at least half a century As Bob Hogan has pointed out, this topic has “consumed the best minds of the discipline” yet little substantive progress has been made in resolving this complex and hotly debated issue 34
Friday, 1 October 2010 ∙ 10:15 - 11:15 Breakout 4-2: The PISA Shock! Germany Seeking for Assessment Improvement: Germany’s Challenges to Follow the Progress of Pupils
with some commentators arguing that faking seriously compromises the validity of personality tests in selection contexts. This session will consider how we define faking behaviour; what is the current thinking on solutions to reduce faking; does perspective (applicant versus employer) play a role in how we label and address this phenomenon; and what is the role of personality assessment in employee selection?
Room: Macabeo In this session Dr. Tobias Stubbe will first give a short introduction about the National Assessment System in Germany. Given this background we will present some large scale assessments like Pisa, Pirls and the Comparison among federal states. We will discuss the different functions of these assessments; how German schools appreciate them, and will focus especially on how the schools handle data-feedback resulting from these student assessments. Furthermore we will consider the advantages and disadvantages of different student achievement measurement systems, and discuss which supporting framework and procedures are needed to enable teachers to use the data provided to enhance their classroom instruction in a constructive and reasonable manner.
Speaker 1: Julie Carswell, Sigma Assessment Systems Speaker 2: Reid Klion, pan – A TALX Company
Subtitle: Making Adaptations of Personality Assessment and Carrying Out Validation Studies in European Countries Showing that assessments have been skillfully adapted to and can predict behaviours, including job performance in another country, is a key requirement for test publishers. Many global test publishers have accumulated substantial evidence of their assessments’ ability to assess accurately in the source country (e.g. US, UK), but evidence of their assessments’ suitability and efficacy in other countries is often sparse at best and hard to obtain. In this session, the speakers will demonstrate how global test brands like the 16PF, Myers Briggs Type Indicator, and FIRO-B have been sensitively translated, adapted and validated in different European countries and share their experiences in confronting and overcoming some unexpected challenges they faced when conducting psychometric research in these countries.
Relative to this, Tom Duindam will go into the introduction of the Cito-Pupil-Monitoring-System in Germany since 2008. The development and adaptation of the Cito-Pupil-MonitoringSystem was carried out in schools for secondary education in Essen and Duisburg (North-Rhine Westphalia), as a partnership between Cito and the “Institut für Schulentwicklungsforschung”, TU, Dortmund. The Cito-Pupil-Monitoring-System, designed for grades 5 and 6 for all types of secondary education, with tests for Mathematics and Reading Comprehension, will be further explained. The results of the students that were first measured, (twice in grade 5 and twice in grade 6), will be discussed, as well as the question of how this instrument is perceived by schools in North-Rhine Westphalia.
Speaker 1: Robert McHenry, IPAT Speaker 2: Penny Moyle, Opp Ltd.
In addition, a comparison will be made between the Cito-Pupil-Monitoring-System and the 35
Friday, 1 October 2010 ∙ 10:15 - 11:15 assessments of the National Assessment System in Germany. Finally some strategies will be considered for the implementation of the Cito-Pupil-Monitoring-System in an educational environment that is not yet accustomed to such a system.
language learning and language testing, in line with the Common European Framework for Languages (CEFL). According to a more modern view on language acquisition and language skills, new tasks were created for testing people’s communicative language skills (reading and listening comprehension). In line with the CEFL, more items were created for functional and communicative testing, based on a description of the professional domain in which the testee is going to work. The introduction of smart (technological) question types allowed for the use of authentic language materials (multimedia) in the exam. Finally, a new online preparatory learning environment has been created, including thousands of exercises and simulation exams, with tracking and logging of people’s progress and results, thus allowing a truly useful training before the exam. This presentation’s focus will be the close interaction between technology (the Edumatic framework), psychometric methodology and (language) didactics.
Speaker 1: Tom Duindam, Cito Speaker 2: Tobias C. Stubbe, Institute für Schulentwicklungsforschung
Breakout 4-3: Language Testing Room: Monastrell Subtitle: Dial “A” for Assessment This presentation will explore the role that mobile technologies can play to support assessment of spoken language. The work will draw on the experience of a large scale project in the field and outline future potential area for study. The focus of the study will be the assessment speaking and listening components of the language curriculum. Experience has shown that these are the domains that often prove most challenging to assess at large scale in an effective, efficient and reliable manner. The study when presented will be supported by a practical demonstration of the technology involved.
Speaker: Bert Wylin, Televic Education
Discussion 4-4: Best Practice in Test Security Room: Syrah Test Security remains a critical issue for high stakes testing and assessment across all applications from clinical, to educational, employment and certification testing. This session builds on the experience of three leading experts in the field of test security who will talk to experience of what works in practice as well as innovations in strengthening test security. John Fremer will describe how test publishers have collaborated with the State Assessment Program Managers to develop a suite of best practices that deal with security planning, training in test security, responding to security breaches as well as protecting test materials at all stages from distribution, storage and return.
Speaker: Patrick Craven, Cambridge Assessment
Subtitle: E-Assessment for the Belgian National Language Exam This paper reports on a scientific screening, adaptation and extension of the national language exam for federal employees in Belgium. The complete overhaul of the exam can be seen as a model for state-of-the-art integration of new technology, sound methodology and the introduction of a contemporary vision on 37
Friday, 1 October 2010 ∙ 10:15 - 12:30 Eric Shepherd has recently proposed a security framework via his popular blog that draws on the fraud triangle and focuses on three threats to assessment security: impersonation, content theft and cheating. David Foster will describe how a combination technology and online proctors (remote proctoring) has enabled high stakes testing to be deployed to the homes, schools and workplaces of test-takers, and will share evidence gathered from over 10,000 test-takers. The session will ask each of the speakers to provide a short summary of what they consider effective best practice as a prelude into a discussion and the involvement of attendees through question, answer and comment to engage with this very experienced panel of experts.
Michael will demo the SJT application and present findings from a survey of candidates who sat the SJT. Wendy Lord will explore the application of SJTs to the fit between style of leadership and demands of a situation, and will describe the suitability of the SJT format for assessing this important aspect of working with and through people. The session will begin with by each speaker explaining a client issue, how SJTs addressed that issue, and the evidence of the value delivered to the client. The session will end with an opportunity for attendees to interact with the presenters through questions and comments. Speaker 1: Wendy Lord, Hogrefe Speaker 2: Michael Tack, Adforum Speaker 3: Carly Vaughan, SHL Group Ltd.
Speaker 1: William Dorman, Kryterion Speaker 2: John Fremer, Caveon Test Security Speaker 3: Sue Martin, SAP Speaker 4: Eric Shepherd, Questionmark
Breakout and Discussion Session 5: 11:30 – 12:30
Discussion 4-5: Employment Testing and Innovations in Situational Judgement Tests
Breakout 5-1: Developing Talent through Integrated Learning and Assessment Programmes
Room: Garraf
Room: Parellada
SJTs present candidates with possible actions to resolve the issues involved in a situation. This assessment type has rapidly grown in popularity given its high face validity, and this session provide the opportunity to catch leading edge applications with this still relatively young approach to assessment. Carly Vaughan will answer the question what do SJTs measure by proposing a construct driven approach using a behavioural framework. She will present findings from a client project that resulted in a £1.2 million saving in staff recruitment. Michael Tack will describe a project undertaken for a public sector client where issues of fairness, employer branding and a positive candidate experience were critical.
Enabling individuals to access learning and testing when they require it and when it is convenient to them, are key features of both distance learning and CBT. And yet so often these two inextricably linked activities have been developed in isolation. This presentation explores the benefits for both the candidates and the awarding body, of developing learning programmes alongside high stakes assessment programmes, and how candidates can be provided with easy access to the learning and assessment. Speaker 1: Marc Carpenter, Pearson VUE Speaker 2: Nick Smith, Lloyd’s of London 38
Friday, 1 October 2010 ∙ 11:30 - 12:30 Breakout 5-2: Delivering High-Stakes Assessments in the Medical Sector Across the EU
derailment based on personality data and multisource performance ratings collected from 250 Romanian leaders from a range of industries. The data will be discussed in the context of an innovative conceptual model of leader derailment and work directed toward validating two new related measures: a personalitybased assessment of risk for derailment and a multi-source assessment of derailmentrelated behaviors. In the question and answer period, topics for discussion will include: What is leader derailment? How should derailment be measured? What behaviors contribute to derailment? What are the personality predictors of derailment? Should other factors be considered? How are models of effective and ineffective leadership related? Is derailment simply the inverse of effective leadership? What is the generalizability of the model and results to North American samples?
Room: Macabeo The European Board of Orthopeadics and Traumatology (EBOT) is one specialist section of UEMS (Union Européenne des Médecins Spécialistes) which represents national associations of medical specialists in the EU and other European countries with 37 specialist sections. The UEMS promotes the free movement of European specialists within the EU and sets various standards and policies. In this regard, EBOT is charged with setting and maintaining standards for the specialty through their fellowship examination. The challenge for EBOT was to devise and deliver a fair and valid examination programme accessible throughout the European Union for prospective candidates. The presentation will explore the challenges faced and solutions chosen by EBOT and participants will be given the opportunity to discuss the programme in the context of maintaining medical standards while at the same time promoting the free movement of talent across European borders.
Speaker: Julie Carswell, Sigma Assessment Systems
Discussion 5-4: Addressing Counter Productive Behaviours in Organisations: The Case of Managers and Leaders
Speaker 1: Ivan Coutinho, Pearson VUE Speaker 2: Jorge Mineiro, EBOT
Room: Syrah Recent years have seen heavy publicity on the failure of senior managers, executives and leaders, specifically in terms of unethical behaviours and the inability to manage and mitigate significant risks of organisational and wider events. This session brings together three experienced practitioners in the field of employment and organisational behaviour to discuss how assessment tools can help organisations take stock of the people risks among their senior teams, and address those risks. Eugene Burke will describe the results of an international survey of managers and
Breakout 5-3: Understanding and Measuring Leadership Derailment Room: Monastrell Leader derailment is common and costly, with an estimated 30% to 50% of executives failing resulting in costs that can exceed 20 times an executive’s salary. While it would seem critical to understand what predisposes leaders to derail, the topic is “curiously understudied”. This session will elucidate the concept of leadership 39
Friday, 1 October 2010 ∙ 11:30 - 12:50 professionals across 21 countries, 17 languages and a total sample of 338,000 people. He will describe, using a global benchmark, what the data says about the capabilities of typical managers and the actions that organisations should consider in addressing the issue of managerial bench strength. John Jones will share a study evaluating differences in over 1,500 executives’ business ethics attitudes. Several personal characteristics related to experience in managing projects, stress tolerance, number of stressful events, degree of accountability and leadership style showed the strongest relationships with unethical business orientations. Using these results, John will describe organisational interventions that support ethical decision making. John Weiner will draw from real world executive assessment programs to describe assessment strategies, issues, and research results examining individual differences that provide useful insight in distinguishing high potential employees in succession planning and executive development.
As contemporary studies suggest an increasing gap in the demand for, and supply of IT professionals, the reality of producing relevant, scalable and trusted certifications is of critical importance. The certification challenges presented by the Retail Distribution Review (RDR) for the UK financial services market will be discussed by Calibrand. RDR is the process by which UK financial advisors are mandated by the regulator to raise their skill levels by 2012. This helps to eliminate poor advice, but also nurtures public confidence in the abilities of financial advisors, that has been badly damaged over the last few years. With help from the Scottish Qualifications Authority, specialist item writers and content providers, the certification path has been developed to be sufficiently robust to hold real value to the financial workforce and the customers they serve.
Speaker 1: Eugene Burke, SHL Group Ltd. Speaker 2: John W. Jones, Vangent Inc. Speaker 3: John Weiner, PSI Services LLC
Speaker 1: Richard Coady, Certiport Speaker 2: Denis Saunders, Calibrand
Final Considerations: 12:40 – 13:00
Discussion 5-5: Myth and Reality of a Professional Certification Room: Garraf
Room: Sitges Ballroom I
Developing a complete career-orientated certification path is increasingly a key requirement for workers in many industries. For IT professionals and workers, Certiport is helping industry-leading certification providers bring certification programmes to market. With secure delivery of exams from internationally respected IT organisations such as Microsoft and CompTIA, the challenges of raising the visibility and credibility of such exams will be discussed.
Speaker 1: Eugene Burke, SHL Group Ltd. Speaker 2: Marten Roorda, Cito
40
Special Thanks William G. Harris, CEO, ATP
Marketing Committee
2010 President and Chair of ATP
Marketing Chair – Aranka Krechting, Senior Account Manager, Vocational Education & Businesses, Cito
Marten Roorda, CEO, Cito
Geoff Chapman, Director, Business Development, Calibrand
2011 President and Chair of ATP
Quinn Sutton, Senior Vice President, Marketing, Certiport
Peg Fagan Jöbst, Senior Vice President, Graduate Management Admission Council
Ron Wohl, Vice President, Wohl Communication Services, Inc.
E-ATP Executive Committee
Sponsorship Committee
E-ATP Division Chair – Eugene Burke, Science and Innovation Director, SHL Group Ltd.
Sponsorship Co-Chair – William G. Harris, CEO, ATP
E-ATP Division Vice Chair – Erwin van Schaffelaar, Business Developer, Cito
Sponsorship Co-Chair – Lauren Scheib, Administrator, ATP
E-ATP Division Secretary – Michael Kayser, VP, Director of Programme Management (EMEA), Pearson VUE
Cathy Donath, Vice President, Client Services, The Donath Group
Programme Committee Programme Chair – Cor Sluijter, Director, Vocational, Adult and Higher Education, Cito Lise-Lotte Appelgren, Test Consultant, Enlight KS Susan Martin, Global Certification, Global Education Portfolio Management, Enablement and Operations, SAP Deutschland AG & Co. KG Matthew White, Test Operations Manager, UFI
41