Irrigation Association Case Study From Paper and Pencil Testing to Computer Based Testing Co-authored by the Irrigation Association and Kryterion, Inc. October 2010
Background Every
year
Association™
the
Irrigation
(IA)
delivers
thousands certification exams across the country to landscape contractors, irrigation designers, water managers and agricultural engineers. Until recently, the exams were delivered via the traditional method of paper and pencil. Exam sites were scheduled across the country on a “on demand” basis. Packages of exam papers with instruction booklets were sent via secure delivery to proctors at the facilities and returned afterwards. However the number of candidates was growing and the administration of exams was becoming more and more challenging.
Problems The administration of the exams was becoming overwhelming and it was difficult to scale the program. In some cases candidates had to wait months to be able to take the exams and then they had to wait weeks for their results. Candidates looking to
www.KryterionOnline.com
Page 2 of 5
certification to improve their employment situation were being put off by long wait times. An unknown number of candidates were dropping out of the process. The IA is looking to become NCCA accredited. Consistency of delivery and test security were issues that needed to be addressed. Computer based testing (CBT) seemed like a good solution but the Board had a preconception that it would be difficult to do. Some on the Board were not ready to go to computer based testing. Some members of the board were not as technology based, and had, as well, a general fear of the unknown.
Method/Solution We (the management of IA) did a gap analysis to start building a picture of what the differences were between paper and pencil and CBT. Our concerns included our current standing running a program that was both NCCA accredited and legally defensible. We looked at our current vendor to see if they could meet our needs.
“Initially we wanted to run paper and pencil and CBT simultaneously. However, after a review of our needs Kryterion advised against it.”
SMT, AMP and Kryterion also responded to an RFP we sent out.
Initially we wanted to run paper and pencil and CBT
simultaneously. However, after a review of our needs Kryterion advised against it. They recommended against running a paper pencil program. They explained that paper and pencil testing would probably increase the workload rather than simplifying everything. After lengthy discussions in December 2009 the board decided to bring together the three vendors for a face-to-face meeting in January 2010. Kryterion presented the only complete end-to-end CBT solution – online item banks, ability to manage candidates, high levels of security, a network of over 300 testing centers in the US
www.KryterionOnline.com
Page 2 of 5
Page 3 of 5
and live access to information (we had none prior to this). Once the board was able to see the demonstration they were able to visualize how this would be a relatively straightforward transition. The board decided to move to CBT.
Results Achieved We had originally planned for a one and one half year transition to CBT. However, once we understood how simple the transition could be we did it over just a few months, during the summer when things were quieter.
The first
thing we had to do was upload our item banks into Webassessor. While doing the transfer, we discovered
several
mistakes
in
questions,
“We are more confident that we are giving good exams. We have the ability to write new sections quickly and efficiently.� Sherrie Schulte, Certification & Education Director
corrections and mis-keyed items that we thought had been fixed. We now have the ability to fix mistakes and be assured that they were fixed properly including an automatic audit trail. This gives us great confidence moving forward. During the transition we went through every item, painstakingly reviewing every question. We updated where necessary to allow for changes in technology and the industry. Because of our tight time schedule it was completely exhausting, yet the process itself was relatively simple, fast and quite satisfying. Our certification director built the test forms then the board reviewed and approved them. Now registration is much faster. Candidates can register to test within a few weeks. Candidates still must apply to the certification program but they can then register for an exam location. While previously candidates had to wait 30-45 days for results, they now get results right away. That quick turnaround time could make the difference in career changes. This is an industry with high-turnover, getting trained and certified quickly is important.
www.KryterionOnline.com
Page 3 of 5
Page 4 of 5
While it is too early to tell if there will be much in the way of cost savings, if there are they will probably not be significant. However we have gained competitive advantage and are able to make the exams more readily available, which increases people’s willingness to test. For example, one program requires three separate tests. These could now be taken in a matter of weeks versus several years with the previous testing sytem. The perceived benefit to our candidates is huge. With regards to Kryterion we have had very positive interaction. If we had issues they were handled in a timely manner. Sherrie Schulte, Certification & Education Director, was intimately involved in the transition. “I was new and they (Kryterion) were very helpful with my many questions. In the past our other vendor did all the testing and item banking so now it was being thrown at me which was a totally different thing – now I wonder why we didn’t switch a long time ago” “We are more confident that we are giving good exams. We have the ability to write new sections quickly and efficiently. In the old process we had there were too many gaps and room for error and everything took weeks and months. Now everything is all in one place, so it is much faster, and I can track where we are. It’s a weight off my shoulders. We are no longer risking security of our program on emails and mailing exams all over the country. It doesn’t take much to send an email to the wrong person by mistake. Overall we have greatly improved our efficiencies and are able to keep the exams relevant, accurate and accessible in a timely manner”
www.KryterionOnline.com
Page 4 of 5
Page 5 of 5
Irrigation Association Lessons Learned CBT vendors are not all the same. Some have very similar solutions to PNP vendors and don’t deliver the same benefits. Do your homework when seeking a new vendor and talk to references. Don’t be afraid to switch, as long as you have a good plan, it can be done quite quickly and with relatively few headaches. Avoid trying to do both PNP and CBT at the same time as a transition plan as this is not really necessary and might create an overwhelming amount of work for your administrators.
About Kryterion Kryterion is a full-service test development and delivery company that serves the Association, Technology and Education markets with world-class secure testing solutions. Our solutions include the five key elements of certification and testing: Psychometric services, exam management, security, real time access and test delivery, all managed through our proprietary Webassessor test development and deployment system. Our clients test with confidence and convenience. For more information on how Kryterion can streamline your testing program contact: Kryterion Certification and Licensure Sales Sales@KryterionOnline.com
www.KryterionOnline.com
Page 5 of 5