ACVREP Case Study Online Testing vs. Paper and Pencil - The Difference is Night and Day
Overview This case study details needs, process for change, results and lessons learned that one organization experienced when moving from traditional Paper and Pencil Testing to Online Testing.
Background The Academy for Certification of Vision Rehabilitation and Education Professionals (ACVREP) is a private, not-for-profit 501 c6 status certification board. ACVREP certifies over 3,000 professionals within the United States and Canada working in the vision rehabilitation and low vision services field. ACVREP offers the following three (3) professional certificates: Certified Low Vision Therapist® (CLVT®) Certified Orientation and Mobility Specialist® (COMS®) Certified Vision Rehabilitation Therapist® (CVRT®) As part of ACVREP’s certification requirements, the individual candidate must sit for and pass the applicable ACVREP certification exam (CLVT, COMS, or CVRT) for the professional discipline in which the candidate is seeking to obtain certification. Historically, the administration of ACVREP certification exams was facilitated through paper and pencil format.
www.KryterionOnline.com
With the introduction of new executive leadership in April 2008, ACVREP completed an organizational effectiveness study. One of the areas of significant need identified within the study was the administration of the certification exams via paper and pencil format.
Areas of Need The organizational effectiveness study completed by ACVREP identified the following concerns related to the administration of certification exams via paper and pencil format:
Exam Scheduling
ACVREP Areas of Need x Exam Scheduling x Distribution of Exam Materials x Cost to Candidate x Reporting of Results x Accessibility of Materials
ACVREP certification exams are administered at various testing centers located in the U.S. and Canada. In accordance to organizational policy and procedure, ACVREP staff would coordinate the scheduling of exam dates and times, which resulted in the need to support several points of contact between the testing center and the individual candidate. This action represented a significant burden to ACVREP staff. To further complicate the matter, if an individual candidate had to reschedule the administration of the exam, the entire process would be repeated, resulting in additional stress and burden on staff and candidate. Furthermore, this action served to reduce the amount of time available to staff to answer stakeholder inquiries regarding certification programs and related requirements, resulting in poor responsiveness to stakeholders’ needs and requests and decreased customer satisfaction and autonomy in the scheduling process.
Distribution of Exam Materials Given the paper-based format of the exam, ACVREP was responsible for the distribution of hard copy exam materials. As per organizational policy and procedure, ACVREP would distribute materials via Fed-Ex delivery. Given the physical transport, several concerns were noted. First, the increased risk relative to test security during transport. Second, increased burden on staff in terms of preparing materials, quality checks, and coordinating scheduled deliveries and pick-up of exam related materials. Lastly, a lack of control in stabilizing costs related to the transport of exam-related materials.
www.KryterionOnline.com
Page 2 of 7
Relative Costs to the Candidate In addition to paying the certification exam fee, the candidate was also required to pay for the costs of proctoring the exam. In review of the organizational effectiveness study, it was found that proctoring costs varied widely depending upon geographic location and that costs were increasing at an unprecedented rate. Furthermore, there was no method available to ACVREP to help stabilize costs to the candidate relative to proctor fees given the current method of exam administration (including format).
Reporting of Exam Results Given organizational policy and procedure, the reporting of exam results to candidates for certification would take up to three (3) weeks. This action represented undue burden on the candidate, resulting in poor customer satisfaction and delay in obtaining certification.
Accessibility of Exam Materials Given that ACVREP provides professional certification to individuals working in the vision rehabilitation and low vision field, it is critical to recognize that many candidates for ACVREP certification are blind or visually impaired. Given the paper-based format of the exam, as per organizational policy and procedure, ACVREP was able to provide a Braille format, a large print format, and or a test reader option to accommodate test-taker needs. However, these formats were often identified as cumbersome to the test taker due to the volume of paper need ed to support the Braille and large print formats, in consideration of exam administration time requirements. In reviewing the findings of the study with ACVREP’s testing vendor (Prometric), the executive leadership of ACVREP was unsatisfied with the recommendations provided, as said recommendations did not take into consideration ACVREP’s size, significant increases in cost to administer the exam, or the needs for individual test takers, including individuals who are blind or visually impaired.
www.KryterionOnline.com
Page 3 of 7
Resolution Beginning in 2009, the executive leadership of ACVREP began to research options to improve the organization’s exam administration methodology (format, procedures, etc.) to address the specific areas of need identified through the organizational “... the utilization of the effectiveness study. After extensive research of and discussion with several potential vendors, Webassessor™ online ACVREP selected Kryterion™ as its vendor for testing portal has allowed test development and delivery. The selection of ACVREP to stabilize exam Kryterion™ was based on the following factors:
related costs for the consumer, specifically proctor costs.”
The Webassessor™ online testing platform allows for candidates to schedule testing dates, times and locations. While ACVREP staff is available to help with the scheduling if requested Garett A. Holm, President by an individual candidate, this action has of ACVREP dramatically reduced the time spent by staff relative to scheduling and coordinating exams for individual candidates. Furthermore, candidates report to ACVREP that the online portal is “user friendly” and provides them greater “control” in the process, specifically if the administration of the exam needs to be rescheduled. Since moving to the online format, reported customer satisfaction has increased significantly and ACVREP staff has reported that the scheduling process is now less burdensome to manage, allowing for more time to respond to stakeholder questions and requests. The Webassessor™ online testing platform significantly decreases risks relative to test security, as it eliminates the need to transport hard copy exam related materials. In addition, this action allows ACVREP to stabilize costs, given consistently increasing organizational costs related to the transport of said materials. With the costs of proctoring the exam included in the per-test delivery fee, the utilization of the Webassessor™ online testing portal has allowed ACVREP to stabilize exam related costs for the consumer, specifically proctor costs. With the utilization of the Webassessor™ online testing platform, exam results are reported immediately to the test taker and to ACVREP. This action has significantly improved processing timelines for certification, which, in turn, increased stakeholder satisfaction. Furthermore, with the incorporation of said technology,
www.KryterionOnline.com
Page 4 of 7
ACVREP now has the opportunity to review its certification policies and procedures with the intent to quicken the timelines relative to certification. This would not be possible without the implementation of Webassessor™. Given the Webassessor™ online testing platforms functionality, accessibility of exam forms can be supported with zoom text, voice, and other mediums. Additional factors relative to the decision to work with Kryterion™ include: x x x x
The integration of development and delivery functions on the Webassessor platform, allowing for the opportunity to reduce costs over time; Greater control over the management (form creation, item banking, etc.) of exam programs; The ability to start implementation of the platform quickly (less than a month in ACVREP’s case); Customer support that is responsive to the needs of the organization and individual test takers.
www.KryterionOnline.com
Page 5 of 7
ACVREP Lessons Learned Given ACVREP’s traditional utilization of a paper and pencil format to facilitate the administration of its certification exams, ACVREP was placed in a position to maintain policies and procedures to support said exam format that proved detrimental to organizational efficiency and stakeholder satisfaction. Given the costs involved provided by various vendors in switching to an online platform and the relatively low numbers of exams administered per year by ACVREP, it did not seem possible to enhance the testing program. In light of these circumstances, ACVREP learned the following lessons during the process of identifying its organizational needs and resolving needs relative to the testing program:
Carefully Identify Stakeholder Needs Carefully identify your organizational and stakeholder needs relative to your testing program. This process will assist you in articulating program objectives and goals. Furthermore, without carefully articulated objectives and goals, you will not be able to communicate with potential vendors or select the best options for your testing program. Taking this concept one step forward, it is important to make these needs, objectives and goals transparent throughout the organization, including the Board and other stakeholders.
Find a Provider That Meets Your Needs Speaking metaphorically, if you feel like a small fish in a big pond, there are two options available to you. You can stay in the pond and hope for the best. Or, you can start to look for a different pond. Your satisfaction (in terms of cost, ease of administration, security, customer service, etc.) is what matters most. If you feel that you are sacrificing satisfaction for maintaining something that works at a basic level, then you are sacrificing your ability to enhance your programs and improve customer satisfaction. Take the time to identify program needs, articulate program objectives and goals, research providers, and select a provider that recognizes your specific needs, understands your objectives and goals, and is willing to work with you to accomplish said objectives and goals.
Invest Time in the Beginning Do not be afraid of a little hard work in the short-term to enhance your programs, as it will save you a lot of work down the road in maintaining operations for something that may only meet some of your overall program goals and objectives. The same can be said for start-up costs and long-term costs.
www.KryterionOnline.com
Page 6 of 7
Demand Responsiveness If it takes more than two business days to get a response from your provider, this is one day too long. Truth be told, testing programs are difficult to manage for the largest of certifying agencies. For smaller certifying agencies, like ACVREP, the task can be daunting. Responsiveness of the vendor should be a primary consideration when selecting a vendor or deciding to maintain the relationship.
Conclusion The positive results of ACVREP’s decision to move to the Webassessor™ online testing platform through Kryterion™ were noticed immediately. Garett A. Holm, President of ACVREP, states, “Prior to making the move to online testing, it seemed that no one was satisfied with the administration of the exam. Staff expressed concerns about security and staff time. Candidates also expressed great frustration with the time it took to schedule the exam and then to receive exam results. Plain and simple, the program was just not working.” Holm continues, “Since moving to online testing, all parties have expressed greater satisfaction regarding the administration of exam. It’s like night and day.” Written by Garett A. Holm, President of ACVREP, November 2010.
About Kryterion Kryterion is a full-service test development and delivery company that serves the Association, Technology and Education markets with world-class secure testing solutions. Our solutions include the five key elements of certification and testing: Psychometric services, exam management, security, real time access and test delivery, all managed through our proprietary Webassessor test development and deployment system. Our clients test with confidence and convenience. For more information on how Kryterion can streamline your testing program contact:: Kryterion Certification and Licensure Sales Sales@KryterionOnline.com
www.KryterionOnline.com
Page 7 of 7