53).' -5,4)0,% /.,).% 3%#52)49 -%!352%3 4/ $%,)6%2 3%#52% #/523% %8!-3 4/ $)34!.#% %$5#!4)/. 345$%.43 ! 7HITE 0APER
$R $AVID &OSTER AND .ANCY -ATTOON +294%2)/. )NC
2ICK 3HEARER
4HE 0ENNSYLVANIA 3TATE 5NIVERSITY´S 7ORLD #AMPUS
0ROFESSIONAL /PINIONS $ISCLAIMER !LL INFORMATION PRESENTED AND OPINIONS EXPRESSED IN THIS PUBLICATION REPRESENT THE CURRENT OPINIONS OF THE AUTHOR S BASED ON PROFESSIONAL JUDGMENT AND BEST AVAILABLE INFORMATION AT THE TIME OF THE PUBLICATION #ONSEQUENTLY THE INFORMATION IS SUBJECT TO CHANGE AND NO LIABILITY FOR ADVICE PRESENTED IS ASSUMED
53).' -5,4)0,% /.,).% 3%#52)49 -%!352%3 4/ $%,)6%2 3%#52% #/523% %8!-3 4/ $)34!.#% %$5#!4)/. 345$%.43 !BSTRACT 4HIS REPORT DESCRIBES THE SUCCESSFUL ½RST EVER PILOT OF /NLINE 3ECURED 4ESTING 4HE 7ORLD #AMPUS OF 0ENNSYLVANIA 3TATE 5NIVERSITY 035 AND +294%2)/. )NC TEAMED UP TO ADMINISTER COURSE EXAMS SECURELY AND ONLINE TO DISTANCE EDUCATION STUDENTS LOCATED IN SEVERAL STATES ACROSS THE COUNTRY AND TO A STUDENT OUTSIDE THE 5NITED 3TATES /VER EXAMS WERE ADMINISTERED TO STUDENTS IN (ORTICULTURE AND 0HYSICS COURSES MOSTLY IN THEIR HOMES USING A VARIETY OF NEW SECURITY TECHNOLOGIES INCLUDING ONLINE PROCTORING 2EAL 4IME $ATA &ORENSICS‰ BIOMETRIC AUTHENTICATION AND OTHERS 4HE COMBINATION OF THESE TECHNOLOGIES ASSURED THE IDENTITY OF EACH STUDENT AND THE SECURITY VALIDATION OF THE TEST SCORES 4HE STUDY CLEARLY DEMONSTRATED THE FEASIBILITY OF SECURE TESTING OVER THE )NTERNET WHEREVER STUDENTS ARE LOCATED )NTRODUCTION $ISTANCE EDUCATION INSTITUTIONS HAVE STRUGGLED IN RECENT YEARS TO ½ND A CONVENIENT AND SECURE WAY TO ADMINISTER ONLINE PROCTORED EXAMS TO THEIR STUDENTS 7HILE THE INSTRUCTION IS GENERALLY DELIVERED CONVENIENTLY TO A STUDENT´S HOME OR WORK THROUGH THE )NTERNET IT HASN´T BEEN POSSIBLE TO PROVIDE THE TESTS USING THE SAME TECHNOLOGY PRIMARILY BECAUSE THE SECURITY OF THE EXAM COULD NOT BE ASSURED AND BECAUSE IT HAS BEEN DIF½CULT TO AUTHENTICATE THE IDENTITY OF THE STUDENT TAKING THE TEST !S A RESULT UNIVERSITIES AND COLLEGES OFFERING ONLINE DISTANCE EDUCATION COURSES AND EVEN SOME HIGH SCHOOLS HAVE RESORTED TO A NUMBER OF DIF½CULT OPTIONS 3OME ORGANIZATIONS ASK THE STUDENTS TO ±½ND YOUR OWN PROCTOR² /THERS PROVIDE PAPER AND PENCIL TESTS SENT BY MAIL "OTH OF THESE REQUIRE THE DIF½CULT STEPS OF ½NDING SOMEONE TO ASSIST THE STUDENT IN TAKING THE TEST GETTING CONTRACTS SIGNED DEALING WITH SCHEDULES AND SOMETIMES PAYMENTS AND HOPING THAT SECURITY MEASURES ARE TAKEN 4HESE OPTIONS TAKE A GREAT DEAL OF BOTH THE ORGANIZATION´S STAFF AND STUDENT´S TIME 4HE PROCESS THEN HAS TO BE REPEATED FOR THE NEXT EXAM 3OME DISTANCE EDUCATION INSTITUTIONS PROVIDE UNPROCTORED OR UNMONITORED )NTERNET TESTING 4HEY HAVE THE STUDENT SIMPLY SIGN AN ±HONOR CODE² AGREEMENT AS THE SOLE SECURITY PROCEDURE HOPING THAT THE STUDENT WON´T CHEAT WON´T SHARE THE TEST WITH OTHERS OR WON´T HAVE SOMEONE ELSE TAKE THE TEST FOR HIM OR HER 4HE SECURITY ISSUES HAVE BEEN SEVERE ENOUGH THAT SOME PROGRAMS HAVE DONE AWAY WITH TESTING ALTOGETHER /F THESE MANY USE OTHER FORMS OF ASSESSMENT INCLUDING STUDENT PORTFOLIOS PAPERS PROJECTS AND OTHER TYPES OF STUDENT WORK /F COURSE WITH THESE METHODS THE SAME SECURITY PROBLEMS REMAIN )T IS LOGICAL TO ASSUME THAT DISTANCE EDUCATION PROVIDERS WOULD PREFER A TESTING SOLUTION THAT IS )NTERNET BASED AT LEAST AS SECURE OR MORE SECURE THAN TESTING THAT OCCURS NORMALLY FOR ON CAMPUS PROGRAMS AND WHICH ALLOWS FOR CONVENIENCE THAT IS SO HIGHLY PRIZED IN THIS MARKET
ˆ +294%2)/. ¯ !LL 2IGHTS 2ESERVED
WWW KRYTERIONONLINE COM
+294%2)/. )NC BASED IN 0HOENIX !RIZONA HAS DEVELOPED A PRODUCT CALLED 7EBASSESSOR‰ WHICH PROVIDES /NLINE 3ECURED 4ESTING WHEREVER AND WHENEVER NEEDED 3ECURITY TECHNOLOGIES ESPECIALLY DEVELOPED FOR 7EBASSESSOR‰ INCLUDE
+EYSTROKE !NALYTICS‰ AND DIGITAL PHOTOGRAPHS THAT BIOMETRICALLY IDENTIFY AND AUTHENTICATE
/NLINE PROCTORING USING WEBCAMS THAT ARE ENABLED WITH AUDIO TO MONITOR TEST TAKERS AND
A STUDENT TO DEAL PROFESSIONALLY AND OBJECTIVELY WITH STUDENTS WHO HAVE TECHNICAL DIF½CULTIES WITH THE EXAM OR WHO ATTEMPT TO CHEAT
2EAL 4IME $ATA &ORENSICS‰ IS A SOPHISTICATED ONGOING STATISTICAL ANALYSIS OF ANSWER AND ANSWER LATENCIES THAT IS USED TO DETECT UNUSUAL PATTERNS OF HOW STUDENTS ARE RESPONDING DURING A TEST
3ECURE BROWSER CONTROL TO MAKE SURE STUDENTS CANNOT ACCESS )NTERNET AND OTHER RESOURCES
+EYSTROKE MONITORING TO DETECT AND PREVENT THE USE OF UNAUTHORIZED KEYSTROKES E G !LT
3ECURE TEST AND )TEM DESIGNS WHICH MINIMIZE EXPOSURE OF QUESTIONS AND QUESTION CONTENT
!BILITY FOR FACULTY TO REVIEW RECORDED TEST SESSIONS WHEN THEY FEEL IT IS NECESSARY
ON THEIR HARD DRIVE DURING THE EXAM 4AB 0RNT 3CRN AND OTHERS
7ITH THESE MULTIPLE AND EFFECTIVE SECURITY MEASURES IN PLACE TESTS GIVEN ONLINE AS PART OF A DISTANCE EDUCATION PROGRAM ARE AS SECURE IF NOT MORE SO THAN TESTS GIVEN IN CLASSROOMS CAMPUS TESTING CENTERS OR OTHER PROFESSIONAL TESTING LOCATIONS
2ECENT FEDERAL GOVERNMENT LEGISLATION THE ±(IGHER %DUCATION /PPORTUNITY !CT OF ² PROVIDES FOR ACCREDITING AGENCIES OR ASSOCIATIONS TO REQUIRE THAT DISTANCE EDUCATION PROGRAMS VALIDATE THE IDENTITY OF THE DISTANCE EDUCATION STUDENT AND PROVIDE ASSURANCE THAT THE COURSE WORK COMPLETED INCLUDING
TESTS WAS COMPLETED BY THE ACTUAL ENROLLED STUDENT 5SING THE 7EBASSESSOR‰ SECURITY METHODS DATA CAN BE ROUTINELY GATHERED TO ESTABLISH COMPLIANCE WITH THE !CT ! #ASE 3TUDY 0ILOTING /NLINE 3ECURED 4ESTING AT 035 7ORLD #AMPUS ! PILOT OF /NLINE 3ECURED 4ESTING USING 7EBASSESSOR‰ WAS CARRIED OUT FOR TWO 035 7ORLD #AMPUS COURSES (ORTICULTURE AND 0HYSICS DURING THE SUMMER OF
"ILL ( 2 WAS SIGNED INTO LAW !UGUST
( 2 3ECTION 0ART ( ¯ 0ROGRAM )NTEGRITY " II ² THE AGENCY OR ASSOCIATION REQUIRES AN INSTITUTION THAT OFFERS DISTANCE EDUCATION OR CORRESPONDENCE EDUCATION TO HAVE PROCESSES THROUGH WHICH THE INSTITUTION ESTABLISHES THAT THE STUDENT WHO REGISTERS IN A DISTANCE EDUCATION OR CORRESPONDENCE EDUCATION COURSE OR PROGRAM IS THE SAME STUDENT WHO PARTICIPATES IN AND COMPLETES THE PROGRAM AND RECEIVES THE ACADEMIC CREDIT ²
ˆ +294%2)/. ¯ !LL 2IGHTS 2ESERVED
WWW KRYTERIONONLINE COM
4EST QUESTIONS FOR THE 0HYSICS AND (ORTICULTURE COURSE TESTS WERE ENTERED INTO 7EBASSESSOR‰ BY 7ORLD #AMPUS ,EARNING $ESIGN STAFF 4HE PUBLISHED EXAMS WERE THEN MADE AVAILABLE TO THE STUDENTS THROUGH 7EBASSESSOR‰ OVER THE )NTERNET %ACH COURSE INCLUDED THE ADMINISTRATION OF THREE EXAMS OVER THE SEMESTER WITH TESTING WINDOWS ESTABLISHED BY THE INSTRUCTORS 0ARTICIPATING STUDENTS WERE ASKED TO PURCHASE A SPECI½ED WEBCAM AND WERE ALSO PROVIDED WITH THE NAMES OF VENDORS THAT HAD THE CAMERAS IN STOCK 4HE PRICE OF THE CAMERA WAS VIEWED AS MINIMAL FOR PARTICIPATION IN THE COURSE AS THE CAMERA COULD BE USED FOR OTHER COURSES AND OTHER ONLINE COURSE FEATURES I E 6IDEO #HAT #OLLABORATIVE 'ROUPWORK ETC 035´S 7ORLD #AMPUS ALREADY REQUIRED BROADBAND )NTERNET AVAILABILITY AS A PRE REQUISITE TO TAKING COURSES ONLINE SO THE WEBCAM REQUIREMENT DID NOT RESULT IN ANY ADDITIONAL EXPENSE BEYOND THE CAMERA´S PURCHASE PRICE 0REPARING TO 4AKE 4ESTS 3TUDENTS NEEDED TO COMPLETE A FEW STEPS A SINGLE TIME AS PREPARATION TO
TAKE THE COURSE EXAMS SECURELY THROUGH 7EBASSESSOR‰ /NCE THESE WERE DONE THE ½RST TEST°AND ALL SUBSEQUENT TESTS EVEN FOR OTHER COURSES°COULD BE EASILY TAKEN 3TEP !FTER PURCHASING THE WEBCAM STUDENTS HAD TO MAKE SURE IT WAS INSTALLED PROPERLY 4HEY DID THIS BY FOLLOWING THE INSTRUCTIONS AND USING THE INSTALLATION SOFTWARE
3TEP 3TUDENTS THEN ENROLLED OR REGISTERED INTO THE TESTING SYSTEM 5SING 7EBASSESSOR‰ STUDENTS
ACCESSED THEIR HOME PAGE &ROM THERE FOLLOWING ON SCREEN PROMPTS THEY WERE ABLE TO TAKE AN
INITIAL ENROLLMENT DIGITAL PHOTOGRAPH USING THE WEBCAM 4HIS PHOTOGRAPH SERVED AS THE VERI½CATION PHOTOGRAPH THROUGHOUT THEIR TIME WITH 035´S 7ORLD #AMPUS 3TEP #ONTINUING THE REGISTRATION PROCESS STUDENTS BIOMETRICALLY ENROLLED USING THE +EYSTROKE
!NALYTICS‰ METHODOLOGY IN 7EBASSESSOR‰ 4HIS SIMPLE PROCESS TOOK APPROXIMATELY THREE MINUTES 4HE STUDENT WAS GIVEN A SYSTEM GENERATED CHARACTER PHRASE AND REQUIRED TO TYPE IT AN AVERAGE OF NINE TO TIMES +EYSTROKE !NALYTICS‰ THEN PRODUCED A ±DIGITAL SIGNATURE² FOR THE STUDENT THAT CANNOT BE REPLICATED BY ANYONE ELSE 7HEN A PERSON REPRODUCES HIS OR HER ±DIGITAL SIGNATURE² CORRECTLY TESTS ARE ABLE TO BE LAUNCHED 3TEP 3TUDENTS WERE ALSO INSTRUCTED TO DOWNLOAD 7EBASSESSOR 3ENTINEL 3ECURITY 3HIELD‰ 3ENTINEL
A VERY SMALL PROGRAM RESPONSIBLE FOR LOCKING DOWN THE BROWSER AT EXAM TIME $URING THE PILOT THE BROWSER WAS RESTRICTED TO 7INDOWS )NTERNET %XPLORER AND -OZILLA &IREFOX BOTH FOR THE 7INDOWS
OPERATING SYSTEM )NSTRUCTIONS ON HOW TO DOWNLOAD THIS SMALL PROGRAM WERE AVAILABLE ON 035´S !NGEL ,EARNING -ANAGEMENT 3YSTEM ,-3 AS WELL AS ON THE 7EBASSESSOR‰ STUDENT HOME PAGE &OR 3ENTINEL‰ TO WORK !DOBE &LASH 0LAYER NEEDED TO EXIST ON THE STUDENT´S COMPUTER )F IT WASN´T ALREADY ON THE COMPUTER IT WAS DOWNLOADED AND INSTALLED 3TEP 7HILE NOT REQUIRED STUDENTS WERE ALSO ENCOURAGED TO VIEW A TUTORIAL TO FAMILIARIZE
THEMSELVES WITH THE TECHNOLOGY AND THE VARIOUS STEPS THAT WOULD BE REQUIRED AT TEST LAUNCH TIME
4HE 3AFARI BROWSER FOR THE -ACINTOSH OPERATING SYSTEM IS NOW SUPPORTED
ˆ +294%2)/. ¯ !LL 2IGHTS 2ESERVED
WWW KRYTERIONONLINE COM
3CHEDULING %XAMS 5PON SUCCESSFUL COMPLETION OF THESE PREPARATORY STEPS THE STUDENTS COULD
THEN SCHEDULE THEIR ½RST EXAM &OR THIS PROCESS THEY HAD TO NAVIGATE TO THEIR HOME PAGE AND ENTER A PRE DETERMINED EXAM CODE THAT WAS GIVEN TO THEM WITHIN THE !NGEL ,-3 !S LONG AS THE EXAM SCHEDULING WINDOW WAS OPEN THE STUDENTS COULD SELF SCHEDULE FOR AN EXAM BY CHOOSING A DAY AND TIME CONVENIENT FOR THEM !LL EXAMS HAD TO BE SCHEDULED AT LEAST HOURS PRIOR TO THE CLOSE OF THE TESTING WINDOW #HANGES TO THE SCHEDULE FOR EXAMS WERE HANDLED BY THE 7ORLD #AMPUS (ELP $ESK AND THE INDIVIDUAL INSTRUCTORS ,AUNCHING %XAMS !T THE SCHEDULED TEST TIME LAUNCHING THE EXAM REQUIRED A FEW SIMPLE AUTHENTICATION STEPS &IRST THE STUDENTS TOOK ANOTHER DIGITAL PHOTO OF THEMSELVES WITH THE WEBCAM CALLED THE ±TEST
LAUNCH² PHOTOGRAPH 4HIS IS USED BY ONLINE PROCTORS DESCRIBED BELOW TO LAUNCH THE TEST BY COMPARING IT TO THE ORIGINAL ENROLLMENT PHOTO ALREADY PROVIDED 3ECOND THE STUDENTS WERE PROMPTED TO TYPE THEIR ±DIGITAL SIGNATURE²°THE UNIQUE CHARACTER AUTHENTICATION PHRASE°AND WERE GIVEN THREE OPPORTUNITIES TO TYPE IT AS THEY DID WHEN THEY ENROLLED )F THE +EYSTROKE !NALYTICS STEP COULD NOT BE COMPLETED THE STUDENT CALLED A +294%2)/. ONLINE PROCTOR WHERE AN IMMEDIATE VISUAL MATCH BETWEEN THE LAUNCH PHOTOGRAPH AND THE ORIGINAL ENROLLMENT PHOTOGRAPH COULD BE DETERMINED 4HE PROCTOR WAS THEN ABLE TO OVERRIDE THE +EYSTROKE !NALYTICS‰ REQUIREMENT AND AUTHORIZE THE TEST LAUNCH !S PART OF THE LAUNCH PROCESS THE STUDENTS WERE ASKED ONLINE TO AGREE TO HAVING THE TEST SESSION AUDIO AND VIDEO RECORDED !S PART OF THAT ONLINE AGREEMENT THE STUDENTS WERE ALSO ASKED TO READ AND AGREE TO THE 035 !CADEMIC )NTEGRITY 3TATEMENT (AVING SUCCESSFULLY PASSED THE AUTHENTICATION AND ONLINE AGREEMENT STEPS THE TEST LAUNCH BUTTON WAS ACTIVATED AND THEY COULD BEGIN THE EXAM 0ROPER PLACEMENT OF THE WEBCAM WAS CRITICAL IN ALLOWING THE PROCTOR TO VIEW THE HANDS KEYBOARD AND FACE OF THE STUDENT DURING THE TEST $OCUMENTATION WAS SUPPLIED TO AID STUDENTS WITH POSITIONING
THEIR WEBCAM 4AKING THE 4EST $URING TESTING STUDENTS WERE MONITORED BY TWO LEVELS OF HUMAN PROCTORING AND
SEVERAL TYPES OF VIRTUAL OR COMPUTER PROCTORING 4IER HUMAN ONLINE PROCTORS WATCHED VIDEO AUDIO THUMBNAILS OF ALL STUDENTS TESTING ON A PARTICULAR DAY AND TIME SCHEDULED EXAMS WERE SPREAD ACROSS EXAM WEEKS FOR EACH COURSE AND WERE SPECI½ED BY 035 AND THE COURSE INSTRUCTORS 4HESE 4IER PROCTORS MONITORED THE THUMBNAILS FOR OVERT SECURITY AND OTHER ISSUES AND SENT ALERTS WHEN PROBLEMS WERE DETECTED 4IER HUMAN ONLINE PROCTORS HAD ACCESS TO MORE DETAILED VIEWS OF THE VIDEO AUDIO STREAM AS WELL AS THE PHOTOGRAPHS TAKEN DURING ENROLLMENT AND PRIOR TO THE LAUNCH OF THE EXAM 4HESE 4IER PROCTORS WERE RESPONSIBLE FOR DEALING WITH ANY SECURITY OR TECHNICAL ISSUES THAT CAME UP WHEN THE TEST WAS LAUNCHED OR WHILE IT WAS BEING TAKEN 4HEY HANDLED ANY ALERTS FROM 4IER PROCTORS AND THE AUTOMATED VIRTUAL PROCTORING DETECTION PROCESSES RUNNING
7HILE NOT AVAILABLE AT THE TIME OF THIS PILOT 7EBASSESSOR‰ NOW SUPPLIES AN ±ON SCREEN² LIVE CAMERA VIEW TO THE STUDENTS LETTING THEM SEE THEMSELVES AS THE PROCTOR SEES THEM 4HIS ELIMINATES ANY ±GUESSWORK² AS TO THE PROPER CAMERA PLACEMENT AND REDUCES THE NEED FOR TECHNICAL SUPPORT
ˆ +294%2)/. ¯ !LL 2IGHTS 2ESERVED
WWW KRYTERIONONLINE COM
/NE METHOD OF VIRTUAL OR AUTOMATED PROCTORING DETECTION CONSISTED OF 2EAL 4IME $ATA &ORENSICS‰ WHICH MONITORED THE STUDENTS´ RESPONSES DURING THE TEST LOOKING FOR STRANGE PATTERNS SUCH AS TOO FAST RESPONDING )NAPPROPRIATE KEYSTROKES WERE ALSO MONITORED &OR EXAMPLE IF A STUDENT ATTEMPTED TO USE THE 0RNT3CRN 0RINT 3CREEN KEY DURING THE TEST IN ORDER TO RECORD AND SAVE A COPY OF A TEST
QUESTION THIS ACTION WAS NOTED AND THE ONLINE PROCTOR WAS ALERTED !LL ABERRATIONS AND SECURITY RISKS DURING THE TEST WERE HANDLED BY THE ONLINE PROCTORS WHO PROVIDED ONLINE WARNINGS TO THE TEST TAKER WHEN APPROPRIATE AND PAUSING OR SUSPENDING THE TEST WHEN NECESSARY !LL OF THESE ALERTS OR ±RED ¾AGS² WERE NOTED IN SESSION LOGS AND THE 7ORLD #AMPUS STAFF WAS NOTI½ED IN CASE FURTHER ACTION WAS REQUIRED 2ESULTS #OMPLETION 2ATE ! TOTAL OF STUDENTS COMPLETED EXAMS PROVIDING IMPORTANT SCORES FOR COURSE COMPLETION !LL EXAMS WERE PROCTORED UNDER SECURE CONDITIONS WITH AUTHENTICATION AND COMPLETE
SECURITY MONITORING BY THE SYSTEM AND STAFF AS DESCRIBED 4HE )NTERNET CONNECTIONS WERE CONSISTENT AND THE WEBCAMS FUNCTIONED AS DESIGNED 3IX OTHER STUDENTS BEGAN THE COURSE S AND MAY HAVE EVEN SCHEDULED AN EXAM BUT DID NOT TAKE ANY EXAMS 'LOBAL REACH /F THE TESTS ADMINISTERED MOST WERE IN 0ENNSYLVANIA HOWEVER THERE WERE A NUMBER
OF STUDENTS PARTICIPATING FROM OTHER STATES AND EVEN ONE STUDENT FROM OUTSIDE OF THE 5NITED 3TATES 4HERE WERE NO OBVIOUS RESTRICTIONS TO WHERE /NLINE 3ECURED 4ESTING COULD TAKE PLACE PROVIDING 035 WITH THE OPPORTUNITY TO CON½DENTLY PROVIDE ITS SERVICES ON A GLOBAL SCALE 3ECURITY 4HE DETERRENCE VALUE OF /NLINE 3ECURED 4ESTING WAS CLEAR 7ITH CLOSE HUMAN AND AUTOMATED OBSERVATION ONLY TWO SECURITY PROBLEMS ONE EACH FOR TWO STUDENTS WERE DETECTED 4HIS REPRESENTS
OF ALL TESTS GIVEN°A VERY LOW RATE OF SECURITY PROBLEMS COMPARED TO PUBLISHED RESEARCH ON THE GENERAL RATES OF CHEATING IN COLLEGE COURSES 4HE SYSTEM´S CLOSE MONITORING MADE IT VERY DIF½CULT FOR CHEATING TO OCCUR AND GO UNDETECTED (ERE IS A BRIEF DESCRIPTION OF THE TWO SECURITY INCIDENTS
/NE STUDENT WAS DISCOVERED BY ONLINE PROCTORS TALKING WITH ACCOMPLICES IN THE TESTING AREA
DESCRIBING A TEST QUESTION AND REQUESTING THE CORRECT ANSWER !T LEAST TWO ACCOMPLICES WERE OUTSIDE OF THE VIEW OF THE CAMERA 4HE STUDENT WAS WARNED BY THE PROCTOR ACKNOWLEDGED THE
WARNING AND COMPLETED THE EXAM WITHOUT FURTHER INCIDENT
! SECOND STUDENT WAS SEEN USING HEADPHONES WHICH ARE NOT ALLOWED DURING TESTING 4HE STUDENT WAS WARNED BY THE ONLINE PROCTOR THE HEADPHONES WERE REMOVED AND THE EXAM WAS COMPLETED
7HILE STUDENTS COULD ATTEMPT CERTAIN UNAUTHORIZED KEYSTROKES SUCH AS 0RINT 3CREEN 3ENTINEL‰ DID NOT ALLOW THE ACTUAL COMMAND TO OCCUR
4HE STUDENT ACTUALLY STATED THAT HE HAD ±FORGOTTEN THAT AUDIO WAS ALSO PICKED UP²
ˆ +294%2)/. ¯ !LL 2IGHTS 2ESERVED
WWW KRYTERIONONLINE COM
4ECHNICAL PROCESS ISSUES 7HILE THERE WERE ONLY TWO ACTUAL SECURITY INCIDENTS THE INTRODUCTION OF A NEW WAY TO TEST SECURELY ONLINE DID REVEAL SOME PROCEDURAL AND TECHNICAL STEPS NEEDING
IMPROVEMENT 4HERE WERE A TOTAL OF TECHNICAL OR PROCEDURAL ISSUES LOGGED AND DEALT WITH BY +294%2)/. SUPPORT STAFF AND ONLINE PROCTORS )T IS IMPORTANT TO NOTE THAT OF THE TECHNICAL ISSUES WERE NOT RELATED DIRECTLY TO TAKING EXAMS BUT OCCURRED DURING THE SET UP PROCESS OR DURING BIOMETRIC ENROLLMENT AND TEST SCHEDULING 4HIRTY SEVEN OF THE TESTS MOSTLY EXAMS AND OCCURRED WITHOUT ANY PROBLEMS AT ALL &OUR STUDENTS COMPLETED THE SETUP REGISTRATION AND SCHEDULING AND ALL THREE EXAMS WITH NO TECHNICAL ASSISTANCE AT ALL 4HE MOST PREVALENT TECHNICAL ISSUES EXPERIENCED BY STUDENTS WERE
&AILING TO DOWNLOAD 3ENTINEL‰ AND OR !DOBE &LASH 0LAYER AS A PRE REQUISITE 4HE TEST WILL
NOT LAUNCH WITHOUT 3ENTINEL‰ INCIDENTS
)MPROPER INSTALLATION OR PLACEMENT OF THE WEBCAM 7ITHOUT AN ACCEPTABLE VIDEO SIGNAL INCIDENTS OR THE PROPER VIEW OF THE TEST TAKER´S HANDS KEYBOARD TORSO AND HEAD
INCIDENTS THE ONLINE PROCTOR WOULD NOT AUTHORIZE THE LAUNCH OF THE EXAM
)MPROPER SCHEDULING OF EXAMS 7HILE MORE OF A PROBLEM WITH FOLLOWING INSTRUCTIONS ½VE STUDENTS FAILED TO SCHEDULE WITHIN THE TIME LIMIT AND SIX STUDENTS SCHEDULED THE EXAM
BUT DIDN´T TAKE THE TEST 4WO STUDENTS SCHEDULED THE EXAM BUT NEEDED TO RESCHEDULE FOR PERSONAL REASONS 4HESE TYPES OF SCHEDULING ISSUES ARE COMMON FOR ANY GROUP OF STUDENTS AND THE SUPPORT STAFF WAS ABLE TO ASSIST WHEN NEEDED 4HE REMAINING TECHNICAL OR PROCESS ISSUES WERE MIXED
)NTERNET CONNECTION FAILURE TWO INCIDENTS 4HE NEED TO RESET THE +EYSTROKE !NALYTICS‰ ENROLLMENT PROCESS TWO INCIDENTS 6ARIOUS MINOR SOFTWARE ISSUES
¯ ¯ ¯ ¯
! STUDENT WAS ABLE TO LAUNCH A PREVIOUSLY COMPLETED EXAM ONE INCIDENT
¯
5NEXPLAINED 3YSTEM %RRORS OCCURRED FOUR INCIDENTS
4HE "ACKSPACE KEY WOULD NOT WORK ONE INCIDENT !N ENROLLMENT PHOTO WAS NOT AVAILABLE FOR THE ONLINE PROCTOR TWO INCIDENTS 4HE ONLINE PROCTOR´S 2ESUME COMMAND USED TO RESUME A TEST AFTER PAUSING IT DIDN´T FUNCTION THREE INCIDENTS
!S EXPECTED DURING THE EARLY WEEKS OF THE PILOT WHEN SETTING UP REGISTERING AND TAKING THE ½RST EXAM MOST OF THE STUDENTS OR OF REQUIRED SOME TYPE OF TECHNICAL ASSISTANCE EACH
3OME CHANGES TO STUDENT INSTRUCTIONS AND TO 7EBASSESSOR‰ FUNCTIONALITY WERE MADE DURING THE PERIOD OF THE PILOT AND IMMEDIATELY REDUCED THE NUMBER OF SUPPORT CALLS 0LUS CHANGES MADE AFTER THE PILOT ½NISHED MADE THE PROCESS EVEN EASIER FOR STUDENTS
!S NOTED EARLIER 7EBASSESSOR‰ NOW PROVIDES AN ³ON SCREEN´ LIVE CAMERA VIEW DURING THE TEST LAUNCH STEPS
ˆ +294%2)/. ¯ !LL 2IGHTS 2ESERVED
WWW KRYTERIONONLINE COM
PROBLEM NEEDING TO MINUTES FOR A SUPPORT TECHNICIAN TO RESOLVE !LL TECHNICAL PROBLEMS WERE EASILY RESOLVED AND ALL STUDENTS WERE ABLE TO TAKE THE REQUIRED TESTS WITHOUT MUCH DELAY /F MORE INTEREST OVER THE COURSE OF THE PILOT AND BECAUSE SETUP ISSUES HAD ALL BEEN DEALT WITH EARLY ON
STUDENTS BECAME MORE FAMILIAR WITH THE LAUNCH PROCEDURES AND TOOK SUBSEQUENT TESTS THEIR SUPPORT NEEDS REDUCED TO A MUCH LOWER RATE AS WELL SEE &IGURE &IGURE 4ECHNICAL PROBLEMS DURING THE PILOT ACCOUNTING FOR MOST OF THE SUPPORT CALLS BY STUDENTS
REDUCED DRASTICALLY OVER THE TIME FRAME OF THE PILOT AND AS STUDENTS GAINED EXPERIENCE
3URVEY 2ESULTS 3TUDENT 3URVEY 3TUDENTS IN BOTH THE (ORTICULTURAL AND 0HYSICS COURSES WERE ASKED TO COMPLETE A
SHORT SURVEY ON THEIR ONLINE TESTING EXPERIENCE 4HIS SURVEY WAS ADMINISTERED BETWEEN THE SECOND AND THIRD EXAM IN EACH COURSE 4HE RESULTS OF THE SURVEY WERE MIXED IN REGARDS TO SATISFACTION WITH THE ONLINE PROCTORED TESTING EXPERIENCE 4HIS WAS EXPECTED AS STUDENTS WERE BEING EXPOSED TO A NEW METHODOLOGY AND MANY WERE USED TO THE PREVIOUS PAPER BASED SYSTEM OF EXAMS !PPROXIMATELY OF THE STUDENTS WERE EXCITED WITH THE ONLINE PROCTORED EXAMS AND COMMENTED ON THE CONVENIENCE
WHILE THE OTHER HALF WERE LESS ENTHUSIASTIC 4HESE LATTER STUDENTS COMMENTED ON PROBLEMS WITH THE CAMERA INSTALLATION PROBLEMS LAUNCHING THE EXAMS OFTEN DUE TO THE 3ENTINEL‰ NOT BEING INSTALLED AND ISSUES WITH SCHEDULING EXAMS SOMETIMES DUE TO THE CORRECT TEST CODE NOT BEING ENTERED IN A STUDENT´S PROFILE !LSO A FEW COMMENTED THEY WERE UNCOMFORTABLE WITH THE IDEA OF BEING OBSERVED ONLINE 4HE SURVEY RESULTS INDICATED THAT MANY STUDENTS DID NOT SEE AS HIGH A VALUE AS WAS ANTICIPATED FOR THE IMMEDIATE RECEIPT OF EXAM SCORES 4HIS IS A SOMEWHAT UNEXPECTED ½NDING AS IT WAS ASSUMED THAT THIS WOULD BE A BIG BENE½T FOR STUDENTS AS THEY DID NOT HAVE TO WAIT THE USUAL THREE TO FOUR WEEKS TO RECEIVE THEIR EXAM RESULTS AS WAS THE CASE WITH THE PAPER BASED TESTS
ˆ +294%2)/. ¯ !LL 2IGHTS 2ESERVED
WWW KRYTERIONONLINE COM
7ORLD #AMPUS 3TAFF 2ESPONSES ! POST PILOT REVIEW WAS HELD WITH THE 035 7ORLD #AMPUS STAFF TO GET THEIR OPINIONS ON THE PILOT )T WAS FOUND THAT
4HE TECHNOLOGY WORKED TO PROVIDE CONVENIENT EXAMS AND IMPROVE SECURITY
+294%2)/.´S SUPPORT STAFF WAS RESPONSIVE AND HELPFUL
4HE TECHNOLOGY PILOTED WAS PROMISING AND INTEND TO MOVE FORWARD WITH IMPLEMENTATION
4HE INSTRUCTORS WERE PLEASED WITH THE EXPERIENCE AND WILL CONTINUE TO PROVIDE THEIR TESTS
&ACULTY WOULD LIKE TO SEE SOME ENHANCEMENTS IN 7EBASSESSOR‰ TO MAKE GRADING SHORT ANSWER
IN OTHER COURSES THROUGH 7EBASSESSOR‰ MATCHING AND ½LL IN THE BLANK TYPE QUESTIONS MORE CONVENIENT &OR SOME STUDENTS WHO DO NOT HAVE EASY ACCESS TO COMPUTERS WHERE UPDATES OR SOFTWARE CAN BE LOADED AS WITH DEPLOYED MILITARY STUDENTS AN OPTIONAL METHOD FOR TESTING MAY NEED TO BE IMPLEMENTED
#ONCLUSIONS AND .EXT 3TEPS 2EASONABLE CRITERIA FOR SUCCESS FOR THIS PILOT INCLUDE EFFECTIVE TESTING OF STUDENTS IN 0ENNSYLVANIA AND ACROSS A BROAD GEOGRAPHICAL REACH EASY COMPLETION OF ALL SETUP STEPS FOR THE EXAMINATION PROCESS LAUNCHING AND COMPLETING ALL EXAMS CONSTANT MONITORING AND DETERRENCE OF SECURITY PROBLEMS THE DETECTION OF ACTUAL SECURITY PROBLEMS WHEN THEY OCCUR STUDENT SATISFACTION
AND FACULTY STAFF SATISFACTION #ONSIDERING BOTH THE POSITIVE AND NEGATIVE RESULTS THE FOLLOWING LETTER GRADES COULD BE ASSIGNED %FFECTIVE 4ESTING !NYWHERE
!
%ASY #OMPLETION OF 3ETUP
#
,AUNCHING #OMPLETING 4ESTS
"
3ECURITY $ETERRENCE
!
3ECURITY $ETECTION (ANDLING
!
3TUDENT 3ATISFACTION
#
&ACULTY 3TAFF 3ATISFACTION
"
!CCORDING TO THESE CRITERIA THE PILOT CAN BE CONSIDERED VERY SUCCESSFUL $ESPITE SOME OF THE TECHNICAL ISSUES STUDENTS DISCOVERED DURING THE LAUNCH PROCESS EACH STUDENT WAS ABLE TO LAUNCH HIS OR HER EXAM S COMPLETE THE EXAMS AND RECEIVE VALID SCORES 3ECURITY PROBLEMS WERE RARE $ESPITE THE CONSTANT AND STRINGENT DETECTION METHODS OR PERHAPS BECAUSE OF THE LARGE MAJORITY OF STUDENTS COMPLETED THE TESTS WITHOUT ANY SECURITY ISSUES 7HEN A STUDENT DID ATTEMPT TO CHEAT IT WAS DISCOVERED IMMEDIATELY AND DEALT WITH EFFECTIVELY
ˆ +294%2)/. ¯ !LL 2IGHTS 2ESERVED
WWW KRYTERIONONLINE COM
2EASONABLY THE FACULTY AND 7ORLD #AMPUS ,EARNING $ESIGN STAFF FELT THAT WITH THE ONGOING MODI½CATIONS TO THE REGISTRATION ENVIRONMENT THE ADDITION OF A CAMERA PLACEMENT VIEW AND WITH MODI½CATIONS TO THE STUDENT INSTRUCTIONS ONLINE SECURED TESTING THROUGH 7EBASSESSOR‰ WILL CONTINUE TO IMPROVE RAPIDLY AS A SOUND AND VALID METHOD THAT PROVIDES STUDENT CONVENIENCE SATISFACTION AND A SECURE TESTING ENVIRONMENT THAT HELPS ASSURE ACADEMIC INTEGRITY +NOWN SOFTWARE PROBLEMS WITH 7EBASSESSOR‰ WHILE FEW WERE CORRECTED DURING THE PILOT SINCE THE SOFTWARE AS A SERVICE OR 3!!3 MODEL ALLOWED THE IMMEDIATE ½X OF BUGS AND OTHER PROBLEMS &OR SUBSEQUENT PILOT TESTING NEW SOFTWARE FEATURES WILL MAKE THE SETUP AND ENROLLMENT PROCESS EASIER AND MORE FOOLPROOF 4ESTS WILL BE EASIER TO LAUNCH )T IS EXPECTED THAT THE RATE OF SUPPORT CALLS DURING THE EARLY STAGES WILL BE REDUCED FROM TO A GOAL OF UNDER 4HE GOAL IS REASONABLE THE PILOT RESULTS INDICATED THAT AS STUDENTS GAINED EXPERIENCE WITH 7EBASSESSOR‰ THE NUMBER OF SUPPORT CALLS WERE SIGNI½CANTLY REDUCED 4HIS RESULT WILL CONTINUE TO IMPROVE AS THESE STUDENTS TAKE THEIR NEXT COURSES )N ADDITION TO THE DEMONSTRATED ADVANTAGES IN A TIME OF RISING FUEL COSTS AND OTHER ENERGY ISSUES
+294%2)/.´S 7EBASSESSOR‰ PROVIDES A ±GREEN² SOLUTION THAT ALLOWS STUDENTS TO TAKE THEIR COURSE EXAMS AT HOME 4HERE IS NO NEED TO SPEND THE TIME AND TROUBLE OF GOING TO A TESTING CENTER TO TAKE AN EXAM OR TO ½ND AND SIGN UP SOMEONE TO PROCTOR THE EXAM 4HIS PILOT REPRESENTS AN IMPORTANT DEMONSTRATION OF WHAT HAS BEEN PREVIOUSLY THOUGHT TO BE IMPOSSIBLE THE HIGH STAKES TESTING OF STUDENTS ANYTIME ANYWHERE EVEN IN A PERSON´S OWN HOME )T MARKS A WATERSHED MOMENT IN THE HISTORY OF TECHNOLOGY BASED TESTING WHETHER IN THE DISTANCE EDUCATION MARKETPLACE OR ANYWHERE ELSE HIGH STAKES TESTS ARE ADMINISTERED
ˆ +294%2)/. ¯ !LL 2IGHTS 2ESERVED
WWW KRYTERIONONLINE COM