Effectiveness Of Win Career Readiness Courseware – The Syracuse University Study In a recently released report 1, Dr. Tom Hadlick of Syracuse University confirmed that using WIN’s Career Readiness Courseware helped jobseekers prepare for and pass both the National Career Readiness and National Work Readiness Certificate exams. The study examined the preparation for these Certificates and the actual test-results in the Industrial Certification and Training (ICT) programs at five Workforce Investment Board career centers in Western New York.
Overview of WIN Career Readiness Courseware WIN’s Career Readiness Courseware® provides foundational training based on assessed skill needs and achievements for specific career interests and/or opportunities. The applied academics courseware consists of nine modules: Applied Mathematics, Reading for Information, Locating Information, Applied Technology, Observation, Teamwork, Writing, Listening and Work Habits.What makes WIN’s courseware unique and of great value to jobseekers nationwide, is that the instruction builds and certifies workplace skills, leading to more than 26 National Career and Work Readiness Certifications, two of which were the focus of the Syracuse study. Skill levels in WIN Career Readiness Courseware build sequentially, with each level incorporating the skills mastered at the previous levels. This skill metric allows WIN users to achieve mastery, not just test preparation, by reverse engineering the courseware based on specific learning objectives. The WIN Courseware quickly became viewed as an important enhancement to the ICT program through its Web-based, self paced, independent learning format which give program participants a tool for making skill gains in a timely and cost-effective manner. Each participant’s current skill levels were benchmarked to create a customized instructional program tailored to their educational needs. Through built-in learning management systems, the courseware facilitated monitoring of each participant’s progress. Pre-tests built into the WIN Courseware were used to help participants gain a clearer understanding of their current level of work ready skills and establish realistic career goals.
1
Hadlick, Tom. (February 2011). Using the WIN courseware to prepare for the National Work Readiness Credential. The Industrial Certification and Training Program: Syracuse University.
Page 1
PRIMARY WIN MEASURES
Partipants in the five Western New York Career Centers studied by Dr. Hadlick used three of the WIN Career Readiness courseware modules—Applied Mathematics, Reading for Information and Locating Information—far more extensively than the other modules. These three most frequently used modules map directly to the National Career Readiness Certification exam and partially to the National Work Readiness exam. Using these three skill areas from the WIN Career Readiness courseware as the focus of the study, the Syracuse researchers were able to put foundational skills training into a job-seeking context. As such, the Syracuse researchers set out the comparisons of skill levels in the WIN courseware and the percentage of jobs requiring a specific skill level, as well as the percentage of national test takers who scored at that skill level in the previous two years. Figure 1. Applied Mathematics Skill Level Comparisons As Figures 1 through 3 show, the Syracuse researchers quoted data from more than 540,000 Career Readiness courseware test records nationally with 4,600 job profiles taken over the past five years. On the charts, numbers in the center column represent the WIN Career Readiness skill levels; the bars to the left represent the percent of profiled jobs requiring that specific skill level; and the bars to the right represent the percent of testers who scored at that skill level. Figure 2. Reading for Information Skill Level Comparison
Page 2
Figure 3: Locating Information Skill Levels Comparison
Effectiveness of WIN Courseware COURSEWARE USAGE AND SKILL GAINS
Thie Syracuse study measured the impact of WIN courseware from May 2010 through December 2010. Researchers looked at whether the ICT participants chose to use the WIN Courseware, whether their math, reading and locating information skills improved within the WIN instructional modules, and whether the courseware helped participants succeed in the credentialing process. For the eight months of the study, the average time spent on the WIN Career Readiness courseware (among 170 active participants) was 6.5 hours.
The study gathered compelling evidence showing that use of WIN Career Readiness courseware, even for a relatively short period of time, enhanced an individual’s succesful completion of the ICT program and successful performance on national and state certification exams. Participants who spent more than two hours in the WIN Courseware performed much better in the assessment phase of the program. For example, the average math score on the Certification Exams for participants who spent less than two hours in the courseware was 67; the average score for those who spent from two to six hours in the courseware was 84—a 17 point differential.
Page 3
INCREASED SKILLS WITH WIN COURSEWARE
WIN Career Readiness courseware begins with a “pre-test” to determine participants’ current skill level and appropriate instructional modules. Participants are tested at the end of each lesson to determine their mastery in building individual skills. In this Syracuse study, participants’ gains were measured through comparisons of pre-test and post-test scores. Gains were noted both in terms of the WIN courseware and in terms of the US Department of Labor ‘s Educational Functional Levels (DOL EFL), used as a common measure for comparing skill gains using various national assessments and certification tools. The Syracuse study found dramatic changes for all participants across all three skills measured in the national work readiness certifications—Applied Mathematics, Reading for Information, and Locating Information. For all five sites over the eight-month study period:
Skill gains in Applied Mathematics (n=140) showed a 1.1 level increase. In terms of DOL EFL; this parallels improving skills from beginning basic education (BB) to low-intermediate basic education (LI).
Skill gains in Reading for Information (n=60) showed a 1.3 level increase. In terms of DOL EFL; this parallels improving skills from beginning basic education (BB) to high intermediate basic education (HI).
Skill gains in Locating Information (n=69) showed a 2.0 level increase. (No parallels are defined by DOL EFL for this measure).
A closer look at usage data suggests some participants, as indicated by the rapidity with which they completed pre-tests, used the WIN courseware to brush-up on skills they had previously acquired but which had deteriorated. Many other participants, however, as indicated by the large number of hours invested in the WIN Courseware, definitely aquired new skills that they previously did not possess.
VALIDATION OF INCREASED SKILLS THROUGH CREDENTIALING
The ICT program makes extensive use of both the NWRC, endorsed by the New York State Department of Labor, and the NCRC, more widely used nationwide. The NCRC is endorsed by several industry groups, including the National Association of Manufacturers, the Center for Energy Workforce Development and the National Center for Construction Education and Research. However, the NWRC is gaining momentum. Steck Vaughn, a subsidiary of Houghton-Miflin publishing company, and the exclusive publishers of the GED preparation material has secured the rights to promote the NWRC nationally and will be releasing NWRC test prep materials in 2011. The.NYS DOL has also recently launched an on-line NWRC test prep program. Page 4
Several of the Syracuse study’s conclusions directly support the use of WIN’s Career Readiness Software to prepare for the National Career Readiness exams. NWRC: Of the 88 participants who chose to enroll in testing for the NWRC, 59.1% (52) passed the assessment and received the NWRC.
Because the Mathematics portion of the NWRC test is most difficult for participants, additional data were analyzed for the math scores. Findings showed that participants who scored higher in the WIN Courseware also received a higher score on the Math portion of the NWRC Assessment. NCRC: Of the 46 participants who sat for the NCRC, 95% (43) passed. Of those, 41% received a Bronze (qualifying for 35% of all American jobs , 47% received a Silver (qualifying for 65% of American jobs), and 7% received a Gold Certificate (qualifying for 90% of American jobs).
Additional analyses also suggested that participants generally needed to reach at least a critical level (level 4) in the WIN courseware to assure passing the NWRC Math Assessment, whether they were using the WIN courseware to brush up on existing skills or build skills.
SUMMARY
The Syracuse study showed that WIN Career Readiness courseware builds foundational skills. Participants made significant skill gains within the WIN Courseware itself and these gains were validated using assessments associated with both the NWRC and the NCRC. The WIN Courseware was helpful to all and also proved extraordinarily useful for assisting English Language Learners to acquire work-ready foundational skills.
Page 5