The Peopleway® Learning Benchmark

The Peopleway® Learning Benchmark Peopleway® Press, September 2011 By: Lars Julin, Erik N. Hansen and Iben D. Jakobsen

This article presents the Peopleway® Learning Benchmark and selected key findings from the 2011 benchmark research. The purpose of the learning benchmark research is to provide organizations and learning specialists with measures to evaluate and guide their improvement of corporate learning, HR and talent development. The evaluation data of this report comprises more than 8 million data points, sampled within 21 languages for the period 2000 to 2010. All data has been obtained using the Peopleway® Approach. Tests have been conducted, following the same standards to secure quality, validity and reliability of the measurements. Measurements have been sampled using the Ease® evaluation solution.

We hope this research will inspire your work.

Lars Julin, CEO

CONCLUSION The merge of the 2000-2009 and the 2010 learning impact measurements data has consolidated the Peopleway® Learning Benchmark. The total 2000-2010 impact measurement data has been computed into two separate sets of Benchmarks. One set for Instructor-led training and one for e-learning. Both sets are summarized in the table below.

The Peopleway® Learning Benchmark Pre

Post

Job

Reaction

Learning

Transfer

Instructor-led

34.1%

77.1%

69.7%

4.05

65.2%

82.8%

e-learning

42.9%

80.1%

68.6%

3.83

65.2%

69.0%

Data from 2000-2010 is included.

Comparing the year-on-year samples, as well as comparing different learning approaches, the research reveals several interesting findings. A substantial increase is found in the average Pre-test scores from 2009 to 2010.

Moreover, as much as 15.6% of all the participants who attended training in 2010 had a Pre-test score >75% indicating that they initially did not need the training. For Instructor-led training, the percentage of participants taking training without having the need was 5.6% in 2009 increasing to 7.8% in 2010. For e-learning the share is substantially higher for both years though it remains basically unchanged. In 2009 it was 17.6% and in 2010 it is 17.4%.

Compared to Instructor-led training, it seems that a much broader group of participants enrol for e-learning, and this is causing a substantially higher level of waste. It is speculated whether the increase of participants attending Instructor-led training, without having a real need in 2010, is due to a decreased prioritization of HR

Copyright© Peopleway®

resources and thus less focus on needs assessment within the HR function caused by the global financial crisis.

When it comes to learning effectiveness, on average, Instructor-led training and e-learning seem equally effective in closing participants’ initial learning gap. The average learning effectiveness for the whole period is 65.2% for both types of learning. Comparing this research with a previous study by Julin et al. (2009), two tentative conclusions are drawn. Firstly, e-learning has a potential to perform better than Instructor-led training if assessment is done properly so that only participants with a real need engage in the e-learning. Secondly, Instructor-led training has the potential to match e-learning, if Continuous Improvement Processes (CIP) are used to improve communication and content of the training.

The present study also reveals a significant increment in the subjective rating of Instructor-led training from 2009 to 2010. Comparing e-learning and Instructor-led training, it is also found that participants in general are more satisfied when rating Instructor-led training than e-learning.

Initially, the study found that learning transfer is 13.8% lower for e-learning programmes than for Instructorled training. A further analysis reveals that the lower level of e-learning transfer is partly caused by a lower relevance, as more participants attends e-learning without having the need to do so. Another contributing factor might be a difference in management involvement. Namely that managers tend to be more involved and supportive in the case of Instructor-led training because of the higher level of training and higher cost in terms of salary and loss of contribution, and also because the employee will be absent from work during training. The specific impact of management involvement in e-learning is yet to be analysed.

Finally, a critically low level of e-learning transfer was observed in 2010. Only 57.1% of the learning acquired through e-learning in 2010 seems to be retained which is a very low level compared to the total 2000-2010 Benchmark of 70%. This indicates that the investment made in e-learning programmes has not been protected properly during the year 2010. Likewise, the 2010 decrease in transfer is assumed to be related to poor assessment and presumably also a decrease in managers’ involvement in implementation of e-learning on the job by employees in 2010.

Altogether this research supports findings by Brinkerhoff (2006) and Peopleway (2011) stressing the important roles of securing the Relevance of training, Assessing the needs properly, and growing Management Involvement as key drivers in developing High Impact Learning from programmes and organizations, protecting the investment in HR, corporate learning and talent programmes.

Copyright© Peopleway®

2

DATA The data for this research is collected from 2000 -2010 in various industries using the Peopleway® Approach.

It includes enterprises of all sizes within the public and private sector. All in all, 4,946 different courses, 25,829 different learning activities, and 273,298 participants have been measured and sampled within 21 languages.

In total, the research data represents 8,238,403 data points. Thousands of learning developers have been involved in producing, reviewing and validating the learning objectives and tests.

Two approaches have been used for aggregating the observations. The Benchmark analysis is based on an aggregation of responses at the level of single learning activities counting 25,829 activities in total. For the purpose of time series studies, the responses from participants attending the same e-learning at different dates have been interrupted into time series increasing the number of activities with the number of time series. For the purpose of regression analysis and correlation studies, on occasion aggregation has been made at the level of individual participants’ test scores.

APPROACH AND DESIGN The research data behind this article is a detailed sample of Learning and Impact Measurement data collected by using the Peopleway® Approach. The evaluation design behind the Peopleway® Approach is inspired by the evaluation frameworks of Donald Kirkpatrick, Jack Phillips and Robert O. Brinkerhoff. This design, among other things, includes: a Pre-test, conducted 0-14 days before training; a Reaction measurement, conducted immediately after training to measure participants’ perception of the learning activity; a Post-test conduced 0-14 days after training to measure the learning gain; and a Job-test, conducted 3-6 months after the training, to measure learning transfer and change of critical job behaviours. Learning capabilities are measured at different points in time depending on the variables. The Peopleway® Approach applies a design using test-methods prioritizing objective criteria rather than subjective criteria and effect measurement rather than single surveys.

Questionnaires have been validated by the learning provider, SMEs and/or by Peopleway.

BENCHMARKS This section presents results of the accumulated 2000-2010 impact measurement data. Moreover, the 2010 data sample is compared with 2009 figures.

The accumulated impact measurement data from the period 2000 to 2010 has been computed into two sets of Benchmarks. One for Instructor-led training and one for e-learning. As the number of blended solutions is not sufficient, these have been excluded from the Benchmarks. The analysis is summarized in Table 1, The Peopleway® Learning Benchmark.

The Peopleway® Learning Benchmark Pre

Post

Job

Reaction

Learning

Transfer

Instructor-led

34.1%

77.1%

69.7%

4.05

65.2%

82.8%

e-learning

42.9%

80.1%

68.6%

3.83

65.2%

69.0%

Table 1: Data from 2000-2010 is included. In all figures data has been aggregated by the level of learning activities.

Copyright© Peopleway®

3

Pre-test Comparing Instructor-led training and e-learning, a substantial and significant difference of 8.8% in participants’ Pre-test score is found. Several external factors may have had an influence on this difference. One important factor is most likely the higher level of heterogeneity among e-learning participants, which can be seen from the high standard deviation among e-learning participants. Also, it was found that more participants attending e-learning were overqualified (i.e. having a Pre-test score equal to or above 75%) than was the case for participants attending Instructor-led training.

Part of the explanation is due to the low marginal cost of e-learning, which generally promotes managers to enrol more employees including participants who do not really need the training. In addition, e-learning programmes are less frequently adapted to the changing needs of learning. Due to the higher costs of reengineering e-learning programmes, organisations tend to use e-learning programmes for longer periods of time than Instructor-led training, before revising or updating them.

Reaction It was observed that participants in general seem to be more satisfied by Instructor-led training (4.05) than by e-learning programmes (3.83). This difference between the average satisfaction scores has not been fully analysed yet but a correlation analysis between reaction and learning was found to support previous research by Julin et al. (2009), which found a zero-correlation relationship between learning and reaction, i.e. between objective and subjective measures.

Learning On average, the learning effectiveness measurement of Instructor-led training and e-learning was the same (65.2%). Thus, the two approaches seem equally effective in closing participants’ initial learning gap. However, in a previous study made on 2007-2008 measurements by Julin et al. (2009), the mean difference between the two approaches was substantial. In that study, the rate of e-learning effectiveness was more than 10% higher than for Instructor-led training.

The explanation of the difference between the two studies is twofold. Comparing 2007-2008 with 20092010 data, a 7.3% increase in the learning effectiveness of Instructor-led training appears. In contrast, a 3.6% decrease in the learning effectiveness for e-learning programmes is found. Analyses reveal that the decrease in e-learning effectiveness is primarily caused by the increase in participants’ heterogeneity in 2010. In 20072008, only 7.3% of the participants had a Pre-test score above 75% but in 2010 as much as 17.4% did not need training. When excluding participants with a Pre-test score above 75% in the analysis, the 2010 learning effectiveness matches the findings of the 2007-2008 study.

It is therefore concluded that when participants have a very high Pre-test score (i.e. they know too much) and do not close their remaining micro-learning gaps during training, it is most likely due to a disengagement of the participants. If the match between the need of the participant and the programme is too low, motivation and learning will decrease.

The 7.3% increase in learning effectiveness for Instructor-led training from 2007 to 2010 is interesting. This increment of the rate of learning effectiveness seems to be caused by the implementation of learning CIP in the years 2009-2010. In 2009, CIP was implemented in the Peopleway® Standards and Workshops, and in 2010 CIP was implemented as part of the evaluation solution Ease®. The question is why does CIP not influ-

Copyright© Peopleway®

4

ence e-learning as it seems to do for Instructor-led training? Most likely because CIP is about change, about continuously and rapidly changing according to needs. Changing content and communication of Instructor-led training is much easier, faster to implement, cheaper and thus applied more frequently than is the case for elearning.

Two tentative conclusions are drawn from the research on learning effectiveness (closing the learning gap): 1.

E-learning has the potential to perform better than Instructor-led training if assessment is done properly so only participants with a real need engage in the e-learning; and 2. Instructor-led training has the potential to match e-learning, if CIP are used to improve communication and content of the training.

Transfer Peopleway® has repeatedly found a significant correlation between learning retention and behavioural changes (3-6 months after training). From the research it is concluded that learning transfer in most cases can be linked to changes of specific critical behaviours. Learning transfer is therefore seen as an important indicator of successful implementation of training into the job.

In the benchmark research the learning transfer was substantially different for the two learning approaches. In the case of Instructor-led training, the rate of learning transfer is 82.8% and thus 13.8% higher than for elearning. We do not yet have a single answer to explain the overall difference in the learning transfer between the two approaches, but one hypothesis is related to relevance of the programme and relevance of the people participating and another hypothesis is related to management involvement. With respect to relevance, elearning is often enrolled to a large part of the organization, which makes it more difficult to address individual needs. Relevance is one major important factor with respect to securing a high learning transfer. With respect to management involvement, managers’ Pre and Post training engagement in employees’ learning and implementation is presumably less in the case of e-learning compared with Instructor-led training. Research by Peopleway (Julin et al. 2011) supports the work of Brinkerhoff et al. (2001), concluding that the managers’ involvement is critical to the employees’ training success. Julin et al. (2011) found that managers’ post-training engagement to employees improved their learning transfer by 14%. Further research is required to investigate the differences in management involvement in e-learning and Instructor-led training.

In the following section, data from 2010 is compared with data from 2009 to analyse short-range changes.

Table 2 presents the computed key figures from the year 2010 as regards impact measurement data. These figures have been compared with 2009 figures to identify year-on-year changes. Arrows in Table 2 indicate observed significant changes in four cases.

The Peopleway® 2010 Learning Benchmark Pre

Post

Job

Reaction

Learning

Transfer

Instructor-led

37.9% ↑

79.2%

71.8%

4.22 ↑

66.5%

81.9%

e-learning

54.5% ↑

84.1% ↑

71.4%

3.92

64.9%

57.0%

Table 2: Only 2010 data is included. In all figures data has been aggregated by the level of learning activities. Arrows indicate a significant change in scores between 2009 and 2010. Significance level: 0.05.

In 2010, 15.6% of all participants passed the Pre-test (score above 75%). In 2009, the share of participants who didn’t need the training was 14.4%. As can be seen from Table 3 below, the overall 2010 increment in participants passing the Pre-test but still attending training is caused by an increase within instructor-led

Copyright© Peopleway®

5

training. From 2009 to 2010, the share of participants passing the Pre-test but still attending e-learning is very high, but has changed only marginally from 17.6% in 2009 to 17.4% in 2010. The share for Instructor-led training is much lower, but has increased significantly from 5.6% to 7.8%.

Participants who passed the Pre-test 2009

2010

Instructor-led

5.6 %

7.8 %

e-learning

17.6 %

17.4 %

Table 3: In all figures data has been aggregated by the level of learning activities.

Altogether the increment of the Pre-test scores and share of participants attending training without having the need indicates that e-learning generally is enrolled to a target group being too broad, compared to Instructor-led training. Increasing resources spent on needs assessment of the e-learning participants generally produce a convincing business case. But the waste due to including participants without the need of the training for Instructor-led training can be seen as critical as is the case with e-learning. Instructor-led training is more expensive with regards to participants’ salary and loss of contribution because the training time is often 4 times that of e-learning. Therefore the 7.8% of participants in 2010 who attended Instructor-led training without having the need can be said to be at least as critical as the 17.4% share of participants attending elearning.

Post-test The e-learning Post-test score has increased both substantially and significantly from 79.2% in 2009 to 84.1% in 2010 (see Table 2). This change parallels the Pre-test score and therefore is not interpreted as an increase in learning effectiveness.

Reaction Furthermore,, the study reveals that there has been a significant increment in the subjective rating of Instructor-led training from 2009 (4.02) to 2010 (4.22), see Table 2.

Transfer Finally, the learning transfer analyses performed 3-6 months after the training indicate that Instructor-led training (81.9%) performed better than e-learning (57%) in 2010. These findings support previous research by Julin & Ejlskov (2009) concluding that Instructor-led training performs significantly better in terms of learning transfer compared to e-learning.

More importantly, in 2010 the e-learning transfer was only 57%, which is far below the Peopleway® Benchmark of 70%. This indicates that e-learning programmes in 2010 have been far less successful in transferring learning into the job. In other words the investments made in e-learning programmes have not been protected effectively in 2010. A full understanding of this decrease is yet to be analysed but one factor seems to be related to the Relevance of training. If participants who passed the Pre-test (score >75%) are excluded, the learning transfer rate increases to 64.3%. If the cut-off point for excluding participants is set to 60%, the learning transfer rate reaches the Peopleway® Benchmark of 70%. It is concluded from the study that at least a partial explanation of the decrease in the overall 2010 learning transfer rate is due to the previous discussed increase in Pre-test score, caused by improper needs assessment and/or too broad enrolments of the programmes in 2010.

Copyright© Peopleway®

6

Generally this Benchmark research supports the findings by Brinkerhoff (2006) and Julin et al. (2011) stressing the important role of securing the Relevance of training, Assessing the needs properly, and growing Management Involvement as key drivers in developing High Impact Learning from programmes and organizations.

LITERATURE Brinkerhoff, Robert O. & Apking, Anne M. (2001), High Impact Learning: Strategies For Leveraging Performance And Business Results From Training Investments. New York.

Brinkerhoff, Robert O. (2006), Telling Training’s Story: Evaluation Made Simple, Credible, and Effective (1st edition) Berrett-Koehler Publishers, Inc..

Julin, Lars & Nedergaard, Erik (2009), Zero correlation between Satisfaction and Learning! Peopleway® Press.

Julin, Lars & Ejlskov, Kasper (2009), Impact of E-learning and Classroom training. Peopleway® Press.

Lars Julin, Erik Nedergaard, Jette Nissen and Tanja Nyborg Sørensen (2011), Case study: Organisational drivers,

reinforcing the implementation of learning at Vestas. Peopleway® Press.

Copyright© Peopleway®

7