Data Fluency in Learning | July/August 2020

J U LY/A U G U S T 2 0 2 0

FOLLOW THE EVIDENCE-BASED DATA | 18 Become an L&D Detective to Uncover Value

DATA SCIENCE AND L&D | 30 Harness Learning Data in a Meaningful Way

THE COST OF SCRAP LEARNING | 40 Use Predictive Analytics to Increase Learning Transfer

BUSINESS

PERSPECTIVES

ON

MANAGING

WORLD-CLASS

TRAINING

KEN TAYLOR

FROM THE EDITOR

DATA FLUENCY ACROSS THE BUSINESS

To navigate and withstand the changes businesses are seeing over the course of the COVID-19 pandemic, organizations must be able to effectively use data to remain innovative and agile. With so much uncertainty for the future, leaders must harness the data available within their organization to make informed business decisions.

EMPLOYEES ACROSS THE ENTIRE ORGANIZATION MUST LEARN TO SPEAK THE LANGUAGE OF DATA.

This issue of Training Industry Magazine focuses on data fluency and leveraging learning analytics to enhance business performance. It is no longer enough to be aware of our data and where to source it; we need to champion the use of that data to improve business outcomes.

As leaders, we need to consult the data available to us during times of crisis and allow those metrics to lead us through the decision-making process. Understanding data is no longer just a leadership exercise.

Employees across the entire organization must learn to speak the language of data and apply it to improve productivity and performance.

This requires a shift in mindset for many of us in the workplace. We must tap into our innate curiosity to find the correct data that will lead to significant insights. There is an abundant amount of data available today, and one major challenge is finding the right data to solve the right problem. Making connections between business milestones and the data available in your organization can serve as guideposts for your evolution toward a more data-driven organization.

Speaking of data, Training Industry has been conducting an ongoing pulse survey to better understand the effects of the pandemic on the learning and development (L&D) industry. The study has revealed the top four challenges facing L&D professionals, which include transitioning to remote learning, maintaining employee motivation, increasing cybersecurity, and adhering to new compliance regulations and restrictions.

Our study revealed that more than half of training professionals and organizations are in flux right now – whether retooling current programs or developing new solutions in response to new challenges.

To provide training leaders the context they need on their own journey to the new normal, we included an infographic demonstrating key findings from the survey on page 4 of this issue.

The pandemic has created a multitude of business challenges, but it has also created an opportunity to reimagine learning at our organizations. By leaning on data, we can eliminate ineffective programs and processes and replace them with more efficient and creative alternatives.

As always, we love to hear your thoughts on the insights and perspectives shared in the magazine. Please feel free to send any suggestions for future editions of Training Industry Magazine for us to consider.

Ken Taylor is the president and editor in chief of Training Industry, Inc. Email Ken.

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

|3

LEADING LEARNING DURING T COVID-19 has jolted businesses into changing behavior holistically, and learning and development (L&D) departments are uniquely positioned to be at the center of that change.

As social distancing, sheltering-in-place and virtual gatherings became the norm, Training Industry opened an ongoing pulse survey to L&D professionals to track and understand the rapidly developing effects of the pandemic on our industry.

PLANNING AND PROGRAMS 9%

34%

31%

Have a plan, confident about executing it

MORE THAN

BUT ONLY

have a plan in place.

Feel it will be executed.

Have a plan, not confident about executing it Developing or revising a plan No plan

26%

CURRENT PROGRAMS

are retooling or repurposing their programs.

FUTURE PROGRAMS

are continuing their programs as is.

are delaying the rollout of new programs.

TAKE PART IN OUR ONGOING PULSE OF L&D RES |4

THE COVID-19 PANDEMIC LEARNING LEADERS’ TOP CHALLENGES

ENGAGEMENT

COMPLIANCE

100% REMOTE

SECURITY

SPENDING AND DELIVERY 10% 24% of those whose spending has been affected are spending more now.

25%

41%

6%

22% AFTER COVID-19

BEFORE COVID-19

32% Instructor-led training Virtual instructor-led training

41% eLearning Other

STRATEGIC ALIGNMENT don’t know what their stakeholders need from them right now.

Maintaining clear communication with employees and stakeholders will ensure L&D is creating the right solutions to the right problems.

SEARCH AND RECEIVE UPDATES ON FUTURE DATA. T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

|5

CO N T E N TS

TA B L E O F VOLUME 13

|6

I

ISSUE 5

I

JULY/AUGUST 2020

FEATURES

18 FOLLOW THE EVIDENCE-BASED DATA

18 23 26 30 34 38

30 DATA SCIENCE AND L&D

40 THE COST OF SCRAP LEARNING

BUILDING THE CASE FOR IMPACT INVESTIGATION

By Kevin M. Yates

Leverage evidence-based data analytics to demonstrate the value of your learning solutions.

USING ANALYTICS TO PRIORITIZE LEADERSHIP DEVELOPMENT INITIATIVES

By Stephen Jeong, Ph.D., Stephen Young, Ph.D., and Cheryl Flink, Ph.D.

Keep your learning solutions learner-centric with these actionable tips.

HOW DO WE MEASURE OUR D&I EFFORTS?

By Brynne Hovde

Add these diversity and inclusion metrics to your toolkit for analytics that drive results.

DATA SCIENCE: HOW IT MAKES L&D INTEGRAL TO BUSINESS SUCCESS

By Tom Ridley

Use analytics to make strategic decisions aligned to business outcomes

MEASURING THE IMPACT OF A BAD BOSS

By Paul Leone, Ph.D.

Is poor leadership affecting your peoples’ happiness and productivity?

FILLING IN THE GAPS: THE MOST IMPORTANT DATA AND ANALYTICS CAPABILITIES FOR TODAY’S COMPANIES

By Mike Galvin

Close the tech skills gap by upskilling workers in today’s most critical data capabilities.

40 44

PINPOINTING THE UNDERLYING CAUSES OF SCRAP LEARNING

By Ken Phillips

Eliminate scrap learning with a data-driven approach to learning transfer.

DATA-DRIVEN PRACTICES FOR CONTENT DEVELOPMENT

By Sarah Dreyer, Meghan Cipperley and Merav Yuravlivker

Create impactful training content with lessons learned from data science.

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

|7

IN THIS ISSUE

THOUGHT LEADERS

3

FROM THE EDITOR

By Ken Taylor

Employees across the entire organization must learn to speak the language of data.

10

GUEST EDITOR

13

SCIENCE OF LEARNING

15

By Dr. Sydney Savion

Develop systems of prevention and preparedness rather than recovery in times of crisis.

By Srini Pillay, M.D.

Drive business and learning outcomes with brain-based learning and metrics.

PERFORMANCE MATTERS

By Julie Winkle Giulioni

Amid uncertainty, greater employee engagement can be traced to the learning function.

17

BUILDING LEADERS

By Sam Shriver and Marshall Goldsmith L&D must identify the data that matters from the data that doesn’t tell you much at all.

53

WHAT’S NEXT IN TECH

55

SECRETS OF SOURCING

57

By Stella Lee, Ph.D.

Gain actionable insights on where to start on your journey toward data fluency.

By Doug Harward

Cultivate data fluency to identify your organization’s most pressing training needs.

LEARNER MINDSET

By Michelle Eggleston Schwartz Harness the learning spirit of your employees amid the coronavirus pandemic.

INFO EXCHANGE

48

CASEBOOK

50

GLOBAL OUTLOOK

Follow Amtrak’s journey from highly technical classroom learning to engaging computer-based training.

Ensure global training efforts are clear and consistent with these eLearning solutions.

CONNECT WITH US

|8

58

CLOSING DEALS

59

COMPANY NEWS

1 (866) 298-4203

Discover how Guild Education and Entangled are supporting workers displaced by COVID-19.

Keep up with the latest in the training industry by reading news from the last quarter.

editor@trainingindustry.com

TrainingIndustry.com

ABOUT OUR TEAM

STAFF CHIEF EXECUTIVE OFFICER Doug Harward dharward@trainingindustry.com

ASSOCIATE EDITOR Sarah Gallo sgallo@trainingindustry.com

DESIGNER Kellie Blackburn kblackburn@trainingindustry.com

EDITOR IN CHIEF & PRESIDENT Ken Taylor ktaylor@trainingindustry.com

ASSOCIATE EDITOR Hope Williams hwilliams@trainingindustry.com

DESIGNER Alyssa Alheid aalheid@trainingindustry.com

EDITORIAL DIRECTOR Michelle Eggleston Schwartz meggleston@trainingindustry.com

CREATIVE DIRECTOR Amanda Longo alongo@trainingindustry.com

ADVERTISING SALES sales@trainingindustry.com

MANAGING EDITOR, DIGITAL Taryn Oesch toesch@trainingindustry.com

DESIGNER Mary Lewis mlewis@trainingindustry.com

EDITORIAL BOARD JUDI BADER, CPTM Senior Director of Learning Arby’s Restaurant Group

MATTHEW S. PRAGER, CPTM Executive Training Manager U.S. Government

MICHAEL CANNON, M.ED. Senior Director, Head of Learning & Development Red Hat

MARC RAMOS Vice President, Chief Learning Officer Sitecore KELLY RIDER Vice President, L&D Content Strategy & Experience SAP Learning & Development

MEGAN CASADOS Director of Training DISH

DR. SYDNEY SAVION General Manager, Learning Air New Zealand

BARBARA JORDAN, CPTM Group Vice President, Global Learning & Development Sims Metal Management

KERRY TROESTER, CPTM Director, North America Sales Training Lenovo

CATHERINE KELLY, MA, BSN, RN, CPTM Director of Learning Programs Brookdale Senior Living

NATASHA MILLER WILLIAMS Head of Diversity & Inclusion Ferrara

SHIREEN LACKEY, CPTM Talent Management Officer, Office of Business Process Integration Veterans Benefits Administration

KEE MENG YEO Adjunct Professor Grand Valley State University & Davenport University

LAURA MORAROS Global Head of Sales Learning Facebook

A S B P E Aw a r d s o f E x c e l l e n c e

A | S | B | P| E Fostering B2B editorial excellence

American Society of Business Publication Editors

2018 Cross-Platform Package of the Year Top 10 Award

Training Industry Magazine connects learning and development professionals with the resources and solutions needed to more effectively manage the business of learning.

SUBSCRIPTIONS ELECTRONIC:

Sign up at TrainingIndustry.com to receive notification of each new digital issue.

PRINT:

Print copies are available for purchase at magcloud.com for $15.95.

ARTICLE REPRINTS To order reprints of articles, please contact Training Industry at editor@trainingindustry.com.

PUBLISHER Training Industry Magazine is published bi-monthly by: Training Industry, Inc. 6601 Six Forks Rd Ste 120 Raleigh, NC 27615

SCOTT NUTTER General Manager, Research, AQP & Development Delta Air Lines A Z B E E S

MISSION

A | S | B | P|E Fostering B2B editorial excellence

American Society of Business Publication Editors

2017 National

ONLINE Award Winner

|9

DR. SYDNEY SAVION

GUEST EDITOR

LEARNING TRANSCENDENCE: PREPARING FOR CRISIS

Since the beginning of the 21st century, we have experienced five global pandemics: severe acute respiratory syndrome in 2003, H1N1 influenza pandemic in 2009, Middle East Respiratory Syndrome in 2012, Ebola virus in 2015 and now COVID-19 in 2020.

However, the coronavirus pandemic has been the most crippling by far. As this crisis threatens to devastate business operations, companies must not only improve business continuity plans (BCP) but also drive a resilient learning culture that can weather disaster.

There is a pressing need to reimagine a system of prevention and preparedness rather than recovery. Companies that have discovered learning transcendence – the proven capability to bolster simplification, harness modernized technology and employ workforce cross-skilling during a bull market – will protect their people and corporate performance against unknown events.

CROSS-SKILLING BOOSTS PRODUCTIVITY AND FOSTERS THE AGILITY COMPANIES NEED TO PIVOT IN TIMES OF CRISIS. A business continuity plan has been the traditional system of prevention and recovery from potential business threats, including supply chain interruption and loss of or damage to critical infrastructure such as machinery, technology network resources, buildings or employees.

However, learning is often overlooked.

| 10

If the BCP is supposed to ensure that assets and personnel are protected and can continue functioning in the event of a disaster, why isn’t it working? Can learning transcendence ensure it will fulfill its intended purpose?

Much like the science-fiction film “Transcendence,” COVID-19 has taught us that it is not enough to create a BCP with the best of intentions. We must verify the application is reasonably stable. In the film, Dr. Will Caster and his team build a living computer in which they upload a digitized form of Caster’s consciousness to outlive the death of his body. In his digital afterlife, Caster builds a virtual utopia focused on advancements in medicine, biology, energy and nanotechnology. As he grows more powerful, Caster’s team grows suspicious of his intentions. To prevent potential harm, the team creates a computer virus to destroy Caster, as well as the utopic digital civilization.

Although the film is fictional, it illustrates the necessity to have a fully vetted and sound plan in place.

According to a study conducted by Mercer, 51% of organizations around the world do not have a BCP in the case of emergencies or disasters. Yet, roughly 31% of organizations have a plan that has not yet implemented. So, where do learning professionals start?

To reliably inoculate against company collapse and loss of performance and morale, organizations must create an organizational learning utopia by integrating three advancements into the flow of work.

1. BOLSTER SIMPLIFICATION Steve Jobs once said, “Simplicity is the ultimate sophistication.” He simplified products by getting laser-focused on their essence and eliminating unnecessary features. Hence, there are no instructions in the Apple iPhone box; you simply need to follow the intuitive on-screen instructions.

Simplification means streamlining processes by eliminating unnecessary steps and automating to drive business efficiency. Rewarding efforts to map processes and workflows to simplify unnecessary tasks yields a culture of endless curiosity, prioritizing the efficiency of organizational processes.

During a crisis, your organization operates from a position of lower cost-basis, heightened transparency, increased time, human resource and asset management efficiency, productivity, and minimized risk related to process and workflow complexities.

2. HARNESS MODERNIZED TECHNOLOGY The COVID-19 crisis has exposed risks associated with legacy technology systems, especially learning platforms.

It has proven that ignoring outdated technology until it is broken does not bode well in a crisis and is harmful to a company’s brand and customer satisfaction. To get and stay ahead with learning, put cloud adoption transformation and investment at the center of your business strategy to continuously adapt and modernize technology.

During a crisis, your organization should operate from a secure, cloudonly position by investing in prudent solutions that are cost-effective, quicker to test and implement, and better suited for learning and business continuity.

This prevents disruptions caused by having to patch or replace deteriorating legacy systems, software and platforms while striving to meet ever-increasing employee and customer experience expectations in an uncertain market.

of service. To ensure employees’ skills stay refreshed, implement recurrent job rotation.

3. EMPLOY CROSS-SKILLING

TEST PREVENTION PLAN CAPABILITIES

Conduct a skills risk analysis of businesscritical units that drive the highest value proposition. Then, begin cross-skilling select workforce populations for your new business strategy.

For example, contact centers are crucial during this crisis, yet many companies have not been able to respond quickly. Create a squad of cross-skilled employees to perform both customer-facing tasks and tasks enabling them to shift amid changing business needs to prevent discontinuity

During a crisis, your organization operates from a position of developing a sustainable workforce of transferrable business critical knowledge and skills.

Furthermore, cross-skilling boosts productivity and fosters the agility companies need to pivot in times of crisis.

To test prevention capabilities, it’s important to mimic real-life scenarios.

Extended reality (XR) – a mixed reality environment intended to simulate physical presence in virtual scenarios – can do that. The military has used this type of application for decades, from disaster-preparedness to analyzing military maneuvers.

This allows organizations to monitor, track and evaluate performance with the utmost

detail. It is a proven approach to fortify learning and assess the strength of your prevention plan before, during and after a crisis.

PRACTICAL WISDOM The notion of learning transcendence may be new, but the practical wisdom drawn from research and best practices are not. These insights can help create a workplace and workforce where curiosity, knowledge sharing, reflection, digital acuity and simplicity are fostered, forging a resilient company able to quickly adapt to unplanned events.

Learning transcendence has become both a human and an organizational need. It’s time to move beyond an analog BCP of basic recovery needs and strive toward consistent betterment of organizational learning and preparedness.

Dr. Sydney Savion is general manager of learning at Air New Zealand. Email Sydney.

boost your at-home advantage WITH

TRAINING INDUSTRY WEBINARS From emerging learning technologies to bridging skills gaps to creative leadership strategies, Training Industry webinars take a deep dive into the business of learning so that you, as a learning leader, can help your learners swim.

Browse free webinars or on-demand presentations. www.trainingindustry.com/webinars

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

| 11

We have spent the last decade researching how organizational size, structure and industry affect training performance.

Read the report for actionable practices that will take your training from good to great.

READ THE NEW REPORT

SRINI PIL LAY, M.D.

SCIENCE OF LEARNING

LEARNING DATA: FACT OR FALLACY?

Learning data provide an objective way to measure learning outcomes.

When we correlate learning data with business goals, metrics tell us whether the learning was relevant and effective.

Yet, we should examine learning data cautiously, as several factors impact what we interpret from the numbers.

Eventually, you must interpret data for individual learning. Moreover, statisticians should work with social psychologists, brain scientists and business leaders.

1. No two brains are the same: If I compare two different interventions’ impacts on business outcomes, does the group outcome tell me which intervention is worth using?

Yes. Learning is dependent on context. A tense work environment can impact how people learn. The brain is sensitive to context, and if the environment changes, the learning outcome.

Group averages give us a population statistic and an average impact factor.

However, there is no way to apply these metrics to any individual outside of the original sample.

For example, 80% of participants reported one learning solution was more effective than another intervention. However – only knowing eight of 10 people benefitted from one intervention over an alternative – you cannot predict which solution will be more effective for learners outside of the original population.

Moreover, the population changes every time you experience turnover.

Old datasets no longer apply. The new hire could be part of the 80% or part of the 20%. Individuals and brains differ.

Solution: Continuously measure learning’s impact on business outcomes in various teams and at different points throughout the year. Additionally, pay less attention to double-blind, controlled trials and more to prospective data with multiple datapoints.

2. Context matters: Can the same intervention have a different impact at a different time?

THE WAY PEOPLE FEEL CHANGES HOW THEY LEARN AND CAN IMPACT BUSINESS OUTCOMES. Some people may learn well under stress while others don’t. Even if the intervention is the same, the next cohort will learn differently and in the context of their unique environment.

Solution: Test the same learning under different contexts. Also, cultivate a learning-friendly environment, and pay attention to changes.

Additionally, team learning can occur collectively. Collective intelligence depends on social sensitivity, turntaking and valuing diverse perspectives.

Collective intelligence differs from the average or highest intelligence in a group. In a group setting, our brains learn differently, and soft skills can support that learning.

Even for the simplest processes, soft skills matter. Learning how to use a program or how to execute a process can be a simple and straightforward process. But consider how much easier it is to learn when learners’ minds are at ease.

Solution: The way people feel changes how they learn and can impact business outcomes. Invest in quality soft skills training; do not skimp on your soft skills learning solutions.

Data is neither fact nor fallacy. A crosssectional datapoint is a measure of learning at a specific time. Informed examination of data can help you make meaning of analytics as you guide your organization toward data fluency.

3. Soft skills matter: Can soft skills be correlated to business outcomes?

When integrated into the unique social context of an organization, brain-based learning can help training professionals leverage accurate metrics that drive business and learning outcomes.

Anxiety can help or hurt learning, and uncertainty can do the same. Knowing how to manage anxiety and uncertainty can improve learning. Therefore, it’s important to build a learning platform for soft skills.

Dr. Srini Pillay is the CEO of NeuroBusiness Group. He is a Harvard trained psychiatrist and neuroscientist, and on the Consortium for Learning Innovation at McKinsey & Company.

Email Srini.

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

| 13

JULIE WINKLE GIULIONI

PERFORMANCE MATTERS

LEARNING: THE ULTIMATE BUSINESS CONTINUITY STRATEGY

Recent events have challenged every part of our personal and professional lives. Yet, the disruption has served at least one positive purpose: To reveal the depth of what learning and development (L&D) is capable of delivering.

L&D has long been recognized as a value-adding function of the business, but it’s taken a global crisis to demonstrate that learning can operate at an even more strategic level. In fact, many organizations have discovered that learning is a powerful business continuity strategy.

Previously, leaders and employees have enjoyed just one dimension of the contributions learning can make to an organization: the ability to disseminate information, build competencies, and magnify capacity. In the process, L&D has contributed to organizational culture and results. However, this was just the tip of the iceberg.

CAPACITY CONTINUITY Enter COVID-19 and the upending of business as usual. Organizations were forced to pivot overnight in many cases.

Employees needed to adapt to different ways of interacting quickly. Processes changed and new systems were instituted. L&D responded by supporting reskilling, business transformation, and shifts to digital channels and strategies.

But in many organizations, L&D has done much more, distinguishing itself and offering services associated with supporting continuity of additional elements required for business continuity.

HUMAN CONNECTION CONTINUITY Decades ago, researchers Deci and Ryan determined that connection is one of the three powerful psychological needs people bring to the workplace.

This awareness has recently become increasingly acute. In a study I conducted with Advantage Performance Group, respondents reported that isolation was the second most challenging dimension of working from home.

Interestingly, L&D has found itself at the center of data gathering efforts around this issue. The training director in a professional services firm I work with conducted weekly well-being surveys. The L&D manager for a manufacturing client didn’t collect new data but mined employee assistance program trends. Other organizations used their employee monitoring software to identify individuals and parts of the organization that might not be engaging or communicating sufficiently.

Organizations then used available data to create strategies for keeping staff connected; many have leaned heavily into L&D activities. From targeted webinar series to development coaches and partners, L&D took action to promote learning that offers human connection continuity in service of business continuity.

ORGANIZATIONAL CONNECTION CONTINUITY While maintaining vital connections among employees is important, ensuring strong connections between the individual and the organization is also necessary to safeguard the strength of a

business during both normal and unusual times. L&D professionals facilitated this through developmental experiences that reinforced the big picture and learning that reminded people their role in the mission, vision and needs of customers.

L&D HAS FOUND ITSELF AT THE CENTER OF DATA GATHERING EFFORTS. CONTRIBUTION CONTINUITY Learning has played a role in helping to maintain the well-being of employees and their ability to contribute despite volatility and uncertainty. Training capable of capturing learner attention delivers more than learning outcomes; it delivers opportunities for people to help regulate their nervous systems by offering something productive to focus upon.

Additionally, many organizations have expanded their offerings related to mindfulness, work-life balance and leadership skills to support the holistic well-being of teams.

In the months and years to come, we’ll likely be able to trace greater engagement, discretionary effort, retention and loyalty back to the efforts of L&D. And in this way, learning will have become the ultimate business continuity strategy.

Julie Winkle Giulioni has 25 years of experience working with organizations worldwide to improve performance through learning. Email Julie.

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

| 15

CAN THE CERTIFIED PROFESSIONAL IN TRAINING MANAGEMENT (CPTM™) CREDENTIAL HELP YOU DRIVE BUSINESS RESULTS DURING ECONOMIC UNCERTAINTY?

Take this quiz to see if the CPTM™ credential is the missing link to keeping you and your business on track for success.

TAKE THE QUIZ

SAM SHRIVER & MARSHALL GOLDSMITH

BUILDING LEADERS

DATA FLUENCY AND YOUR HIGHEST PROBABILITY OF SUCCESS Probability has long been the distinguishing factor between physical and behavioral sciences. In physics, force equals mass times acceleration – every time! If you know how much a baseball weighs and how fast it travels when your granddaughter throws it while playing catch in the driveway, you can accurately determine the force the ball had when it shattered the windshield of your car.

Consider the difference between that scenario and determining the relationship between your organization’s leadership training efforts and business outcomes.

You must first come to terms with the fact there is no uniform standard of measurement. Frustrating though it may be, evidence produced to evaluate the impact of training in the past may not be relevant predictors moving forward.

In a world with an intimidating abundance of information available, your first challenge is to distinguish the data that matters from the data that might be interesting from the data that really doesn’t tell you much.

From our perspective, we consider data fluency the ability to connect the dots along a continuum flowing downward from results through habits and leading indicators to the learning itself.

RESULTS What is the bottom-line impact of training?

This has always been a tough question for learning leaders. Primarily because there are many intervening variables, and they are next to impossible to isolate. It’s such a tough question that many consistently high-performing organizations avoid

it altogether. The costs associated with confirming their belief that learning is embedded in achievement of corporate goals are simply too high.

In short, we agree. If you are in a set of circumstances where you must sit down with your chief financial officer and definitively project a return on this year’s investment in training, may the force be with you!

HABITS Charles Duhigg discussed the potential of keystone habits in his best-selling book “The Power of Habit.” A keystone habit is a measurable pattern of behavior with the clear potential to increase the likelihood of achieving desired results. Duhigg asks readers to take safety as an example. If you can get employees in a manufacturing organization to become laser focused on safety, it impacts communication, accountability, efficiency and more.

Clearly establishing, continuously improving and relentlessly measuring the connection between the learning function and keystone habits defines the depth of data fluency in an organization.

LEADING INDICATORS Long before a change in behavior becomes a habit there is a predictable ebb and flow in achieving mastery. That development is primarily governed by reinforcement provided by the manager of the trainee.

The quality of that reinforcement is a product of the manager’s proactive inclusion in the learning event.

Organizations that are serious about connecting learning to habits collect data that measures the degree of synergy between the primary stakeholders who navigate that ebb and flow (trainer, trainee and manager).

HOW LEARNERS FEEL AFTER THEY HAVE COMPLETED TRAINING CAN TELL US A LOT ABOUT THEIR PROPENSITY TO CHANGE BEHAVIOR. LEARNING It appears, the more sophisticated our measurement capabilities have become, the greater the tendency to discount the impact of Level 1 feedback. We would caution against that. How learners feel after they have completed training can tell us a lot about their propensity to change behavior and develop new habits moving forward.

It will never be as straightforward as a baseball crashing through a windshield, but if learners feel good about the learning event and are connected to a stream of ongoing reinforcement intended to build keystone habits, the learning function will have the highest probability of ongoing success.

Marshall Goldsmith is the world authority in helping successful leaders get even better. Sam Shriver is the executive vice president at The Center for Leadership Studies. Email Marshall and Sam.

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

| 17

| 18

Found at the scene

Cab Driver

Poison pills

L&D detectives search for clues and use data to uncover learning’s influence on behavior, performance, actions and business goals. By learning how to build the case for impact investigations with qualifying criteria and impact facts, you too can be an L&D detective.

CRITERIA FOR IMPACT INVESTIGATIONS How do you know which programs, courses or learning solutions deserve an impact

investigation? Here is the realty: You cannot measure everything nor should you. You can, however, determine when to conduct impact investigations based on these five qualifying criteria.

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

| 19

Foot length 9.8in.

DOES IT HAVE SENIOR LEADERSHIP PRIORITY OR SPONSORSHIP? L&D has the potential to be an undeniable driver of performance for achieving

and positioning of learning as one of the critical drivers for achieving a specific business goal. When there’s agreement that learning is part of the winning strategy, it deserves an impact investigation that reveals fulfillment of purpose.

There is an exception. There may be a small target audience with high expectations for impacting business goals. While the target audience is not large, conduct an impact investigation considering expectations for impact.

ARE THERE SIGNIFICANT INVESTMENTS FOR MONEY AND TIME?

business goals. Achieving business goals is the priority of the chief executive officer; the priority of senior leaders should be the priority of the learning team.

If a senior leader is lending his or her influence by championing a training or talent development program, an impact investigation is necessary and appropriate. When executive leaders use their voice to support and promote learning aligned to business goals, evidence for impact is expected. When the C-suite puts faith in the learning team for being a critical contributor to business outcomes, the impact of learning deserves an investigation.

IS IT ALIGNED TO A BUSINESS GOAL? What does alignment between a business goal and learning mean? It means there’s purposeful connection and agreement that learning (in partnership with other contributors) has the potential to move the business toward a specific target or priority.

It means that learning has purpose and intention.

My work as an L&D detective is primarily focused on training solutions that purposefully exist to help organizations grow, succeed and win. That means there is agreement

| 20

DOES IT HAVE SPECIFIC TARGETS FOR PERFORMANCE OUTCOMES? A measurable performance outcome is not, “You will know…,” “You will learn…,” or “You will understand…”

Those are traditional learning objectives. A measurable performance outcome is how knowing, learning and understanding appear on the job.

Performance outcomes are indicators for behavior and actions. Learning solutions with specific performance outcomes that achieve business goals is the point of intention for purpose and alignment.

If the learning solution has intentional targets for performance outcomes – and not learning objectives –it qualifies for an impact investigation.

DOES IT HAVE A LARGE TARGET AUDIENCE? The larger the number of people whose performance is needed to achieve the business goal, the greater the number of people who are targeted for learning. When the target audience is large, there’s higher visibility and expectation for impact.

Learning solutions and programs with significant depth, breadth and reach deserve an impact investigation.

There are two ways to think about investments for training and learning: Monetary costs for delivery and employee hours engaged in learning.

The higher the investments, the greater the expectation for return on investment.

Learning projects with substantial investments in time, money or both deserve an impact investigation.

DOES THE TRAINING OR LEARNING SOLUTION NEED TO MEET ALL FIVE CRITERIA FOR AN IMPACT INVESTIGATION? A question I am asked often is, “Do all five qualifying criteria have to be met for an impact investigation?” There is no one right answer. Learning teams and their business partners will have to decide for themselves.

For me, the first three criteria are absolutely necessary for deciding whether to conduct an impact investigation. I am flexible with question 4, and I would need to be convinced not to consider question 5. Careful consideration must be given to the time it takes to solve measurement mysteries.

The more criteria met, the more likely it is that you have the facts, evidence and data to prove training is worth the time and effort required.

COLLECTING IMPACT FACTS There are three facts that are critical to an impact investigation. These are facts about business goals, performance

Suspect smokes trichonopoly cigars

Suspect 6 ft tall, square-toed boots. requirements to achieve those goals and evidence for performance outcomes.

These three facts will inform decisions for designing an impact-based training or learning solution and will also satisfy two of the five criteria for conducting an impact investigation.

WHAT IS THE BUSINESS GOAL? This is the most important question to ask.

Learning fulfills the highest purpose when it helps people use their performance to achieve business goals. How can L&D help people achieve business goals if we do not know what the goal is?

the person you are interviewing cannot answer questions regarding performance, use a job or role description as a clue to help the interviewee describe the information you need.

business results. You will also need to get agreement from business partners and stakeholders on the facts and data you will use as evidence for impact before the training is launched.

The answer to this question is the foundation on which impact investigations are built. If you cannot discover specific performance requirements for achieving business goals, the impact investigation will fail and there is no direction or intention for training and learning.

Training and learning fulfill its purpose when it builds or sustains performance outcomes that help people achieve business goals.

I want to highlight that you are asking this question and questions about business goals and performance outcomes before learning is delivered. You are not asking questions about “training.” The answers, however, inform decisions for training and learning with the highest potential for impact.

Curiosity should drive L&D detective work, and that is not simply curiosity about how many people liked training, the consumption of learning by modality or the number of people who finished the program. I’m talking about curiosity regarding how a learning experience influenced thought, how that influence in thought triggered a change in behavior or actions, how that change in behavior or actions shows up in performance, and how that performance measurably impacts business goals.

As an L&D detective, not collecting facts about business goals leaves a hole in the investigation. You need specifics, for example: • The business is pursuing 30% revenue growth by expanding into new markets.

• The company wants to decrease production errors 5% by investing in new technology.

• Employees resignations are up 5% and exit surveys show lack of manager engagement is a contributor, so the goal is to improve manager-employee engagement scores by two points.

Collect facts about business goals – not training goals. You cannot solve the measurement mystery without knowing the answer to this question. Understanding the business goal connects the investigation to performance.

WHAT ARE THE PERFORMANCE REQUIREMENTS FOR ACHIEVING THE BUSINESS GOAL? You will need descriptions of actual performance, behaviors and actions that move the business goal to target. What does the behavior look like on the job?

How do you know it when you see it? How does knowing or understanding show up in performance, behavior and actions? If

L&D DETECTIVE WORK: NOT EASY BUT POSSIBLE

WHERE AND WHAT IS THE EVIDENCE THAT SHOWS PERFORMANCE IMPACT? This is my favorite part of the investigation. This is where you discover data and signals for the presence of behavior, performance and actions that achieve business goals. This is the part of the investigation where you identify – before the learning solution is launched – the sources of evidence, facts and data that reveal the impact of learning on performance.

Look for evidence like performance ratings, customer satisfaction scores, business performance metrics, financial performance, production key performance indicators, workforce data, employee engagement, or anything that gives signal to connecting the dots between performance outcomes and

L&D detective work should be centered on searching for facts, clues, evidence and data that reveal learning’s impact on business goals. If you are not conducting an impact investigation that leads to solving the mystery of how training and learning influence performance and how performance impacts business goals, you are solving for the wrong mystery.

L&D detective work can be difficult, but difficult does not mean impossible.

I challenge anyone engaged in L&D detective work to be led by curiosity, persistence, commitment and belief in the possibility of using facts, evidence and data to answer the question, “What is the impact of learning?” There is an answer.

Kevin M. Yates is widely known in the L&D industry as the “L&D detective” for his work in measuring efficiency, effectiveness and outcomes for training using facts, evidence and data.

Email Kevin.

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

| 21

USING

TO PRIORITIZE

LEADERSHIP DEVELOPMENT INITIATIVES

By Stephen Jeong, Ph.D., Stephen Young, Ph.D., and Cheryl Flink, Ph.D.

Organizational responses to the COVID-19 pandemic have taken a variety of forms. Many organizations have gone through the process of downsizing their workforces; others have made significant changes to their business models. Still others have focused on rebuilding their brands after public scrutiny of their initial response to the pandemic. The choices made and the pivot to the future have resulted in a variety of new challenges and opportunities. It has never been more important to invest in leadership development as organizations normalize and grow.

Minimizing barriers and maximizing opportunities for growth will require leaders and teams to forge an organizational culture equipped to build the new future. Human resources (HR) budgets for investing in these components have always been limited – and may be even more so now. The business case for such investments must be compelling, clearly showing the path to a tangible return. How can HR leaders create that compelling business case? By using analytics to link individual leader, organizational capacity and culture data to tangible business outcomes.

A FRAMEWORK FOR DEMONSTRATING ROI In 2019, research conducted by the Center for Creative Leadership (CCL) found that global leaders identified big data and analytics as the most important

trend to impact their business over the next five years. Organizations apply data analytics to nearly every aspect of their business – e.g., marketing spend, operational efficiencies and supply change management. However, analytics are rarely applied to leadership development investments. Why? These four challenges are common barriers: 1.

HR leaders may not have a clear understanding of organizational goals and how leaders impact those goals.

2. Leadership and organizational data can be difficult to obtain.

3. Analytics applied to that data can be too simplistic, lacking the power to find statistical relationships.

4. Organizations lack systems and processes for acting on the data.

To overcome these challenges and build a strong business case for investing in leaders, teams and organizations, managers must follow these steps:

CLARIFY ORGANIZATIONAL OBJECTIVES Clarify your organization’s goals.

Leadership development investments must focus on helping the organization achieve its goals — from building the competencies of individual leaders to team efficiencies to organizational capacity for digital

transformation. If you are unclear about your organization’s goals, review your organization’s objectives and key results (OKRs), management business objectives (MBOs), annual reports, operating plan, and mission statement. You will need to crisply articulate what the organization wants to accomplish, the metrics for measuring success, and how investing in leaders achieves business outcomes.

LEADERSHIP DEVELOPMENT INVESTMENTS MUST FOCUS ON HELPING THE ORGANIZATION ACHIEVE ITS GOALS.

GATHER RELEVANT DATA An analytics model for leadership development investment requires at least one of these three things: • Data about your leaders (e.g., competency ratings or nine box scores).

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

| 23

• Data about organizational capacity (e.g., ratings of company’s focus on innovation and risk).

• Data about employee experience or organizational culture (e.g., employee engagement surveys).

FOSTER ACCOUNTABILITY

BY TYING GOALS TO PERFORMANCE RATINGS, BONUSES OR PROMOTIONS.

same store sales, margin improvements or employee turnover.

At the end of the day, you will need to create a purposeful measurement system. Predictive modeling is not about assessing trends; it is about identifying the leader and organizational levers that drive or inhibit outcomes relevant to your business. Leveraging relevant data will create a compelling business case.

USE ANALYTICS TO PREDICT WHAT INVESTMENTS MATTER MOST

In addition, you must have access to business metrics — like year-over-year

Predictive models pinpoint statistically significant drivers of business outcomes.

Models can also be prescriptive, indicating how much a business metric will improve by increasing performance on key drivers. It’s not magic, but these models do require sophisticated data science to produce. Such models are commonly

Innovation Driving Results

Operational Decision Making Facilitating Change

FOUR PERFORMANCE OUTCOMES

Communication Adaptability

Business Acumen Cultural Interpersonal Effectiveness

Leading through Vision & Values Customer Focus Building Trust

Empowerment

Building Org Talent Coaching & Developing Building Strategic Work Relationships

Low Impact

High Impact

PROMOTE

CONVEY INSIGHTS AND TAKE ACTION Even with a robust model, the process for building your business case can break down if you don’t translate insights into action. HR and learning leaders can do this in four ways:

Listening MONITOR

HOW CLIENT IS DOING ON 360 RATING (PERCEIVED BY ALL OTHERS) (AVERAGE RATING)

MAINTAIN

FOCUS

PROMOTE Communication

Building Collaborative Relationships Quality Care Demonstrates Vision Results Orientation

Executive Image

Change Management Leading Employees Knowledge of Job, Business

Low Impact

FOCUS

Driving Execution Influence

| 24

Armed with these evidence-based insights, HR and learning and development (L&D) practitioners can take a strategic role in helping their organization achieve results. Predictive models can make a credible case for investing in leadership development, because it is now linked to a tangible ROI. As a leader in your company, you can embrace the latest methodologies to help your organization normalize and grow amid uncertainty.

FIGURE 2

LEVEL OF IMPACT ON EMPLOYEE ENGAGEMENT AND VBP

High Impact

FIGURE 1

used in most business organizations but are rarely used to understand the return on investment (ROI) of leadership development investments.

MONITOR

MAINTAIN

ORGANIZATIONAL PERFORMANCE (AVERAGE RATING)

• Create simple visualizations. Simple visualizations like heat maps make it easy to understand the impact of key drivers.

3. Analysis: A predictive analytics algorithm identifying eight competencies that impacted job performance.

The Approach

• Create a playbook. Link the drivers to recommended actions. For example, if risk-taking is a key driver of innovation and the organization is risk-averse, you will need to develop a risk orientation across the organization.

4. Taking action: A heat map visualizing which competencies matter most to job performance; the leadership development journey was redesigned to drive better talent outcomes.

2. Data: Individual leader assessment data on 5,500 (individual, peer, boss) and their employee engagement survey scores.

• Communicate results up and down the organization. Everyone responsible for the business outcome must see the results and create action plans.

• Create accountability for change.

Foster accountability by tying goals to performance ratings, bonuses or promotions. A predictive model cannot create change — it can only point to where change needs to occur.

TWO CASE STUDIES The following case studies illustrate how predictive analytics prioritized leadership development investments using the fourstep approach above:

FORTUNE 500 RETAILER The Challenge A Fortune 500 retailer sought to design and validate a new high-performance competency model as part of their succession planning strategy. The vice president of talent development wanted an objective approach to reducing the organization’s 18 competencies to a manageable subset validated as key drivers of business outcomes.

The Approach 1.

Business outcome metric: The client identified four job performance metrics as the critical business objective – annual performance evaluation, 9-box ratings, rate of promotion and a “model director” classification reported by HR business partners.

2. Data: 18 competencies collected from over 600 leaders, including direct report, boss and peer assessments.

Figure 1 shows the results of the predictive model. The four quadrants show a matrix of performance on the competencies (low vs. high) and impact. Those in the blue circle show the eight most impactful drivers — with five of those having low performance scores and requiring change.

Drivers of business impact highlighted important differences between directors with high and low competency scores.

Those in the top 20% vs. those in the bottom 20% had: • 7% higher annual performance scores.

• 32% better 9-box scores.

• 85% more steps per year since assessment.

• 60% greater chance of being rated a model director.

By incorporating this new insight into their competency model, the organization streamlined both its efforts aimed at improving competencies.

NORTH AMERICAN HEALTH CARE SYSTEM The Challenge This North American-based healthcare system experienced persistent low employee engagement scores and below-target value-based purchasing (VBP) metrics (a subset of metrics that all U.S.-based healthcare organizations are evaluated against). VBP metrics gauge patient outcomes and can include such measures as length of stay, patient satisfaction and number of safety incidents. The company wanted to identify the drivers of employee engagement and VBP metrics to pinpoint where to focus their leadership development.

1.

Business outcome metric: Employee engagement scores and VBP metrics.

3. Analysis: A predictive model identifying six competencies impacting employee engagement and VBP metrics.

4. Taking action: A heat map (see Figure 2) showing which competencies mattered most to the business outcome metrics.

Developing programs aimed at improving the six competencies. All leaders were advised to complete self-directed, online development courses relevant to the six areas identified.

As a result of their highly focused approach to leadership development, the client achieved an 8.5% improvement in the VBP metrics, resulting in a $4.22 million reimbursement from the federal government. Impressed with these powerful results, senior management significantly increased their investment in leadership development.

CONCLUDING THOUGHTS HR and L&D leaders must find ways to ensure that their workforce remains competitive and ready to pivot at a moment’s notice. Prioritizing leadership development investments will be critical to the future. They must take an innovative approach to justify budgets and create new approaches to talent and organizational change. Predictive analytics provide the objective, ROI-focused approach to win budget and resources for investing in your organization’s future.

Stephen Jeong, Ph.D., is senior research faculty of insights and impact and leadership analytics at Center for Creative Leadership (CCL). Stephen Young, Ph.D., is manager of leadership analytics at CCL. Cheryl Flink, Ph.D., is vice president and leads the global leadership research, analytics, and insights and impact functions at CCL. Email the authors.

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

| 25

How Do We Measure Our D&I Efforts?

By Brynne Hovde “You can’t measure this stuff!” We hear it all the time, and we can sympathize. Diversity and inclusion (D&I) professionals are frequently working with limited resources and limited measurement training, and the expectations set by data-driven stakeholders can feel impossible.

| 26

Rest assured; you can measure this stuff. Whether you’re starting out on your D&I measurement journey or you’re looking to add to your toolkit, here are three best practices for D&I metrics that work:

1

Redefine “Data” at Your Organization

If your stakeholders are looking for charts and spreadsheets, let’s do it.

But let’s remember that employees’ experiences and stories can show progress – or lack thereof – in ways no survey or demographic analysis ever could. In D&I work, you need both quantitative and qualitative data to get a complete picture. If you’re patting yourself on the back for having a leadership team that’s 51% women, and women elsewhere in the organization are still getting talked over in meetings, then your work is not done.

On the quantitative side, employee and leadership demographic information is important when setting diversity goals. We always recommend putting numbers around what you’re trying to achieve, even if it feels uncomfortable at first. Look at the demographic makeup of the area you’re based in, the areas where your employees live or your customer base, and set concrete goals around representation. For inclusion and belonging to work, consider what options you have for an internal survey to capture employee sentiment. As your resources allow, try to gather data before D&I initiatives launch to establish a baseline, at several points midstream and then on a regular cadence – for example, biannually – to track progress over time.

On the qualitative side, you may have to adopt the mantra: Qualitative data is data! It can sometimes be difficult to persuade stakeholders on the merit of focus group or interview data, but in this kind of work, it’s incredibly important.

The Nova Collective recently worked with a large, multinational company where we conducted interviews with key stakeholders, and nearly every single stakeholder uttered the phrase,

“I know our survey data says we’re doing fine, but…” That “but” was critical for identifying the real work that needed to be done, and without it, some harmful cultural patterns would have persisted unchecked.

Experiences matter, and they can’t all be captured in a survey.

2

Work With What’s Already Available

We did a research study in 2019 to assess the landscape of D&I work in the U.S., and we found that only 65% of D&I practitioners have direct control of their budget, and 7% have no dedicated funding at all. That means most practitioners aren’t in a position to hire a big research firm and do a comprehensive study. So, what can you do for little to no money? Get creative, and take a look at what’s already available.

The two most common spots D&I practitioners find existing data are human resources (HR) talent management systems and employee engagement surveys. Even if HR isn’t tracking anything but gender and ethnicity, that’s a start, and you can formulate some meaningful goals around that. Depending on the talent management system your organization uses, there may be an easy way for HR to start collecting additional demographics for new hires.

EXPERIENCES MATTER, AND THEY CAN’T ALL BE CAPTURED IN A SURVEY. Employee engagement surveys can provide a wealth of data because they are often already asking the questions you want to answer. Survey administrators at your organization

should be able to provide a list of questions asked and the existing data for those questions, and you can set goals around specific measures relating to D&I. If your organization uses a third-party vendor for the survey, the vendor may even be able to suggest some measures for you to use or use the questions to create a D&I index for you.

KNOW YOUR DATA: THREE TIPS FOR SUCCESS 1. Averages aren’t enough. If the majority of your workforce holds dominant identities (e.g., white, male, straight, etc.), averages are going to be misleading.

2. It gets worse before it gets better. D&I work is hard work. It pushes people, it makes people uncomfortable and – if you’re doing it right – it gets messy.

3. Something is better than nothing. Measurement matters, particularly when D&I work is seen as a nice-to-have rather than the business imperative it is.

If you’re interested in holding focus groups or doing interview research, check in with the team who coordinates training and professional development.

Frequently, these folks have training in facilitation, and you may be able to get on their calendar to do some informal research with employees.

One thing we caution people against is attempting to do focus groups yourself.

Focus groups are a fine art. With sensitive topics such as D&I, you can end up doing more harm than good if the conversation isn’t facilitated well.

3

Focus on Behaviors

In D&I measurement, we frequently talk about the difference between what people know, feel, believe and

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

| 27

do. In the end, what people actually do makes the biggest difference in your workplace. Measuring all four of these responses to your work is important.

For example, if you’re doing some basic training, it matters that people leave the room understanding what diversity is. The real work comes in when you ask people to take action, and that’s why we recommend focusing on behavioral measurement.

BEST PRACTICES FOR USING SURVEYS •

Assume you have work to do.

Frame your questions to find out what the issues are, not if issues exist. They exist.

•

Avoid the fluff. It’s tempting to add a few “easy win” questions to make your leadership feel good.

Don’t. Having an incomplete or unrealistic picture of your culture puts the business at risk.

•

Good data is worth fighting for.

In D&I work, you have to pick your battles. When you’re getting pushback on the questions you want to ask, pick that battle. Prioritize getting good data.

With behavioral measurement, begin with the goals you’re trying to achieve.

We always recommend tying your D&I goals directly to your organization’s business goals so the line of sight between D&I work and business success is clear. When you consider your goals, think about what needs to happen to achieve them. What do people in your organization need to do? For example, if you have diversity goals around hiring, what specific things are you asking recruiters

| 28

to do? And how will you know they’ve done it? Likewise, if you’re working to create a more inclusive culture, what are the behaviors your people managers need to actively demonstrate? Again, how will you know if they do?

The strongest behavioral measurements are those behaviors you can independently verify.

For example, are all of your job descriptions being written using the new guidelines for inclusivity you developed? That’s something you can confirm. But can you independently verify that managers are actively and intentionally making space for all voices in meetings? Probably not.

Sometimes, self-report will have to suffice. And sometimes, within the scope of your resources and support, true behavioral measurement isn’t possible. That’s where other know, feel and believe metrics can fill in the gaps.

No matter what kind of data you collect and how comprehensive – or relatively nonexistent – your measurement system is, keep a few quick things in mind as you look at your data: •

•

practitioners to make this clear to stakeholders from the beginning, so there are no surprises later.

•

Something is better than nothing. You may not have an employee engagement survey, support from HR, funding or anything else mentioned here.

We get that. Whatever you can do – do it. Measurement matters, particularly when D&I work is seen as a nice-to-have rather than the business imperative it is. Do what you can to prove your worth.

IN THE END, WHAT PEOPLE ACTUALLY DO MAKES THE BIGGEST DIFFERENCE IN YOUR WORKPLACE.

Averages aren’t enough. If the majority of your workforce holds dominant identities (e.g., white, male, straight, etc.), then averages are going to be misleading. You need to weight survey responses to ensure all voices are represented.

You can measure this stuff, and with the right approach, you will measure it. Getting stakeholders on board can be difficult for many reasons, but data to back up your work shouldn’t be one of them.

It gets worse before it gets better. D&I work is hard work. It pushes people, it makes people uncomfortable and, if you’re doing it right, it gets messy. Trust in the process, and don’t shake up or abandon your strategy at the first sign of falling numbers.

We always encourage D&I

Brynne Hovde is a co-founder of The Nova Collective – a black-owned, women-owned diversity and inclusion firm in Chicago, Illinois. Brynne is also on the board of Race Conscious Dialogues, and a commissioner and advisor to the board of trustees on race and diversity and inclusion in her community. Email Brynne.

TAILORED LEARNING SOLUTIONS

UNDERSTANDING FIRST A strategic, consultative approach to learning is one that understands your key challenges—and delivers outcomes to support your critical objectives. This is what targeted, data-driven learning solutions are all about. This is how Raytheon Professional Services will impact the performance of your people and your business.

rps.com @RaytheonRPS Raytheon Professional Services

© 2018 Raytheon Company. All rights reserved.

Data Science HOW IT MAKES L&D INTEGRAL TO BUSINESS SUCCESS

BY TOM RIDLEY

| 30

When this article was conceived, we were living in a world where businesses operated as they chose, we visited friends and family frequently and publicly, and our reliance upon the digital world was largely governed by how far we wished to integrate. Amid the global pandemic, the need for greater understanding is more important than ever. To survive, the decisions businesses make must be rooted in a healthy degree of logic.

Maturity report revealed, “51% [of training professionals] say they cannot use data effectively due to lacking in-house data skills.”

In 2006, Clive Humby, a UK mathematician, coined the phrase, “Data is the new oil.” 2006 seems a long time ago. Smartphones as we know it didn’t exist, social media was in its infancy and YouTube was celebrating its first birthday. Even in an age where data was much less available, it was still seen as a game-changer. Then a 2018 Towards

Shopping is a great example. Google and Amazon track your browsing history – on-site and across the web.

They link with social media, even examining our friends and family, and then push targeted products and services. These companies have not ignored the “data wave.” Rather, they are riding the crest and seeing massive growth.

How can there be a severe lack of development in an area that permeates all walks of life? The use of technology, particularly those driven by data, has increased exponentially to the point that it is almost impossible to escape the collection of and interaction with data.

TO SURVIVE, THE DECISIONS BUSINESSES MAKE MUST BE ROOTED IN A HEALTHY DEGREE OF LOGIC.

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

| 31

However, data in learning has predominately been about completion rates, time in content, hours delivered and test scores. This data, while useful, isn’t the full picture. Furthermore, it’s not the type of information that excites stakeholders or can be used to make strategic decisions. This data ensures learning and development (L&D) stays useful but not integral to business success.

points inform your understanding of learners. To continue, let’s say we are tracking 100 different data points concerning performance with this piece of technology. Why so many?

L&D must model the best practices of other departments and industries to ensure their value is seen. To have 51% of L&D functions’ ineffectively leveraging data is unsustainable, especially as COVID-19 places unprecedented challenges upon businesses. L&D does have the ability to drive change, galvanize a workforce and deliver business results. The question is how can L&D better connect to meaningful data and how can they use it to articulate their value?

• Additional data points are useful to other business units when trying to make decisions that impact the organization.

BEYOND COMPLETION RATES L&D must stop seeing data as something only they create themselves.

Vast quantities of data exist outside of learning. Human resources (HR) systems, email, chat functions, social events, technology platforms and even shared personal data can be used to broaden our understanding of our audience. These may be familiar to many, but we often use them in a simplistic way, as part of a gap analysis or posttraining key performance indicators.

For example, employees are not using a piece of technology correctly, training is implemented and the same metric can be used to see if things have improved.

But what is the value to the business here? Is it sustainable or tailored to the learner? This is where data science comes in.

In the example above, we are only looking at two data points: use of a piece of technology and completion of training. What about other factors such as time in role, geographical location, learning style, past training and historical engagement with information technology (IT) support? All these data

| 32

• A greater number of data points provides detail on what impacts learning and what combination of factors is best for success.

• More data directly ties L&D to other business functions.

It is naive to believe that we can create a training program for the learner to consume and that it alone will move the needle. Access to learning outside of L&D is greater than ever, and ignoring this external influence reduces our credibility.

Let’s assume that there is a simpler version of the software available. If you track that those with access to the simpler software first have a greater speed to competency when introduced to the full software, you could build this into the instructional design of the program. Maybe you notice that younger team members pick up the technology quicker, or that workers in one location are using a frequently-asked question (FAQ) guide and are having fewer issues.

These data points all add the value. It is not simply about the creation of content.

L&D can evidence the way people are learning and harness that to build better programs and achieve greater return on investment (ROI).

L&D is often siloed away from the organization and, despite its best attempts to partner, is often held at arm’s length. However, this does not have to be the case. Continuing with the previous example, let’s see how additional data can benefit other business functions while building L&D’s value.

“Simple” version of the software. IT can make more informed decisions when

buying licenses for the product. They learn that simple versions of software will suffice for a while, and they do not need to purchase the full version from day one, resulting in instant cost saving.

The department may also experience quicker installation and configuration times. IT can also secure budget for a software that is being well used while making L&D integral to building the business case.

Younger team members. If a younger audience can get to grips with the tool faster, they may not need the same quantity of training. Reducing time can increase speed to competency and gets employees back to the business.

L&D MUST STOP SEEING DATA AS SOMETHING ONLY THEY CREATE THEMSELVES.

FAQ usage. This piece of data could wield so many possibilities. This is evidence of collaboration aiding in the learning process, as well as prompts further questions. Can you provide access to this document for other learners? Can you use FAQs as a way to communicate future changes in the

software? There are benefits across business functions, as well as broader strategies for collaboration.

COMMON CHALLENGES IN LEARNING ANALYTICS What is stopping learning from being better with data? Usually, learning leaders say, “We don’t have access to the data,” or “This is very complicated, and we don’t have the capability.” These may resonate and if so, all is not lost!

You may find it’s an issue of legality, as data and access to it have increasingly become a sticky subject. GDPR legislation and scandals with personal data have only fanned the flames.

However, we are not talking about the abuse of data nor are we often looking at an individual’s data. When assisting business strategy, we look at trends in data collection. Anonymized data can be used extensively, and we can account for different laws across locations. A proper discussion about data’s use and

its benefit to the business usually results in a positive outcome.

Second, more and more platforms are making it possible to share data with other platforms, even if not created by the same company. HR systems can talk to a customer relationship management (CRM) system. That CRM can then talk to a survey tool that talks to an e-commerce platform. The data is there; it just needs to be directed to the right place.

Although it may look complex, the use of dedicated data visualization tools can make light work of sourcing and processing data. These tools are designed to collect vast quantities of data and allow you to sort it in a way that makes sense. They simplify the complex and present information in an engaging form.

It is also likely that you have an inhouse data capability. Utilize them; data scientists love a new data source.

Vendors are also building their own data teams now and offering this

service to their customers. Remember, senior management will be looking at business functions that present data in this way. L&D does not want to be behind the curve.

SUMMARY Data is the new oil, and L&D is perfectly positioned to harness this data in meaningful ways. It does not need to be complex – only thoughtful. L&D can be the arbiter of change and success for the business but also the mechanism by which the capabilities of an organization can be assessed. Learning has a unique opportunity to understand the needs of the business, implement a solution, track the routes to success and help the business make strategic decisions. All based upon evidence provided by data.

Tom Ridley is the UK head of sales and learning consultant at Valamis. He has worked in L&D in various roles for the last 15 years. Increasingly, his focus has been on technology and its links to data and its interpretation. Email Tom.

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

| 33

By Paul Leone, Ph.D. If your boss called in sick tomorrow, would you be happy or sad? Does the thought of no boss in the office or no boss at your afternoon meeting instantly get you pumped? In a recent study, a sample of 341 LinkedIn participants were asked whether they were happier and less stressed when their boss called in sick, was absent from work or went on vacation – 88% said yes!

This means that – instead of providing comfort, security and confidence – most bosses have the exact opposite effect. Why is this unnerving? By its very nature, “good” leadership means that a boss protects and provides advantages to followers that make them feel more secure when the boss is around. So why are so many of us reveling the days they go missing? They may not be nightmare bosses that bully and paralyze their employees, but maybe more of us have “bad” bosses than we like to think.

The question is – even if your boss is a little bad – what is that costing you and

| 34

your organization? And what are some of the costs that aren’t so obvious? We all know about absenteeism and turnover, but what about hidden costs that may be more pervasive yet harder to quantify?

You might think you’re fine because your boss just “rubs you the wrong way,” but much like a skin condition, we know that chronic “rubbing” day after day doesn’t allow for healing and brings on more inflammation. Whether it’s mind or body, we need less aggravation and more soothing support to stay happy, healthy and productive at work.

IMPACT ON MENTAL HEALTH As a society, we’ve come to recognize how important healthy relationships with our family, friends and partners are for our overall happiness, but we often overlook or underestimate the powerful effects of our relationships at work. The truth is most of us spend more hours at work trying to navigate and nurture our relationships with our bosses and peers

than we do at home with our families.

If this full-time relationship with your boss isn’t working or is ripe with conflict and resentment, there will inevitably be direct and dire consequences to your overall feelings of happiness, sense of security and perception of self. Studies show these constructs can decrease anywhere from 10% to 86% depending on the level of daily interaction with a bad boss.

IMPACT ON PHYSICAL HEALTH Beyond mental health, bad boss relationships can wreak havoc on your physical state. When we find ourselves in conflict, our sympathetic nervous system is activated, and our bodies go into a natural fight-or-flight response.

This physiological response is the exact same response we had tens of thousands of years ago if we were getting chased by a saber-toothed tiger. Do you know

• Reproductive organs: Increasing likelihood of sexual dysfunction and infertility in men and women.

• Lungs: Exacerbating symptoms of asthma and any obstructive pulmonary disease.

• Skin: Worsening skin problems, such as acne and psoriasis.

Over the course of just one year, costs can easily exceed $5,000 in treatment or medication per direct report. Multiply that by five direct reports per boss and you’re already up to $25,000 per bad boss – not counting long-term health problems. Multiply that by the number of bad bosses in one large organization, and you’re now in the millions.

IMPACT ON WORKPLACE PERFORMANCE Beyond mental and physical damage, what about the obvious and devastating effects a bad boss has on your work performance? What could a bad boss be doing to things like employee creativity, motivation and productivity? how much energy you’d expend if you were to experience even a fraction of this response every day at work?

Imagine your body has to ramp up for conflict every time you go to a meeting or talk to your boss. These smaller, more frequent sympathetic activations over time are worse for your health than the big bad tiger! This negative and chronic stress response will affect your: • Immune system: Making you more susceptible to viral illnesses and infections.

• Heart: Raising your risk of hypertension and chances of a stroke or heart attack.

• Stomach and digestion: Increasing stomach acid and problems, such as gastroesophageal reflux disease or peptic ulcers.

• Blood and liver: Producing extra blood sugar to give you a boost of energy (to fight) increasing your risk of developing Type 2 diabetes.

IMPACT ON CREATIVITY AND INNOVATION Your decision to channel all your creative and innovative thinking into making your company better will only happen when: • You feel you have a future with the company, and you want to be there long enough to see your ideas become reality.

• You genuinely want your company to succeed, and you want to see everyone in the company reap the benefits.

• You feel confident and safe enough in your environment to suggest new ideas – even if some end up failing.

If these three criteria are fertile ground for creativity, do you think bad bosses get the creative juices flowing? Exactly the opposite. Bad bosses create a negative psychological state that stifles creativity

and innovation. Instead of thinking about helping the company succeed, employees are busy thinking of ways to handle a bad boss. Can you imagine how many great ideas throughout history have been stifled by bad bosses?

IMPACT ON MOTIVATION AND PRODUCTIVITY When it comes to our work, we build a pretty big part of our identity by how hard we work and how we “show up.” A bad boss can quickly drain motivation and nosedive productivity. Impact on productivity research suggests that employees with bad bosses waste between 10% and 52% of their time at work. Instead of working, they spend time withdrawing, avoiding the boss, networking for support and ruminating about their situation. And the time wasted when they’re physically at work is just the half of it.

Bad bosses create a negative psychological state that stifles creativity and innovation.

The other half of the productivity loss – conservatively estimated at $150,000 per boss per year – comes in the more direct and quantifiable time away from work, increasing absenteeism, sick leave and eventual turnover. When employees do show up, some recent workplace productivity statistics found: intentionally decreased their

48% work effort.

intentionally decreased the

38% quality of their work.

lost work time avoiding

63% the boss.

said that their commitment to

78% the organization declined. admitted to taking their

25% frustration out on customers.

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

| 35

Can you imagine how these numbers may affect the bottom line? If we did a rough calculation of time wasted and a conservative cost estimate, we easily reach an average of $75,000 per direct report. Multiply that by the amount of direct reports per bad boss (let’s assume only five again) and you’re already up to an average of $375,000 per year.

While the bad bosses might be telling themselves they’re commanding higher performance for the sake of the business, they’re actually having an opposite and negative effect on productivity and organizational performance. Bad bosses prevent good work from getting done and cause confusion for their direct reports. Employees become exhausted, disillusioned and burnt out. Because of the mess they create, bad bosses cost businesses billions of dollars in lost productivity every year.

IMPACT ON EMPLOYEE TRAINING

boss, training is never going to have the intended effect.

In my research on the training climate over the past 15 years, one consistent finding has been that trainees reentering bad post-training environments will reap less benefits from the training than those who go back to supportive bosses. In fact, differences were so pronounced that the group with supportive managers achieved a positive return on investment (ROI) while the other group had a negative ROI. Having a bad boss can literally make or break training investments and prevent your development. Depending on the cost and the number of participants, I’ve seen anywhere from $1,000 to $20,000 lost in training dollars for all employees under one bad boss.

THE BOTTOM LINE ON BAD BOSSES

If employees are under the influence of a bad boss and experiencing lower motivation, engagement and performance, what does L&D do to close this gap? They give employees more training. But if you have a bad

A bad boss can wreak havoc on people and businesses. Each one can cost us our health and happiness, as well as the company he or she works for over $500,000 every year. Direct reports of bad bosses spend every day anticipating future conflicts and ruminating over previous ones. Employees’ motivation, creativity and performance are devastated, costing them years of

INCLUDED/MONETIZED

NOT INCLUDED/MONETIZED

• Short-term health cost

• Long-term health cost

• Lost productivity at work

• Mental/happiness cost

• Time away from work

• Lost creativity/Innovation

• Turnover

• Stalled career growth

• Lost training dollars

• Bad company image

| 36

career growth. At home, their stress and unhappiness take an unavoidable toll on their personal relationships. Beyond the mental upheaval, they can suffer significant health problems, eventually shortening their lives. Beyond the conspicuous bad bosses, there are those that influence so subtly that you may not even realize why you’re unsatisfied, discouraged and unhappy on your job.

A bad boss can quickly drain motivation and nosedive productivity.

With bosses having this much sway over our mental, physical and professional health, it’s no wonder we’re sensitive to their tones, moods and attitudes. Perhaps that’s why most people – even if they don’t have a really horrible boss – would rather just spend the day without one.

Dr. Paul Leone is an industrial-organizational psychologist with over 16 years of experience as a measurement consultant for top organizations around the world. He is a sought-after thought leader and the founder of MeasureUp Consulting, where he helps companies measure the impact of their training initiatives. Email Paul.

SEE THE DIFFERENCE

Situational LeadershipÂŽ can make in your organization!

Learn More At situational.com

Filling Gaps in the

The Most Important Data and Analytics Capabilities for Today’s Companies

By Mike Galvin

Every day – and every moment – large amounts of diverse data are being generated and collected by companies around the world. This data has a lot of potential business value if companies can appropriately utilize it. Naturally, this growth in data is causing an evolution of certain skill sets in order to meet a rapid and ever-changing market demand, and it’s important that everyone learns at least the basics of data analytics.

In essence, the data is there; the problem is that you need employees who have the skills to leverage that data and use relevant tools to get it done efficiently.

This skill gap applies to everyone within an organization. All roles are changing to have basic data literacy requirements, even if you’ve never previously worked with data. And for those who do regularly work with data, there’s a growing need to stay up to date on tools and techniques as the field evolves.

So, what are some of the most in-demand data analytics skills? Let’s review three of the most popular training topics I’ve seen while working with corporate clients in recent years.

In-demand Data Science and Analytics Skills Data Literacy Data literacy is the ability to read, analyze, work with and argue with data.

| 38

Basic data literacy skills are needed in every role no matter the organization.

Like other fields of study, there can be varying levels of data literacy. At a basic level, everyone should know common terminology, types of data, where data comes from, how data is used, and basic analysis and visualization techniques.

This results in employees being able to: • Speak fluently about data analytics and its uses.

• Identify business opportunities and make more data-driven decisions.

• Ask questions and evaluate proposed solutions.

• Increase understanding of how data is being used in their organization and how decisions are made.

Python Python is today’s fastest growing major programming language and is a favorite with data practitioners. In fact, according to Burtch Works, a leading executive recruiting agency specializing in data science and analytics, Python is now the programming language most preferred by data professionals. Most major companies worldwide are using Python, and an increasing number of companies of all sizes report using it as their primary programming language.

There are many reasons for this widespread adoption. Python is open

source and has a large community of active users. Its flexibility makes it useful in multiple domains, and it has a large number of open source libraries that provide ready-to-go solutions for many common problems. Not to mention, Python has simple syntax and readability, so the learning curve is low. All of these things combined make Python an attractive language for anyone working with data within a company.

Basic data literacy skills are needed in every role no matter the organization.

Machine Learning Machine learning is a subset of artificial intelligence that involves computers learning from data. It’s been around for decades but has increased in popularity over the past 20 years due to larger volumes of data, improvements in hardware, and the development of tools for model development and deployment that decrease the barrier to entry.

Machine learning has applications in all parts of business including marketing, advertising, supply chain, information technology, human resources, finance and more.

Companies are increasing investment in machine learning as they work to improve decision-making, automate processes and leverage untapped sources of data. For many, machine learning serves as a foundation to improve understanding of customer behavior, inform capacity and planning decisions, and forecast performance.

Studies show that companies making extensive use of analytics are far more likely to outperform their competition in sales growth and profitability.

Training is most powerful when companies use real-world data and use cases.

As mentioned previously, there are many reasons for widespread Python adoption. Python training leads to benefits that can include increased productivity, efficiency and innovation, as well as decreased time to market.

As an example, CMA Strategy Consulting, a boutique consulting firm focused on the telecommunications, media and high-tech industries, chose to train 75% of its company in Python, from analysts and managers to principals.

This was because CMA faced a growing problem: Their clients’ datasets were getting too large and diverse for their existing toolset. By training in Python, they would be able to add the language into their existing workflow, allowing them to efficiently work with these larger datasets.

After the training, CMA was quick to adopt the techniques learned, with results indicating that analyses using the team’s new Python skills ran 22.5 times faster than before.

TOP 3 IN-DEMAND DATA SCIENCE & ANALYTICS SKILLS

1 Data Literacy The ability to read, work with, analyze, and argue with data

What to Look for in a Data and Analytics Training Provider How Companies Use Training to Solve Common Problems and Achieve Strategic Goals Data literacy training will help generate business value by enabling teams to: • Make more informed decisions based on data.

• Uncover actionable insights from data and identify opportunities within the business where data can be leveraged.

• Increase collaboration in a data-driven organization.

• Increase adoption and usage of datadriven solutions.

Machine learning training enables staff to better leverage data. This can lead to increases in innovation and reductions in cost.

When it’s time to seek out a training provider for your team, there are a couple questions and goals to keep in mind. First, it’s important to focus on outcomes. Ask yourself, “What do I need my team to be able to do after this training, and how do those outcomes align with our overall business needs?” You’ll need to ensure your training partner can address these outcomes directly.

Additionally, training is most powerful when companies use real-world data and use cases for training purposes, adding relevance and enhancing the connection between the learning and how these new skills will apply directly to their work.

Mike Galvin is the executive director of corporate training at Metis, where he oversees the data science and analytics training programs for all of Metis’s corporate clients, including Wells Fargo, Intel and Fortune 500 companies across industries. Email Mike.

2 Python A major programming language that is a favorite with data practitioners

3 Machine Learning A subset of artificial intelligence that involves computers learning from data.

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

| 39

BY KEN PHILLIPS What is arguably the most prevalent issue facing learning professionals today? The answer is scrap learning – the gap between training that is delivered but not applied on the job.

It’s the opposite of training transfer. It’s also a critical issue for both learning and development (L&D) and the

| 40

organization’s L&D supports because it wastes money and time.

Two studies, one by Rob Brinkerhoff and Timothy Mooney in 2008 and one by KnowledgeAdvisors in 2014, found scrap learning to be 85% and 45% respectively in the average

organization. I’ve conducted three scrap learning studies over the past several years with three different organizations – each using a different training program – and found the scrap learning percentages associated with the programs to be 64%, 48% and 54%.

Combining the scrap learning

percentages from all five studies results in an average of 60%.

the 2| See their job.

To further highlight the magnitude of the problem, consider the effect scrap learning has on time and money.

According to the 2018 ATD State of the Industry research report, the average per employee organization training expenditure in 2018 was $1299, and the average number of training hours consumed per employee was 34. Table 1 shows how much scrap learning costs the average organization.

the program as something 3| View that will enhance their career.

How to Combat Scrap Learning

personally motivated to use the 5| Are new information.

A possible new solution to combat scrap learning is predictive learning analytics™ (PLA). PLA provides L&D professionals with a systematic and credible process for optimizing the value of corporate L&D investments by measuring and monitoring the amount of scrap learning associated. Unlike other training transfer solutions, which focus almost exclusively on training delivery and design, PLA provides a holistic approach to increasing training transfer. The methodology is founded on three research-based training transfer components and 12 researchbased training transfer factors.

Learning Program Design LEARNERS:

1| Acquire new information.

program as relevant to

improvement in a critical 4| See department business metric if new information is applied.

Learner Attributes

an immediate opportunity to 12| Have use the new information learned.

Predictive Learning Analytics Methodology The PLA methodology consists of three phases and nine steps, and it provides L&D professionals with insight on actions required to maximize training transfer (see Figure 1 on page 43).

LEARNERS:

confident in their ability to apply 6| Are the new knowledge learned. on lessons learned and how 7| Reflect they can improve their performance. the program as an opportunity 8| View to learn new things.

Learner Work Environment LEARNERS: actively engaged by 9| Are manager before attending

their the training to discuss how the program will improve their performance.

actively engaged 10| Are manager post-program

by their regarding how learning will be applied.

11| Are supported by colleagues.

Phase 1 | Data Collection and Analysis The objective of Phase 1 is to identify the underlying causes of scrap learning associated with a training program.

During this phase, five specific data sets are identified. Two of these are predictive, and three are data driven.

The two predictive data sets pinpoint include: › Which participants are most and least likely to apply what they learned back on the job.

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

| 41

› Which managers of the participants are inclined to provide support for the training.

The three data-driven data sets pinpoint: › Which research-based training transfer components and research-based training transfer factors contribute to training transfer. › Obstacles participants encountered post-training that prevented them from applying what they learned on the job. › A just-in-time measure of scrap learning.

Data for calculating the two predictive data sets are collected from participants immediately following their participation in a learning program using a survey.

The survey consists of 12 questions developed from the 12 training transfer factors described earlier.

DATA SET 1 To predict which learners are most likely to apply what they learned in a training program, participant responses are summarized into average scores. The average scores are then organized into numeric order from highest to lowest, and the top 15%, middle 65% and bottom 20% scores are calculated.

These percentages align with the results Brinkerhoff and Mooney found in their 2008 training transfer research:

DATA SET 2 To predict which managers of the learners are inclined to provide support for the training requires two sets of scores. One is a composite score based on the 12 survey items described earlier, and the other is an average manager training support score. Composite scores are calculated based on the number of employees a manager sends to training, and the score is an average of the employees’ scores on the 12 survey items. For example, if a manager sends three employees to training and the average scores on the 12 questions for the three is 5.92, 5.43 and 5.69, the composite score would be 5.68 (based on a seven-point scale).

Manager training support scores are also calculated based on how many employees a manager sends to training. The score is an average of employee responses to the two survey items measuring the level of support the manager provides to the employees before and after the training.

For example, a manager sends three employees to training and each one scores the pre-training and post-training manager support survey items as follows: › Employee 1 pre-training support 6, post-training support 7 › Employee 2 pre-training support 3, post-training support 2 › Employee 3 pre-training support 4, post-training support 4 The training support score for the manager would be 4.33 (6 + 7 + 3 + 2 + 4 + 4 = 26 ÷ 2 = 13 ÷ 3 = 4.33).

Predictions regarding which managers are inclined to provide support for the training are calculated by subtracting the composite score from the average

| 42

manager training support score. Positive difference scores indicate that a manager is inclined to provide active support for the training. In contrast, low and negative ratings suggest that a manager is inclined to provide weak support. In the example above, the manager would have a difference score of – 1.36 (4.33 – 5.69), indicating he or she is inclined to provide weak support. Only managers with three or more employees attending the training are included in the predictions to ensure valid results.

DATA SET 3 Data set three identifies which of the training transfer components and training transfer factors described earlier are contributing to training transfer.

Scores are calculated by computing an average for each of the 12 training transfer factors and grouping the factors according to the training transfer component with which they align. An average score for each component is then calculated. To determine if any of the component score differences are significant, a statistical test is performed.

Components identified as contributing the least to training transfer are candidates for corrective action.

DATA SETS 4 AND 5 Data for calculating the final two measures are collected from participants 30 days post-program using a survey or focus groups and consists of three questions:

percent of the program material 1| What are you applying back on the job? confident are you 2| How estimate is accurate?

3|

that your

What obstacles prevented you from utilizing all that you learned?

Waiting 30 days post-program is critical because it allows for the “forgetting curve” effect to take place and provides more accurate data.

The scrap learning percentage score provides a baseline against which follow-up scrap learning scores can be compared. These comparisons serve as a way to monitor the effect targeted corrective actions had on increasing training transfer.

The obstacles data identifies barriers participants encountered that prevented

them from applying what they learned.

Waiting 30 days to collect the data allows for the full range of training transfer obstacles to emerge since some are likely to happen almost immediately while others will occur later. Frequently mentioned obstacles are candidates for targeted corrective actions to increase training transfer.

Phase 2 | Solution Implementation While pinpointing the underlying causes of scrap learning is valuable, being able to monitor and manage the targeted corrective actions taken to address them is of even greater significance. It is the focus of Phase 2 and this is where the “rubber meets the road.” It’s where you can be strategic and use data to connect a training program with job application.

It’s also an opportunity to demonstrate

Phase 1 | Data Collection and Analysis SELECT learning program and identify Calibration Cohort

creative problem-solving and the ability to manage critical business issues to a successful conclusion.

Phase 3 | Report Your Results The objective of the third phase is to share your results with senior executives.

Deliver the data as a story and take the executives on a journey of discovery.

Start with a hook, tell the truth without bias and provide context.

In summary, scrap learning has been around forever. However, there is now a way to measure, monitor and manage it using predictive learning analytics.™ Ken Phillips is the founder and CEO of Phillips Associates and the creator and chief architect of the predictive learning analytics™ learning evaluation methodology. Email Ken.

30 DAYS after previous step

BUILD your PLA survey and COLLECT data

CALCULATE: • LAI • MTSI • TTCI

CALCULATE scrap learning percentage and identify obstacles to training t ransfer

Phase 2 | Solution Implementation TARGET “at risk” and “least likely” to apply learners for reinforcement TARGET managers with low or negative MTSI scores for help and support

DEVELOP targeted corrective actions to mitigate or eliminate the underlying causes of scrap learning

TARGET training transfer components and factors with low TTCI scores for improvement TARGET obstacles to training transfer for elimination

CONDUCT Level 2 and 3 evaluations to validate accuracy of PLA algorithm

RECALCULATE scrap learning percentage following implementation of targeted corrective actions

Phase 3 | Report Your Results REPORT results to key business executive stakeholders

ADD data from LMS/HRIS systems

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

| 43

FOR CONTENT DEVELOPMENT

BY SARAH DREYER, MEGHAN CIPPERLEY AND MERAV YURAVLIVKER

It’s Tuesday morning, and managers from the federal government gather virtually for an online data literacy course. Some hail from human resources; others come from policy teams. However, each of them shares a similar objective for the class: to learn how they can use data more effectively in their roles. With that, the instructor launches into material covering the data analytics maturity model, describing the stages an organization passes through on its way to being data driven.

The maturity model starts with descriptive analytics, which can answer questions about what has already happened. For example, a customer service center may track and report on the number of calls received in an hour, day or week. A learning and development (L&D) organization may track the number of enrolled students or percentage of passing grades. We are then responsible for taking this information and using it in our decisionmaking. This type of analysis is valuable and relatively easy to deploy. Nobody in the class has any questions.

As we move up the continuum, the instructor says, analytics become more valuable yet more difficult. Diagnostic analytics, for example, can explain

| 44

why something happened and help managers identify root causes. However, more detailed data is required. In a customer service center, L&D may tie sudden spikes in call volume to events such as recall announcements or software updates. Participants start to mumble about how difficult it is for them to access data.

Once discussion moves into the realm of data science — which includes predictive analytics, prescriptive analytics and artificial intelligence — the instructor remarks that more than just detailed data is required. The data must also be clean, stable and supported by a coordinated governance strategy. There must also be an innovative environment, the right tools and knowledgeable employees within the organization. The benefits, however, are great. You can leverage data to predict what will happen in the future, let data prescribe how to make decisions, and do so continuously and at scale. Participants rattle off great ideas for utilizing data in these ways but don’t know where to start.

As the course unfolds, participants are equipped with the knowledge and skills necessary to start planning and executing data projects. However, chances are many of you currently feel

the same way these students did in the first hour of the class.

The idea of applying analytics to learning data isn’t new, and the benefits of doing so have been covered extensively. More and more, instructional designers and content creators are called upon to incorporate analytics into their processes, particularly relating to learner analysis and evaluation.

However, most learning professionals don’t see themselves as data scientists and have not been exposed to extensive data analysis.

Meanwhile, data is integral to the role of an instructional designer.

When they design curriculum or pick class activities to reinforce learning, instructional designers are often doing so using research-based best practices.

Likewise, they find the highest levels of the Kirkpatrick Model of Evaluation can only be measured with data-driven assessment approaches.

There is little difference between learning analytics and data analytics as a whole, so there are lessons to be learned from our data science brethren. Here are some steps learning leaders can take to start moving up the data analytics maturity model without returning to school for a degree in data science.

ASSESSMENT: IS YOUR ORGANIZATION DATA DRIVEN? Review the statements below and award yourself between 0 and 3 points depending on whether the statement does not describe your organization at all or describes it quite well. Calculate your score for both categories and review your results on page 46.

DATA INFRASTRUCTURE

DATA LITERACY

I can easily access the data I need without asking for the help of others.

My company routinely offers data trainings and other educational opportunities.

I can easily access the data I need in a timely manner.

Most of my colleagues understand the importance of data.

Data is automatically collected and stored on a continuous basis.

The data we have is accurate and of good quality (e.g., few missing entries, few duplicates, accurate measurements).

Our data is stored securely either internally or offsite.

UTILIZE SEASONED DATA PROFESSIONALS There’s no need to reinvent the wheel.

Any good data scientist will routinely borrow and reuse tested code from colleagues and other projects. If your organization already has a data team, tap into that resource. Take them a specific, measurable and objective question, as well as an idea of the available data.

They’re sure to have some ideas. For instance, to determine the best syllabus for an executive-level course, you might ask your data team to look at what topics are most frequently cited, both favorably or unfavorably, in course evaluations.

The results they obtain with text mining methods are beyond the expertise and ability of many L&D teams.

START WITH PAIN POINTS Data Society offers highly-customized courses that teach complex data science concepts and techniques. Whether it’s a class in R, Python or another programming language, learners need to get their hands dirty and practice new techniques on real data. This once posed a huge challenge for our content creators, since programming languages are dynamic and datasets are constantly changing to meet client needs. It meant

Our organization has a set of data standards that reviews how data should be collected, stored and analyzed.

My organization emphasizes the importance of using data to track initiatives.

I am expected to present data metrics when I explain conclusions and decisions.

constantly updating our training materials with fresh code. Fortunately, data scientists like to optimize. In this case, they built a bespoke content authoring tool, allowing us to automatically update coding snippets in instructional materials and standardize formatting. This resulted in improved quality and a huge savings in time. The team reduced its creation time for an hour of content from 40 hours of debugging, topic alignment and formatting to just five. If that doesn’t get content developers lined up for a ride on the data train, nothing will.

LEARNERS NEED TO GET THEIR HANDS DIRTY AND PRACTICE NEW TECHNIQUES ON REAL DATA.

If you don’t have a team of data scientists on hand to help you innovate at this scale, start smaller. The point is that you start looking at your pain points and thinking about how technology and

data can be used to find a better, more optimized way.

SET INTERIM MILESTONES While you should never shy away from developing a long-term vision for leveraging data, set interim milestones to hit along the way. Data scientists learn basic programming syntax and modeling methods long before they learn advanced neural networks. Some may never learn advanced neural networks if it’s not crucial to their work. If your instruction runs off of Google Docs and PowerPoint, don’t expect to set up an advanced analytics program by the end of the week. You may need to start with a task as simple as cataloging the types and descriptions of data that you have access to — known by data scientists as a data dictionary. Likewise, you may never need to employ artificial intelligence if your business needs stop at predictive analytics.

UNDERSTAND AND ADVOCATE FOR DATA GOVERNANCE Data governance is the management of the overall quality, integrity, relevance and security of available data. L&D

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

| 45

teams must get data governance right before any high-powered analytics are possible, and it is where many learning organizations are currently stuck.

Instructional designers may not be privy to what relevant data is collected, how it is stored and secured, who owns it, or its level of quality. This is especially true if useful data is collected by another department or external entity. Asking around for this information is a good first step, as what you find may trigger ideas for exciting analytics.

Many organizations have designated learning data as low priority, but you can advocate for improved governance if you demonstrate how the data will be useful. Other organizations – especially small ones – will have newly established governance strategies, and you can lead the charge to set standards for data entry, data checking and data ownership relevant to your goals.

ONE OF THE BIGGEST FACTORS INFLUENCING A DATA PROGRAM’S SUCCESS IS THE CULTURE BUILT AROUND IT.

GET THE TOOLS YOU NEED By this point, most L&D organizations have implemented content authoring tools and learning management systems. These tools spit out useful data, such as number of enrolled students, time spent online and final test scores. Some companies may have even invested in the latest technologies like experience application programming interface (xAPI). These are all great tools

BUILD A DATADRIVEN CULTURE

ASSESSMENT RESULTS

| 46

for collecting data, but analytics is so much more. The canned reports and dashboards provided by these tools may not give you what you need. Additional tools for storing, cleaning, analyzing, collaborating on and visualizing data may prove more useful. Ask around your organization to see what tools are available to you and consider taking a data literacy course to understand how they can enable your work.

DATA PREPARED

DATA DRIVEN

You are data prepared if you scored under 8 points in data literacy and 8 or more points in data infrastructure.

You are data driven if you scored more than 8 points in both data literacy and data infrastructure.

DATA NASCENT

DATA LITERATE

You are data nascent if you scored under 8 points in both data literacy and data infrastructure.

You are data literate if you scored 8 or more points in data literacy and less than 8 points in data infrastructure.

One of the biggest factors influencing a data program’s success is the culture built around it. Whether it’s a large analytics program or specifically related to learning, you need a leader to champion data’s usage through actions such as highlighting successful data projects in newsletters or at events, asking for data in meetings, or bringing in experts for “lunch and learns.” You and others may want to attend datarelated trainings and conferences or start a community of practice.

With time and practice, L&D organizations will begin to move up the data analytics maturity model. As they do so, content creators will have more information available to optimize learning experiences than ever before. What would you like your data to tell you?

Sarah Dreyer, M.Ed., and Meghan Cipperley, M.A., are instructional designers at Data Society, a data science training and advisory company and one of Forbes 10 EdTech Companies You Should Know About. Merav Yuravlivker is the CEO. They all enjoy making content compelling, data driven and approachable. Email the authors.

8K R>uK oRK o>]K_o

1aI>xÎm val\i]>GK lKkpSlKm > mG>]>F]K val\PalGK oa \KKi pi vSoR xapl ]K>l_S_Q >_I IKuK]ai^K_o _KKIm² la^ uSlop>] S_molpGoalm vRa >lK m\S]]KI S_ oK>GRS_Q oRK ]>oKmo maPo m\S]]m >_I oKGR_a]aQx G]>mmKmÂoa IKmSQ_Klm >_I Ga_oK_o IKuK]aiKlm lK>Ix oa Ga_uKlo al GlK>oK Gpmoa^S{KI ol>S_S_Q Ga_oK_o­ apl KwiKlSK_GKI Ê o>]K_o R>m oRK miKGS>]S{KI KwiKloSmK oa mpiialo xapl ol>S_S_Q i]>_m² SmGauKl Rav K>mx So Sm oa QKo > GamoÄKPPKGoSuK­ ~KwSF]K FK_GR aP o>]K_o²

0)%62 136) cXLIXVEMRMRKEWWSGMEXIW GSQ 0aPo 0\S]]m >_I 1KGR_SG>] 1l>S_S_Q _molpGoSa_>] KmSQ_ a_oK_o KuK]ai^K_o 1l>S_S_Q K]SuKlx À 1 Ê 7 1à K>l_S_Q 0ol>oKQx SlKGo SlK 0KluSGKm

CASEBOOK

REDESIGNING A HIGHLY TECHNICAL INSTRUCTOR-LED TRAINING COURSE TO A COMPUTER-BASED TRAINING: HOW TO BE ACCURATE AND ENGAGING

BY JOEL KOSANKE AND TRENT BARTHOLOMEW, CPTM

What can a company do when two of its most respected and knowledgeable technical trainers retire and there are no other qualified trainers? Even more challenging, these experts are also responsible for track inspection recertification for close to 600 employees across the country.

Employees whose primary duty is to keep the company’s trains safe and on the tracks — literally.

EFFICIENCY, EFFICACY AND SIMPLICITY WOULD BE PARAMOUNT, ESPECIALLY TO A WORKFORCE NOT COMFORTABLE ENGAGING WITH ONLINE TRAINING.

Amtrak faced this scenario last summer. One may think this would

| 48

be an opportunity for a thorough needs analysis, right? Yet we faced an additional challenge: The recertification of these 600 employees would need to commence by July of 2019. It was May 2019. We were in unchartered and unapproved territory. Moreover, these trainers were both highly regarded, popular, and had performance ratings and course satisfaction rates in the upper 90th percentile. These would be difficult shoes to fill.

We only had one viable option that would provide this number of employees thorough training and remediation for recertification. We would also need a persuasive argument to convince the senior managers to approve the creation of a computer-based training (CBT). We emphasized the cost savings by not bringing all employees to one location on the east coast and assured them that these highly regarded subject matter experts would be integral to shaping the CBT, especially problembased scenarios which would capture their rich professional expertise.

However, as instructional designers,

we wondered if we could create a CBT course as effective — and as popular — as the former instructor-led training (ILT).

Let’s assess Amtrak’s approach to redesigning one of its most highly technical instructor-led safety training courses and transforming it into an asynchronous CBT — an entirely new paradigm for Amtrak’s technical training and development department.

ANALYSIS • Trainees were traditionally trained with hands-on practice exercises.

• Trainees received training primarily by lecture only.

• Experience of trainees ranged from two to five years to more than 25 years.

PREPARATION Preparation was key to completing this technical course. We began by

collecting instructor notes, detailed photos and graphics, group exercises, possible scenarios, and discussion questions from our experts. Next, we began storyboarding the course by asking three questions: 1. What techniques can be used to change from PowerPoint to CBT? 2. Can an in-class activity be converted virtually?

3. How can I make the slide interactive and engaging?

From these three simple questions, the transition from instructor-led course to a virtual course began.

DEVELOPMENT Our employees need this course to keep their positions. Efficiency, efficacy and simplicity would be paramount, especially to a workforce not comfortable engaging with online training. We tackled the learning content on the slides in several ways: 1. Narrated image captures: A visual reference to the content is presented without bogging it down with wordy bullets.

2. Knowledge checks: Participants are prompted to respond to a question about the content. The correct answer is revealed even if the wrong answer is chosen.

3. Reference resource: The track compliance manual (TCM) was incorporated, which must remain with employees while on the job.

Employees must be proficient in navigating the manual, so we created screen captures with highlighted reference points to reinforce key concepts.

4. Chunking: Distilled information to three or four points, with narration lasting no longer than a minute.

These four techniques enabled the CBT to flow logically and be an efficient guide to our employees. Now we needed to answer the question: Can an in-class

activity be effectively converted to a CBT? ENGAGEMENT Calculation exercises using formulas from the TCM were critical to the training. The challenge was making them understandable.

For each calculation exercise, we used a narrated demonstration of the formulas. Next, the learners would practice the exercise. Like all the knowledge checks, the answer would still be provided regardless of whether the learner answered correctly; however, we wanted to take it a step further. We opted to provide a step-by-step guide on placing the measurements in the formula to achieve the correct answer, using narration with a short animation of the calculation process. More than getting the correct answer, we wanted to show them how to get there, because if calculations are wrong, derailments could occur.

OUR MOST SIGNIFICANT LESSON LEARNED IS THAT TRAINING MANAGERS AND DESIGNERS MAY WANT TO ASSIST THEIR ORGANIZATION IN A THOROUGH REVIEW OF ITS MOST VALUABLE SMES. We also needed to convert classroom exercises. We utilized images and narration, but we wanted even more engagement. By simply creating clickable items on the screen, interactivity was built in and engaged participants in the learning process. For example, we found that, by asking the same question regarding track deviation using three different images, engagement increased. Even if the answer entered was incorrect, the course would give them a definition for future reference, replicating what used to take place in our classroom environment.

There were varying levels of knowledge among our employees who needed

recertification. Therefore, we needed an additional engagement task to make it more challenging for employees who use the TCM often but would also allow inexperienced employees to refer to the manual as a resource. We built this variance into the CBT by using hot spots, hyperlinks and clickable objects.

This feature enabled less-experienced employees to navigate the manual while allowing experienced employees the option to simply view it.

EVALUATION We made our deadline. The course went live on time. We reduced seat time from eight hours to three hours with the same effectiveness. The reduction in seat hours not only enables employees to get back to the job faster but also allows employees to remain at locations without additional travel costs and time. Our travel costs to deliver this recertification course have been eliminated. Our evaluations, to date, are about 85% favorable for the first year.

We are continuing to make improvements to the course, so we can increase that percentage next year.

LESSONS LEARNED Due to the technical course content, effectively transitioning an instructorled course to a CBT course was — at first — our biggest concern. Applying a true design and conversion plan solved our concern; it determined which content could be interactive and which needed to be chunked or eliminated.

Too often, designers can take an SMEs availability for content-related questions, activities and stories for granted. Therefore, our most significant lesson learned is that training managers and designers may want to assist their organization in a thorough review of its most valuable SMEs, especially their availability and time to retirement.

Think about the importance of building one’s bench strength. By doing so, these experts can share their expertise with the future workforce whether the training delivery is led in a classroom or online.

Joel Kosanke and Trent Bartholomew, CPTM, are lead instructional designers for Amtrak. Email Joel and Trent.

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

| 49

GLOBAL OUTLOOK

DEVELOPING ELEARNING FOR A GLOBAL AUDIENCE: CULTURAL CONSIDERATIONS

BY LUCY HODGE

Consistency is a vital part of workplace training: We want to ensure that all learners receive the same key messages and have equal learning opportunities.

As workforces continue to globalize, eLearning offers an obvious solution.

Centralized and standardized digital courses are available to anyone in the world with internet access.

But can the same course be equally as effective for learners who live and work thousands of miles apart? Research shows us that the answer is yes – as long as instructional designers respond to their audiences’ local preferences and needs.

Culture is a helpful concept for understanding the distinct requirements of international learners. In her work on multicultural eLearning instructional design, Lyn Henderson describes culture as “the manifestation of the patterns of thinking and behaviour that results through a group’s continuing adaptation to its changing social, historical, geographic, political, economic, technological, and ideological environment. Culture incorporates race, ethnicity, religion, class, gender, values, traditions, language, lifestyles, and nationality as well as workplace and academic cultures.”

Identifying different cultures within a training audience can be challenging.

Dutch social psychologist, Geert Hofstede, suggested one possible solution in the 1980s. His “cultural dimensions” model rated six categories of difference between societies to describe so-called national character.

Hofstede used the categorizations

| 50

to suggest broad differences in how students and teachers interact in different countries. His approach remains popular within the instructional design community as a tool for anticipating how international learners will respond to different learning approaches.

The Hofstede model offers a tempting shortcut to discovering culturally specific learning needs and preferences.

Unfortunately, the real world defies such simple categorizations. Modern societies include multiple cultures that intersect, coexist and compete with each other.

The emerging field of sociocultural theory argues that culture is individual.

It is therefore the role of a trainer to understand each learners’ culture and create teaching programs that align with individual experience. Realistically, of course, this is not simple to implement, as online courses may be designed for thousands of global students.

If we are to reject spurious notions of cultural groups but can’t cater to thousands of students’ individual needs, how can we deliver training that is appropriate for all?

A few strategies are available.

BLENDED SOLUTIONS eLearning for multicultural groups can be supported by regional classroombased training or social group settings.

Local teams can contextualize the training and account for geographical (e.g., health and safety precautions in different climates), legislative (e.g., differing environmental regulations)

and other localized differences (e.g., variations in language, etiquette and prior learning).

Creating online social spaces and encouraging interaction between learners and instructors (e.g., discussion boards, email and feedback) can fulfil the same function. Posing questions relating to the learners’ cultural experiences enables the instructor to modify their teaching practice.

An open dialogue allows cultural insiders to shape the training design and ensure that it does not privilege one group above another. Unfortunately, many corporate eLearning courses – such as mandatory training – are designed to be taken without social support, collaboration or trainer interaction. Instructional designers therefore face the dual challenge of identifying cultural differences in their audience and accounting for this diversity without disadvantaging any one group. On these occasions, the following strategies can be used: INTERNATIONALIZATION This strategy seeks to be culturally neutral.

For example, no cultural markers (e.g., symbols, colloquialisms, settings or colors) that may offend or confuse any identified groups are included.

The internationalization approach cannot really respond to diversity in learning cultures. Although the aim is to be neutral, more frequently the product reflects the learning culture that the designer is most familiar with.

APPROACHES TO GLOBAL ELEARNING: PROS AND CONS BLENDED STRATEGIES Pros

INTERNATIONALIZATION Cons

• Facilitators provide cultural context • Audiences can shape future training • Cultural insiders deliver culturally appropriate training • Learners appreciate training targeted directly to them

Pros

Cons

• Cost of face-to-face sessions

• Cost-effective single version

• Inconsistency in training occurs

• Avoids potentially offensive material

• Online facilitators might struggle to respond quickly

• Training is simple and direct

• Interaction with trainers may not be possible

• Learners share identical training materials

LOCALIZATION Pros

• Difficult to achieve as instructors have their own culture • Can become bland • Doesn’t account for different learning cultures • Excludes non-dominant cultures

CULTURAL INTEGRATION Cons

Pros

Cons

• Flexible and scalable solution

• Simplest form is similar to internationalization

• Cultural inclusion, not exclusion

• Final product may be confused

• Highly targeted learning

• Can become costly if fully implemented

• Exposes learners to multiple cultures

• Impractical to implement with short, simple courses

• Doesn’t expose learners to other cultures

• User-led design

• The broader the audience, the more complex the course

• Inclusion of colloquialisms and cultural references

Nevertheless, the attempt to eliminate culture is an attractive and costeffective strategy for organizations with a limited training budget and a global audience.

LOCALIZATION Localization seeks to adapt products for specific cultures and create varied, targeted products. The localization of a course ranges from the simple (e.g., a translated version) to the complex (e.g., profiling with tailored content). Simple localization strategies can be achieved with low levels of input from cultural insiders, as only small graphic, content and language changes will be made.

More complex strategies require a deeper understanding of the cultures represented within the target audience.

As a result, there is a risk of relying on preexisting cultural frameworks and stereotypical design ideas without clear supporting evidence. Ideally, instructional designers should be guided by an organization’s cultural insiders rather than external research.

• Exploratory, multi-layered course

AN OPEN DIALOGUE ALLOWS CULTURAL INSIDERS TO SHAPE THE TRAINING DESIGN AND ENSURE THAT IT DOES NOT PRIVILEGE ONE GROUP ABOVE ANOTHER.

• Involves effort to avoid tokenism or superficial multiculturalism

key cultures and user preferences for the course.

Creating truly culturally integrated training requires careful avoidance of tokenistic inclusivity and stereotypical assumptions, particularly if preexisting cultural frameworks are used. Communication with cultural insiders and the target audience is essential.

WHICH STRATEGY SHOULD WE USE?

CULTURAL INTEGRATION In contrast to internationalization and localization, cultural integration strategies aim for a single, culturerich course that accounts for cultural diversity among learners.

Methods to incorporate different cultures include cultural research by reviewing learning contexts and strategies, cultural demographics by gaining insight into learner experiences and expectations, and cultural pluralism by targeting audience input. The use of tools such as questionnaires can help identify

Any of these strategies can be used to create eLearning that meets the needs of a global community. To ensure success, instructional designers must work closely with stakeholders to identify the best strategy for a particular organization.

Designers should prioritize discussing the course design with cultural insiders before development. This collaboration is the first and most crucial step toward producing effective eLearning for a global audience.

Lucy Hodge is a senior learning solutions consultant at Walkgrove Ltd. Email Lucy.

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

| 51

INSIGHTS FOR THE MODERN LEARNING LEADER TICE Virtual Conferences bring learning leaders together to share insights on the most prevalent challenges facing today’s training professionals. While in-person conferences have been put on hold, we are adding more virtual conferences to our calendar to fuel your learning and development growth. Sessions are always free and can be accessed following the live event.

Learn more and watch past virtual conferences at trainingindustry.com/ticevirtual.

SEPTEMBER 3 STRATEGIC PLANNING

OCTOBER 15

SKILLS DEVELOPMENT

DECEMBER 3

MEASUREMENT & DATA

TICE Virtual Conferences are pre-qualified for credit hours with leading human resources and training certifications.

STELLA LEE, PH.D.

WHAT’S NEXT IN TECH

BUILDING DATA FLUENCY IN L&D: A QUICK GUIDE

Increasingly, organizations are using artificial intelligence (AI) and machine learning-enabled business platforms to streamline processes, reduce costs, train staff and optimize resources.

These platforms generate a tremendous amount of data, much of which is unused or unanalyzed. For learning and development (L&D) practitioners, there lies a great opportunity to make use of data to demonstrate how L&D directly impacts and supports business, as well as how learning is crucial to establishing alignment with business goals. We can only do that if we have the ability to turn raw data into insights, and we need to build data fluency within our L&D teams to do so.

Data fluency extends beyond data literacy. To have data fluency means that we know where the data comes from, and as a result, we can process, interpret, infer and apply data effectively. Application sets data fluency apart from literacy, empowering you to take informed action.

To build data fluency, start with the following practices: START WITH “WHY” Before we lose sight of the forest for the trees, we must understand why we want to make use of data in learning. Define the challenge or problem you set out to solve within your organization. For example, learning practitioners are commonly questioned about the effectiveness of their training efforts. You need to define “effective” for your learning programs and determine how to measure that

effectiveness. Only then do you know what data to look for.

KNOW YOUR DATA With various online platforms deployed across organizations, you need to identify where the sources of data are. Don’t just collect the obvious – like data from your learning management systems (LMSs). E-libraries, web conferencing systems, learning experience platforms (LXPs), online survey tools, internal social networks, intranets and knowledge sharing repositories are valuable sources of data; make sure you have access to these systems. Then, you need to get comfortable asking questions about the data you collect. The most important questions to ask: Did you find the right data? In other words, are you able to measure what you want to measure? Or are you comparing apples to oranges?

DATA FLUENCY EXTENDS BEYOND DATA LITERACY. ANALYZE THE DATA While big data seems to be all the rage, small data is what we typically handle in L&D. Small data provides information that can be easily analyzed with your every-day spreadsheet programs, such as Excel. Don’t get caught up in complex data analytics software. Many modern data analytics tools are designed with a non-technical audience in mind, provide templates to get you started, and allow you to quickly compare and visualize data.

If you find the task of analyzing data too daunting, consider partnering up with your organization’s business intelligence unit or information technology department who are familiar with manipulating databases.

FROM INSIGHTS TO ACTION Finally, you need to put your insights from data into action. Based on the problem you defined, your data should help you make decisions, implement improvements and solve issues. For example, if you want to know how effective your cybersecurity training program is, you need data that indicates efficacy based on your definition. Is there an increase in the number of people changing to stronger passwords post-training? Can you identify what caused this increase and whether it is related to training? Are you able to take action based on what you know to more effectively design the material?

Data fluency is more than technical skills.

You need to develop an interdisciplinary mindset, and be able to look at data from a wide range of angles including business, communications, statistics and psychology. It also requires us to be curious, continue to ask questions and collaborate with others to implement what you learn.

Dr. Stella Lee has over 20 years of experience in consulting, planning, designing, implementing, and measuring learning initiatives. Today, her focus is on large-scale learning projects including LMS evaluation and implementation, learning analytics, and artificial intelligent applications in learning. Email Stella.

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

| 53

Bridging the digital skills gap by accelerating knowledge worldwide.

The Leading Learning Partners Association (LLPA), the Microsoft Learning Partner of the Year 2020, is a global organization with a presence in more than 55 countries. Our extensive network of trusted members contributes to the development of expert job-ready industry professionals through hybrid learning offerings, consulting, and certification. In addition to these services, the LLPA members offer free online role-based content through its Skills Academy portals giving you the tools you need to dynamically reskill and upskill your workforce at a faster rate. skills-academy.com In this new COVID-19 economy, you need a training service provider you can trust. Consider partnering with an LLPA Member

in your country and gain access to full service digital skills that will ready your workforce from the ground up! thellpa.com/members

DOUG HARWARD

SECRETS OF SOURCING

IDENTIFYING TRAINING SOLUTIONS TO BUSINESS PROBLEMS BASED ON DATA FLUENCY

The purpose of the training organization as it relates to the business is a common topic of discussion in the learning and development (L&D) community. Highperforming training organizations excel at aligning learning solutions with the needs of the business and helping those working within it perform at a higher level.

new role in the organization, you don’t know what tools you will need to be successful. Leaving it up to the learner to determine their own needs falls short of helping them become successful.

Improving performance implies we understand there are issues hindering the organization from performing at its best. Let’s face it, if the organization didn’t have problems, then we wouldn’t need a training function. Training managers play a key role in helping organizations improve efficiency and mitigate performance issues.

Data associated with how the business is performing provides us our best insights into what training is needed. Similar to how a doctor checks vitals to diagnose an illness, learning leaders gather data to understand organizational illnesses.

Data provides direction as to how to treat the condition. Training managers should work with clients to understand the data available and be astute at using data to conduct diagnostics and identify needs.

I like to think of training managers as problem solvers. Training courses are vehicles for helping the organization perform better. We are not in the business of creating courses; we are in the business of creating solutions.

EXCEPTIONAL TRAINING MANAGERS USE DATA TO UNDERSTAND THE ENVIRONMENT THEY ARE WORKING IN. As training professionals, we must identify and apply the right learning solutions to the right problems. Traditional approaches to learning allow students to choose what knowledge and skills they need. But in a work setting, learners do not always know what they need. If you are a new employee or are taking on a

USING DATA TO IDENTIFY TRAINING NEEDS

Of course, training managers are not doctors, but we are leaders. And exceptional training managers use data to understand the environment they are working in and determine solutions for solving existing problems. Training managers, just as any leader of any organization, should use data as a means to remain objective. Emotional decision-making or developing learning curriculums based on what’s trendy at the time may feel good initially but will not achieve long-term results. Great leadership is about being objective and leveraging the facts available.

In the Certified Professional in Training Management (CPTM) certification program, we teach a simple five-step process to identify solutions to business problems based on data fluency:

• Problem statement: Identify the problem you seek to solve based on data related to your clients most pressing needs. Work with your client’s leadership team to understand the most important issues to prioritize.

• Diagnostics: Next you must diagnose the root causes of the problem you have identified by gathering related data, as well as opportunities and approaches for how to solve the problem.

• Solution: Based on data, determine the solution you plan to utilize to solve the problem you’ve identified.

• Implementation: Develop a plan for how you will implement the solution you intend to provide. This includes how you will develop or procure the training and how you will deliver it or provide access to the learner.

• Measure success: Lastly, it is imperative that you measure progress and success using the same, relevant data that you used to identify the problem.

Great training managers help create and manage great training organizations, and great training organizations demonstrate a high degree of data fluency. Data fluency is one of the key competencies of a training manager.

Doug Harward is CEO of Training Industry, Inc. and a former learning leader in the high-tech industry.

Email Doug.

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

| 55

TOP

20

2020

2020

NEW TOP 20 LISTS LAUNCHED

TOP

20

COMPANY

COMPANY

TRAINING OUTSOURCING

CUSTOM CONTENT DEVELOPMENT

CONGRATULATIONS TOP 20 COMPANIES VIEW THE LISTS The Top 20 Companies are a service provided by Training Industry, Inc. Due to the diversity of services offered, no attempt is made to rank Top 20 lists.

MICHELLE EGGLESTON SCHWARTZ

LEARNER MINDSET

DOES IT TAKE A PANDEMIC FOR EMPLOYEES TO MAKE TIME FOR LEARNING? Prior to the pandemic, research suggested that employees do not have time for learning and development (L&D) – despite an overwhelming desire to learn. In fact, I’ve written several articles on the topic. However, that narrative has shifted dramatically over the past few months. The online learning industry has skyrocketed during the COVID-19 pandemic, boasting record-breaking growth in course enrollments.

At Training Industry, we’ve witnessed this change firsthand. TrainingIndustry.com has seen an increase in website traffic and a significant change in the topics and solutions our visitors are seeking.

From transitioning to a remote workforce to boosting employee engagement to addressing cybersecurity concerns, training professionals are searching for solutions to their current challenges.

LEARNING LEADERS MUST PROVIDE EMPLOYEES WITH THE TIME AND RESOURCES TO LEARN, EXPLORE AND TAP INTO THEIR CREATIVE TALENTS. Learning platforms across the industry are experiencing record-breaking engagement. LinkedIn Learning reported a 130% increase in learning by enterprise learners in March and April – the largest spike in the history of the platform.

Mark Onisk, the chief content officer at Skillsoft, shared Skillsoft has seen a 317% increase in learning content accessed

on its learning experience platform, Percipio. Udemy reported a 425% increase in course enrollments across its entire platform. These are just a few of the learning platforms that have seen significant growth during the pandemic.

WHY THE SUDDEN INTEREST IN LEARNING? Learning has taken center stage during the pandemic. Quarantined employees have an increased appetite for development, and the numbers prove it. Employees have always had a desire to learn, so why are we seeing such an increase in consumption?

The pandemic has created a turbulent job market, leaving employees uncertain whether their company will survive or if they will be let go. This uncertainty has led many employees to improve their skill set to make them a more valuable member of their team.

Additionally, in times of uncertainty, people often gravitate toward activities that give them a sense of control. While one person may choose to organize and clean their home to bring structure to their lives, another person may opt to upskill or reskill to make them more effective in their role.

Another factor influencing the significant increase in learning is the vast amount of time people have on their hands. With stay-at-home orders in place all over the country, people finally have the luxury of time, and they’re choosing to use that time to invest in themselves through learning.

Every employee and industry isn’t experiencing the same level of free time, but the metrics certainly support an increase in learning. While it remains to be seen whether employees will continue to prioritize learning once life begins to normalize, one thing is certain: People genuinely want to learn.

HOW CAN L&D HARNESS THIS LEARNING SPIRIT? With the pandemic swiftly changing the course of business, employees are struggling to adapt to a new way of working. Training professionals play a pivotal role in helping their employees and organizations navigate these changes.

Now more than ever, leaders must be in-tune with the needs, challenges and frustrations of their employees. This focus on employee well-being will power the organization to overcome business challenges and move them forward.

Organizations must work to harness the learning spirit we’ve seen over the past few months. From managers regularly checking in with employees to conducting a formal needs analysis, learning leaders must stay abreast of the needs of their workforce. We must provide employees with the time and resources to learn, explore and tap into their creative talents. If time has been holding employees back from learning, leaders must proactively protect their employees’ time moving forward.

Michelle Eggleston Schwartz is the editorial director of Training Industry, Inc. Email Michelle.

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

| 57

CLOSING DEALS

SUPPORTING WORKERS DISPLACED BY COVID-19 AND BUILDING AN EQUITABLE WORKFORCE

BY TARYN OESCH

The U.S. unemployment rate rose to 14.7% — the highest since 1948 — in April, a few months after the first coronavirus diagnosis in the country.

Also that month, the International Labour Organization estimated that 1.6 billion workers worldwide (almost half the global workforce) were “in immediate danger of having their livelihoods destroyed.”

The pandemic is hitting people hard.

But it’s also exacerbating existing inequalities, widening income gaps between white workers and workers of color, for example, and highlighting the educational needs of workers.

“Even before the pandemic,” says Rachel Carlson, chief executive officer and cofounder of Guild Education, “the world’s largest employers were asking us to develop education solutions that could create pathways to economic mobility for their workforce.” They were also looking for “ethical offboarding practices, as they implemented technologies that transformed their workforce needs — and planned for the eventuality of a recession.”

INTEGRATING TRAINING INTO OUTPLACEMENT To meet these demands, Guild Education began a partnership with Entangled, a “product studio” working on an “outskilling” platform, which integrated training into the outplacement process, Carlson says. This collaboration aimed at training displaced workers for new careers using Entangled’s platform, a marketplace “backed by employers that wanted to hire.”

| 58

Shortly before announcing this acquisition, Guild announced the launch of Next Chapter, a product that enables displaced workers to prepare for new jobs in industries seeing an increased demand. Guild built the product in partnership with Entangled, whose jobs marketplace enables employers joining Next Chapter to provide displaced workers with reskilling and coaching.

A HEIGHTENED SENSE OF URGENCY No one enjoys laying off or furloughing employees, particularly in the large numbers many employers have had to cut due to COVID-19. Not only are workers looking for new opportunities, but organizations are looking for ways to support employees they’ve had to let go.

“We continue to witness a tale of two labor markets,” Carlson says. “On one side, we see business leaders faced with unprecedented change and economic pressures that lead to laying off talent.

But we’re still seeing employers that are grappling with huge talent needs [and] skills gaps and are actively working to attract and upskill their workforces.”

“That’s why we launched Next Chapter.

It’s about helping laid off workers have access to not only the career services, but also the job training, that they need to have a chance at, not just surviving, but thriving, in this new economy.”

THE NEW NORMAL Even if we could return to the prepandemic “normal,” we shouldn’t want to,

Carlson notes. “In a recent op-ed, [author and professor] Roxane Gay said that while ‘the rest of the world yearns to get back to normal, for Black people, normal is the very thing from which we yearn to be free.’ We’re taking that sentiment to heart … Our work can’t be about simply returning things to normal — for workers, for employers, it has to be about making things better than they were before.”

For both workers and employers, education can “unlock profound growth,” Carlson says. It provides employers with a “competitive advantage” and is also “a profound lever to address wage and economic inequality that continue to grow at alarming rates.” She cites research from McKinsey that found that workers without a bachelor’s degree are twice as vulnerable to job loss than workers with a college education. “Our focus is on addressing the immediate crisis, but it’s also about helping workers to build skills that will make them much more resilient in the future.”

COVID-19 has made upskilling even more important, not only for business success but for worker survival. “As rising unemployment magnifies economic inequality,” Carlson says, “employers have a heightened responsibility to proactively help workers of all ages, races and demographics gain access to training that can help them become resilient in the face of … the sort of catastrophic health, economic and social crisis we’re grappling with.”

Taryn Oesch is the managing editor of digital content at Training Industry, Inc. Email Taryn.

COMPANY NEWS

ACQUISITIONS AND PARTNERSHIPS Group Management Services, Inc., announced the acquisition of Corporate Business Solutions (CBS), a Georgiabased human resources (HR) outsourcing provider. Both organizations share an emphasis on giving their clients a competitive edge when it comes to the technology, service levels and human resources they provide.

DataCamp, announced its partnership with Anaconda, an open-source tools and enterprise-grade software provider for data scientists and organizations doing data science at scale. Inspired by a shared mission to enable worldwide data fluency for individual learners and organizations, Anaconda Team Edition customers will gain access to a specialized DataCamp learning track.

MindTickle, the leader in sales readiness technology, announced a partnership with Sandler Training, the largest sales, management and leadership training organization. Sandler Training sales and sales manager content can now be tailored with a data-driven approach to develop their skills and readiness, leading to faster time to profitability and ongoing quota attainment.

OpenSesame, the eLearning innovator, announced American Management Association (AMA) has been added to its OpenSesame Plus subscription to expand the leadership, management and business skills offering. Global 2000 companies leverage the OpenSesame Plus subscription with over 7,500 curated courses to develop learners’ leadership and soft skills.

Capgemini, today announced a global partnership with Coursera to expand its libraries of high-quality learning to all of its employees across the globe. The goal of the collaboration is to help team members to further develop their skills in several areas including professional services, technology consulting, cloud adoption, sales productivity and many more.

Templafy, a global leader in enterprise document creation and automation, announced its acquisition of Denmarkbased Napp, a B2B sales enablement platform. The acquisition extends Templafy’s document creation platform, empowering users with enhanced abilities to collaborate with recipients of business documents and track document performance.

INDUSTRY NEWS BOOSTING ECONOMIC MOBILITY FOR FRONTLINE WORKERS WITHOUT DEGREES Merit America, a nonprofit that helps workers chart pathways into highly skilled technology careers, announced a partnership with Amazon’s ambitious Career Choice initiative, helping hourly employees prepare for future career opportunities. During the COVID-19 pandemic, Merit America is offering virtual programming to provide learners with both skills training and job readiness support.

NO DISTANCE IS TOO FAR FOR SALES COACHING SalesFuel® announces the global launch of the next version of SalesFuel COACH, the developmental sales

coaching solution. SalesFuel COACH has been upgraded for teams whose sales reps are in the office and working remotely worldwide. Built on scientific assessments, SalesFuel COACH enables managers to recognize the specific areas of improvement for each rep.

CULTIVATE, MEASURE AND MANAGE A MORE INCLUSIVE WORKPLACE Eskalera, the company behind a new category of employee experience and organizational analytics, announced the launch of Inclusion Index, a new system that quantifies inclusive culture.

The Inclusion Index measures the performance of a company’s diversity and inclusion efforts and offers learning solutions and actionable steps to improve cultural hotspots.

SKILLSOFT ENTERS INTO AGREEMENT WITH LENDERS TO REDUCE DEBT AND POSITION COMPANY FOR LONG-TERM SUCCESS Skillsoft announced it has entered into a Restructuring Support Agreement (RSA) with a majority of its first and second lien lenders. The RSA is expected to result in a comprehensive de-levering of the company’s balance sheet by reducing the company’s existing first lien and second lien debt to $410 million from approximately $2.0 billion, with total debt aggregating $585 million, lowering the company’s annual cash interest by approximately $100 million.

T R A I N I N G I N DUSTR Y MAGAZ INE - DATA FLUENCY IN LEAR NI NG 2 02 0 I WWW. T RAININGINDU S T RY . C OM/ MAGAZ I NE

| 59

LEVEL UP

AT HOME Prioritizing your growth as a great learning leader has never been more important.

Keep learning and developing important skills at this time by investing in virtual professional development designed for you, the learning leader.

View the calendar at the trainingindustry.com/ continuing-professional-development/calendar