Adapting Learning | January/February 2018

J A N U A R Y/ F E B R U A R Y 2 0 1 8

ADAPTING LEARNING FROM DATA TO ANSWERS | 16 Solving Customer Training Problems

UNLOCK EMPLOYEE PERFORMANCE | 32 Optimizing Results Using Development Maps

HR VS. L&D METHODOLOGIES | 44 Linking Frameworks to Transform Learning

BUSINESS

PERSPECTIVES

ON

MANAGING

WORLD-CLASS

TRAINING

adaptive learning Ultimate Personalization — Maximum Business Impact No two brains are alike. Area9’s adaptive platform combines cognitive neuroscience and computer science with your content to cut training time in half, guarantee proficiency and make lasting impacts on careers and business outcomes.

area9learning.com

PERSPECTIVES - KE N TAY LO R

Looking at the cover of this edition of Training Industry Magazine, many of you may have thought, “Was that a typo? Isn’t it ADAPTIVE LEARNING?” Well, in fact, the term is adaptive learning, but we felt the discussion should be broadened to consider all forms of personalized learning, because in the end, there is resounding consensus that the training with the most significant impact is that which is conformed to the individual taking the program.

We understand that in the corporate context personalizing learning may not always be practical. Our opinion is that the learning and development team in any company should still be looking for opportunities to contextualize learning or bring it as close as possible to the on-thejob reality where the new skills should be applied. As business people, we should default to the situation where learning impacts a company’s performance because it is closing skill gaps, or in some way corrects a behavior, action or inaction that is or could be impacting the company’s performance.

What is really exciting about the current reality in corporate training is that we now have the tool kit required to take big strides in the right direction. If we think of personalization at the total experience

level, versus simply at the “course” level, it opens opportunities to truly personalize at scale. These tools and technologies not only facilitate elements of the learning experience that are outside the course, they also allow us to track and modify them as a program rolls out.

For example, personalization may take place in the social learning part of the learning experience. It is not unreasonable to consider focused coaching or mentoring on areas that an employee may have a greater than normal skills gap in. Or perhaps, designing practice that is done on the job to ensure that the skills are applied correctly in the context of the employee’s role.

IF WE THINK OF PERSONALIZATION AT THE TOTAL EXPERIENCE LEVEL, VERSUS SIMPLY AT THE “COURSE” LEVEL, IT OPENS OPPORTUNITIES TO TRULY PERSONALIZE AT SCALE.

This collection of articles will share some insights on how your organization may consider adapting your learning initiatives to the needs of the individual in the context of their role in your company.

Our goal at Training Industry is to provide you with the insights and tools needed to more effectively manage the business of learning. As always, we would love to hear your thoughts about the points of views shared in the magazine.

Ken Taylor is the president and editor in chief of Training Industry, Inc. Email Ken.

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

|3

CO N T E N TS

TA B L E O F VOLUME 11

|4

I

ISSUE 2

I

JANUARY/FEBRUARY 2018

FEATURES

16 FROM DATA TO ANSWERS

16

32 UNLOCK EMPLOYEE PERFORMANCE

44 HR VS. L&D METHODOLOGIES

FROM DATA TO ANSWERS: USING A DATA-DRIVEN APPROACH TO SOLVE CUSTOMER TRAINING PROBLEMS

By Linda Schwaber-Cohen

Improve strategic decision making and resolve customer training issues with the help of data.

21 24 28 32 36 40 44 48

SIX WAYS ADAPTIVE LEARNING CAN ELEVATE CORPORATE TRAINING

By Nick Howe

Elevate corporate training and development with the help of adaptive learning.

FACILITATING A ROBUST SOCIAL LEARNING EXPERIENCE FOR GEOGRAPHICALLY DISPERSED TEAMS

By Daila Boufford

Combat the challenge of promoting engagement within virtual teams with social learning.

CREATING TRAINING APPROPRIATE TO THE ROLE

By Ashley Korneagay, CPTM

Set employees up for long-term success by developing training appropriate to their role.

USING DEVELOPMENT MAPS TO ALIGN AND UNLOCK PERFORMANCE

By Kelly Smith and Brian Blecke

Enhance the learning journey and improve employee performance by using development maps.

FUTURE-PROOF TRAINING WITH ADAPTIVE LEARNING

By Christina Yu

Drive bottom-line results and unlock organizational performance with adaptive learning.

CULTURAL CONSIDERATIONS IN A NON-GLOBAL ORGANIZATION

By Carol E. Brogan

Shape the future of learning by better understanding the culture of individuals in the organization.

LINKING COMPETENCY MODELS TO MODERN LEARNING STRATEGIES

By Matt Donovan

Create new opportunities by incorporating competency frameworks in modern learning experiences.

DEVELOPING TRAINING WITH DIVERSITY AND INCLUSION IN MIND

By Tiffany McLean

Create a more inclusive environment by incorporating diversity and inclusion into training.

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

|5

I N THIS I S S U E

THOUGHT LEADERS

03 09 11 13

PERSPECTIVES

By Ken Taylor

Always be on the lookout for ways to adapt learning to the individual.

GUEST EDITOR

By Brent Engel

Capitalize on the social aspects of learning with peer trainers.

SCIENCE OF LEARNING

By Srini Pillay, M.D.

Consider three scientific principles when personalizing learning.

PERFORMANCE MATTERS

By Julie Winkle Giulioni

Maximize learning by engaging learners before the training begins.

15

BUILDING LEADERS

By Sam Shriver and Marshall Goldsmith

Impactful learning cannot take place without effective leadership.

57

SECRETS OF SOURCING

59

LEARNER MINDSET

By Doug Harward

The ability to adapt in the workplace can enhance job performance.

By Michelle Eggleston

Learning is a personalized experience for both the individual and the business.

61

WHAT’S NEXT IN TECH

62

CLOSING DEALS

63

COMPANY NEWS

By Eric Sharp

Keep these three things in mind when personalizing learning to ensure alignment.

INFO EXCHANGE

52

CASEBOOK

54

MEASURING IMPACT

AdRoll closed a gap in leadership development by investing in their new leaders and managers.

Gauge training effectiveness at the individual level with subscriptionbased learning.

CONNECT WITH US

|6

1 (866) 298-4203

New technologies can assist in developing more effective leaders who understand their employees.

Keep up with the latest in the training industry by reading news from the last quarter.

editor@trainingindustry.com

TrainingIndustry.com

A B O U T OUR TEAM

STAFF CHIEF EXECUTIVE OFFICER Doug Harward

ASSOCIATE EDITOR Stephani Mager

dharward@trainingindustry.com

smager@trainingindustry.com

EDITOR IN CHIEF & PRESIDENT Ken Taylor

SENIOR DESIGNER & ART DIRECTOR Heather Schwendner

EDITORIAL DIRECTOR Michelle Eggleston

DESIGNER Mary Lewis

EDITOR Taryn Oesch

ADVERTISING SALES

ktaylor@trainingindustry.com

hschwendner@trainingindustry.com

meggleston@trainingindustry.com

mlewis@trainingindustry.com sales@trainingindustry.com

toesch@trainingindustry.com

EDITORIAL BOARD

MISSION Training Industry Magazine connects learning and development professionals with the resources and solutions needed to more effectively manage the business of learning.

SUBSCRIPTIONS ELECTRONIC:

JUDI BADER Senior Director of Learning Arby’s Restaurant Group

SCOTT NUTTER General Manager, Research, AQP & Development Delta Air Lines

MICHAEL CANNON, M.ED. Senior Director, Head of Learning & Development Red Hat

MATTHEW S. PRAGER Executive Training Manager U.S. Government MARC RAMOS Head of Education, Google Fiber Google KELLY RIDER Vice President, L&D Content Strategy & Experience SAP Learning & Development

LORNA HAGAN Chief People Officer OnDeck BARBARA JORDAN Group Vice President, Global Learning & Development Sims Metal Management CATHERINE KELLY, MA, BSN, RN, CPTM Director of Learning Programs Brookdale Senior Living ADAM KUCERA Director of Sales Training & Support DISH SHIREEN LACKEY Talent Management Officer, Office of Business Process Integration Veterans Benefits Administration LAURA MORAROS Global Head of Sales Learning Facebook A | S | B | P|E

Fostering B2B editorial excellence

Fostering B2B editorial excellence

American Society of Business Publication Editors

American Society of Business Publication Editors

2017 National Award Winner

KERRY TROESTER Director, North America Sales Training Lenovo NATASHA MILLER WILLIAMS Vice President, Talent Engagement & Development Nielsen KEE MENG YEO Vice President, Enterprise Talent Development

Amway

A | S | B | P|E

ONLINE

DR. SYDNEY SAVION Chief of Education Strategy-Education Services Dell EMC

Sign up at TrainingIndustry.com to receive notification of each new digital issue.

PRINT:

Print copies are available for purchase at magcloud.com for $15.95.

ARTICLE REPRINTS To order reprints of articles, please contact Training Industry at

editor@trainingindustry.com.

PUBLISHER Training Industry Magazine is published bi-monthly by: Training Industry, Inc. 6601 Six Forks Rd Ste 120 Raleigh, NC 27615

2017 Regional

ONLINE Award Winner

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

|7

GUEST EDITOR BRENT ENGEL

SHOW ME THE WAY: CREATING A CUSTOMIZED LEARNING EXPERIENCE WITH PEER TRAINERS

I have designed training for a variety of audiences: technical skills training for hourly employees, soft skills training for management employees, business training for business owners, and everything in between. For me, the key to successfully adapting learning across these audiences has been to capitalize on learning through peer trainers.

A GOOD TRAINER PROVIDES THE ACCOUNTABILITY MOST OF US NEED. LEARNING IS SOCIAL The psychologist Dr. Vygotsky, whose ideas have shaped a lot of thinking in the field of education and learning, argued that learning is a social process.

He argued that learning occurs through interactions with others, especially in guided interactions with a more knowledgeable other. I agree for several reasons. First, it creates curiosity; second, it provides motivation; and third, it provides a model from someone doing what I need to do.

WE DON’T KNOW WHAT WE DON’T KNOW When speaking with someone who has more knowledge or experience, we realize how little we truly know. I was once building a cedar closet in my basement and my coworkers asked whether I was spacing the upright members on 16” centers. After answering with a blank stare, I shared I was paneling

with cedar, which started a discussion of drywall, followed by more blanks stares and some research on construction on my part.

Psychologists such as Malone and Lepper argue the effect of such discussions is to spark our curiosity to find more information. By talking with an expert peer trainer, we find the gaps in our knowledge and seek to fill them. Malone and Lepper argue that this motivates us to learn.

SOMETIMES WE NEED FOCUS I like self-directed learning. It aligns well with the ideas of adult learning and putting the learner in control of the learning experience. I have a stack of unread books with all the great topics I want to learn. Then there’s the online courses that never seem to be completed before the subscription auto-renews. My willpower does not apply the consistent accountability I need.

A good trainer, like a good coach, provides that accountability most of us need. Instead of letting life get in the way, they become part of what moves us forward. A good trainer can help us set goals and performance standards for learning that challenge us. They can get us back on track when we get distracted, such as a friend who asks us how the online course is going.

SOMETIMES WE NEED A START (OR RESTART) Peer trainers serve as role models for a desired behavior. As a new classroom instructor, I was told to make my

instruction activity-based. The problem is, there are only so many times you can play Jeopardy in the classroom. Then I started talking about delivery methods with a peer who had recently been named faculty of the year. She shared ideas and books (still sitting on my shelf ) about her approach to activitybased learning.

I applied those techniques in my classroom and demonstrated them when I led faculty orientation sessions.

I am happy to say, I too was eventually named faculty of the year in large part by following my peer’s model.

The reason this aspect works is the learner sees someone like them performing the task or applying the knowledge on the job. By observing and modeling the behavior, we hope to achieve the same results as our model. Dr. Bandura argued that is often enough to get us started.

In summary, a good peer trainer is what Dr. Vygotsky called a scaffold. The peer trainer acts as a resource and sounding board for the learner that understands the situation and speaks the language.

It’s important to provide peer trainers with development and training to enable them to be successful, and the rewards are many. A good peer trainer can ensure the learner gets the customized program needed to be successful.

Brent Engel is a senior director of training at Willy’s Mexicana Grill. He has over 20 years of experience creating learning courses, launching LMSs, developing corporate universities and creating training departments. Email Brent.

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

|9

BE AN EXPERT. Training Industry research analysts seek insights that benefit you—the training leader.

From understanding the unique process capabilities that differentiate the Great Training Organization™ to exploring the components of the Modern Learning System, gain greater understanding of the training industry through our vast research library.

We do the work, you gain the expertise. www.

TrainingIndustry. com/research

SCIENCE OF LEARNING SRINI PILLAY, M.D.

THE SCIENCE OF

PERSONALIZED LEARNING

The brain is the anatomical environment where organizational learning takes root.

In order to learn effectively and efficiently, people have to be able to pay attention, absorb information, store that information long-term, and recall it when necessary.

But learning is more than a mechanical function. It involves a variety of less tangible factors too.

Theme-based learning (e.g., “customercentricity,” “resilience,” or “agility”) may be helpful, but these words are rarely part of anyone’s everyday vocabulary and they quickly fall on deaf ears. Few people get out of bed and declare, “Today is the day to be agile and resilient.” Studies demonstrate that for people to learn, they should be able to identify with the content.

Given the number of factors that need to be taken into consideration for learning to be personalized, it is virtually impossible to design a learning experience that is perfectly personalized. Below are three of the many scientific principles that could assist in designing learning that is tailored for the individual.

LEARNING IS MORE THAN A MECHANICAL FUNCTION.

BRAIN-BASED DIMENSIONS OF LEARNING Understanding how the brain learns can be valuable to learning professionals. Many studies demonstrate that the brain learns differently with reward, punishment, play and experience. And it is also important to understand what makes people want to learn, how they learn consciously and unconsciously, and how people learn differently from one another too.

Recommendations: Use these brain-based processes to assess your personalized methodologies. For example, you might ask, “Will this case history of a man help female employees attend to the principles it demonstrates?” “Is there reward or punishment built into this learning?”

These kinds of questions will help you to design and iterate on existing programs.

GROUND THE LEARNING IN “IDENTITY” Within an organization, individuals differ in their learning preferences.

Recommendations: Rather than grounding learning in vague terms such as “agility,” “empathy,” or “team dynamics,” which frequently and inaccurately suggest that one must always be agile, empathic or team-oriented, it may make more sense to structure the learning around specific organizational challenges that are relatable.

Individuals can choose that are relevant for case that they are not challenges, assessment way to enhance insight.

the challenges them. In the aware of their tools can be a

COMBINE DATA AND COGNITIVE STYLES There are many instruments to measure learning styles. Honey and Mumford describe four different kinds of learners: Those who learn from experience, from reflective observation, from exploring associations and interrelationships, or from doing or trying things with practical outcomes. Felder and Silverman categorize learners in terms of being sensory or intuitive, visual or verbal, active or reflective, and sequential or global.

However, detailed reviews by Pashler and colleagues and An and Carr found that there is no credible scientific evidence (despite a multitude of studies) for the validity and usefulness of “learning styles” as a way to determine learning success.

Yang and colleagues found that personalized learning is effective when Felder and Silverman’s learning styles and cognitive styles were combined.

The specific cognitive style that they researched was field independence versus field dependence (i.e., the ability of the subject to tune out the surrounding context when information is presented).

Recommendations: Avoid personalizing learning based on learning styles alone.

Rather, to start, combine learning and cognitive styles, and test to see if this actually impacts performance. To do this, learning clearly defines the outcomes you are measuring.

CONCLUSION Personalized learning is both an art and a science. And proper personalization will likely require constant fine-tuning to the specific challenges that people face within organizations. Psychology and brain research can contribute greatly to idea generation as learning professionals improve and iterate on their design of personalized learning.

Dr. Srini Pillay is the CEO of NeuroBusiness Group. He is also assistant professor (parttime) at Harvard Medical School and teaches in the executive education programs at Harvard Business School and Duke CE. Email Srini.

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

| 11

CONGRATULATIONS

TOP 20 COMPANIES

Check Out Our 2017 Watch List Companies

The Top 20 Companies are a service provided by Training Industry, Inc. Due to the diversity of services offered, no attempt is made to rank the “Top 20s.”

PERFORMANCE MATTERS JULIE WINKLE GIULIONI

BEFORE-TRAINING ACTIVITIES

CAN NO LONGER BE AN AFTERTHOUGHT

Considerable time, energy and budget are invested in designing cuttingedge learning. Sophisticated followup mechanisms are developed and implemented. Progress is meticulously monitored and results are measured.

All the pieces are in place…then too frequently, we blow it. We half-heartedly compose and send off a lackluster “welcome to the class” email, setting the tone for less-than-stellar results.

What happens before the learning event is also an opportunity deserving of attention. And when this happens in a more deliberate way, individuals are able to personalize the experiences to their unique needs and gain considerably more value in the process.

Learning professionals who want to make the most of the precious time leading up to a learning initiative might consider these three key practices.

MAKE THE INVITATION MEMORABLE First impressions matter — in relationships and in the way we introduce learning opportunities. A traditional, boring, textheavy invitation sets a tone. Participants form expectations, about what the training will be like based upon that introduction. So, make sure you’re leaving a first impression that’s memorable and motivating. Have you considered: Sending a customized 3-D pop-up card that introduces the topic in a visual fashion?

Giving “voice” to your invitation by producing a podcast-style audio introduction?

PRIME THE LEARNING PUMP

INSPIRE INTENTION

The time between when participants make the decision to engage in learning and when it begins is particularly important. It’s a period that should be mined for as much pre-learning as possible. Presumably a skill or knowledge gap has been identified.

A need exists and, as a result, motivation levels may be elevated.

How frequently have you started a workshop by asking participants to share their expectations? It’s a great interactive opening, but the problem with expectations like these is that they tend to put the onus on the facilitator to somehow meet them.

In contrast, thoughtful personal intentionsetting in advance allows participants to take responsibility for and become active partners in their learning.

FIRST IMPRESSIONS ARE LASTING IMPRESSIONS. Take advantage of this learning limbo by engaging participants with the content.

Offer them appetizing opportunities to prepare and immerse themselves in the content. This is not your father’s pre-work — the old-school worksheet packets that frequently felt like a busy work burden. Rather, it’s provocative articles, short videos and teasers that aren’t required, but that are interesting and relevant enough to keep people clicking and mentally preparing for a richer learning experience.

This pre-learning time is also an excellent opportunity to build a sense of community as well as user-generated content.

Inviting participants to share resources, articles, tools and perspectives about the topic in advance accomplishes multiple objectives. It supports networking and sends a strong message about the active role they’ll be expected to play in their learning. It informs the facilitator/designer about the current level of awareness and knowledge. And it allows participants to consider the content in advance, elevating the quality of the actual experience.

So, invite participants to consider what they’ll be using and to set intentions, offering questions to ask themselves like: Where in my life might I be able to use this information?

What problems might I avoid if I get better at this?

If I improved in this area, what results might follow?

First impressions are lasting impressions.

When it comes to L&D efforts, what happens (or doesn’t happen) in advance of training dramatically affects everything that follows. So, let’s stop making it an afterthought. Because investing a little creativity and effort before is one of the simplest way to make the most of your carefully crafted content, design, facilitation and follow up.

Julie Winkle Giulioni has 25 years of experience working with organizations worldwide to improve performance through learning. Named one of Inc. Magazines top 100 leadership speakers, Julie is the coauthor of the bestseller, “Help Them Grow or Watch Them Go.” Email Julie.

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

| 13

TRANSFORM YOUR SKILLS IN JUST TWO DAYS. Training Industry certificates are designed to meet the specific needs of the training manager.

These programs provide the relevant knowledge and skill sets needed to manage training in support of specific functional areas, building skills as identified in Training Industry’s research-based Training Manager Competency Model™.

DISCOVER WHAT MAKES A GREAT LEADERSHIP TRAINING PROGRAM. LEARN STRATEGIES TO DEVELOP YOUR CURRENT AND FUTURE LEADERS.

LEADING LEADERSHIP DEVELOPMENT CERTIFICATE

EXPLORE THE EVOLUTION OF THE HIGHEST-PERFORMING TRAINING DEPARTMENTS. ACQUIRE THE TOOLS AND INSIGHTS TO TRANSFORM YOURS.

TRANSFORMING THE TRAINING FUNCTION CERTIFICATE

Learn more at www.trainingindustry.com/education.

BUILDING LEADERS SAM SHRIVER & MARSHALL GOLDSMITH

LEADERSHIP & LEARNING The two of us might never have crossed paths had it not been for Dr. Paul Hersey.

We both had the opportunity to work for him decades ago. Recently, we were together at an event and reminisced about our initial exposure to a model and a mentor that literally shaped our careers.

In large part, our reflections converged around the depth of the pioneering research that anchors Situational Leadership®. And in the context of the theme for this issue of Training Industry Magazine, we would offer the competencies that form the foundation of Situational Leadership® have significant crossover effect when considering the objectives of a contemporary learning strategy.

DIAGNOSE Good leaders have a plan! They analyze the circumstances surrounding their leadership opportunity, cultivate and refine awareness regarding their influence related triggers and impulses, then thoughtfully establish measurable objectives that will ultimately determine the degree of their success and effectiveness.

A successful and effective learning strategy follows a substantially similar path. It is with ever-increasing regularity that strategy challenges traditional convention; is thoughtfully tailored to account for cultural nuance; is tangibly measured by parameters that are joined at the hip with the overall business objectives the organization in question aspires to achieve; and is subject to calibration on the basis of emergent circumstance.

ADAPT Regardless of what may be most comfortable for them, good leaders adapt their approach based on the particulars of their diagnosis. In that regard, leaders need to be prepared to respond in a variety of different ways (i.e., directive, participative, empowering) based on the details of the situation and individual/ team they are attempting to influence.

Likewise, effective learning strategy needs to feature operational flexibility.

If content is deemed appropriate (or necessary) for a particular target audience, delivery of that content needs to be readily available in a format that reflects the ever-increasing spectrum of available options without sacrificing the impact of the training message itself.

Same goes for pull-through, transfer or reinforcement strategies.

COMMUNICATE Good leaders develop the ability to deliver an effective message regardless of what that message turns out to be (i.e., “Here’s what I need you to do”; “Let’s discuss what we should do here”). Good leaders also factor in the communication preferences of those they attempt to influence and tailor their delivery to ensure both understanding and acceptance.

Few would argue there is increasing attention in the learning community on the manner in which content needs to be positioned and sustained. Clearly, this is a good thing! By the same token the learning event itself, much like the leadership discussion itself, needs to be both relevant and engaging (regardless

of modality) each and every time. If it isn’t, the probability of transfer is reduced significantly.

ADVANCE Leaders add value by accelerating the development of those they influence and redirecting any performance related regression that may materialize along the way. One thing we know for sure, both of those dynamics will forever be in play in one way or another! As Dr. Hersey used to say (often): “Things are either getting better, or they are getting worse, nothing stays the same!”

GOOD LEADERS ADAPT THEIR APPROACH BASED ON THE PARTICULARS OF THEIR DIAGNOSIS. Much the same on the baseline value added by the professional learning community. Learning is both a mechanism for mastery and a process for developing forward-thinking perspective when we experience disappointments, set-backs, or the occasional “bumps in the road.”

So, in conclusion, we would offer that leadership is really all about learning, and learning is really all about leadership.

Marshall Goldsmith is the world authority in helping successful leaders get even better.

Sam Shriver is the senior vice president of commercial operations and product development at The Center for Leadership Studies. Email Marshall and Sam.

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

| 15

DATA IS NOT ONE-SIZE-FITS-ALL. IT VARIES BASED ON THE AUDIENCE IT’S BEING SERVED TO AND THE ANALYSIS IT’S UNDERGOING, AND IN A BUSINESS SETTING, IT’S NOT ALWAYS AVAILABLE TO EVERYONE. IT COMES IN DIFFERENT VARIETIES, AND CAN EITHER BE A HELPFUL TOOL, OR AN ENORMOUS TIME SINK. IF USED CORRECTLY, HOWEVER, IT’S A POWERFUL WAY TO INFORM STRATEGIC DECISION MAKING.

| 16

It seems like data is everywhere these days, and in fact, it is. According to an article in AppDeveloper Magazine, more data will be created in 2017 than all other years in modern history combined. The training industry is no exception. More tools are touting the ability to report on training activity, and xAPI and the Learning Record Store are increasingly popular. With increased availability, it’s important to

dive deep into the types of analyses that can be performed on training data to uncover insights that can inform program growth and development.

Depending on the analysis, there may be different cadences for measurement and analysis. Regardless, training data should be a routine part of any training professional’s workflow. In this article, the focus will be specifically on customer training and the three distinct

BY LINDA SCHWABER-COHEN

FROM DATA TO ANSWERS: USING A DATA-DRIVEN APPROACH TO SOLVE CUSTOMER TRAINING PROBLEMS

buckets or levels of metrics that are important to investigate.

The first type of data to consider is business-level data. This is used to understand the impact of training on the business as a whole, and is typically reviewed by department leaders and executives. Next is course-level data, which is used to gauge the traction of the training program. It uncovers

information about the effectiveness of course marketing tactics, and learner retention. Lastly, there’s content-level data, which tends to be used primarily by instructional designers for the purpose of evaluating different content assets.

A NOTE ABOUT DATA In companies both large and small, there are challenges around

gathering and analyzing data. In small companies, there may not be an existing infrastructure to collect and store the right data. In larger companies, that may exist, but it’s difficult to get the information from different departments, and security restrictions may require cutting through red tape. If data collection is an afterthought, it may be impossible to gather later.

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

| 17

With these challenges in mind, before investing in program enhancements, it’s important to consider if data is available to measure success. Choosing training tools and systems with robust data availability and integrations can help with this by alleviating some of the burden associated with data collection and storage. Additionally,

A TRAINING PROGRAM IS ONLY AS EFFECTIVE AS ITS CONTENT, AND DATA CAN EVALUATE CONTENT ASSETS. getting acquainted with the business metrics collected at a company and the colleagues associated with managing finance and customer relationship management software will be important for data access down the road.

BUSINESS-LEVEL DATA ROI, or return on investment, is one of the best ways to understand if business activities are yielding positive outcomes. Business- level data is typically tied to this metric.

Why is it so important to measure the ROI of a training program? For one, it helps with resource management and budget allocation. Demonstrating the outcomes of a program can help make sure that the resources

| 18

allotted to it are maintained, and help with the recruitment of new resources for program growth. It also aids in strategic decision making, by reflecting on real past performance rather than speculation. Keeping these motives in mind while evaluating data can help cater the analyses to different audiences, and maintain focus.

For business-level data in customer training, the audience is most likely training leadership, in addition to executives such as the vice president of customer success or professional services. While this audience cares how much the customers enjoy the learning experience, overall, they’re most interested in the impact of the program on their bottom line.

Surfacing the impact that the program has on key business metrics can help them understand ROI. For customer training, it’s typical to see organizations taking one of three approaches to demonstrating ROI. The approach depends on what is important to a given company and how leadership thinks of training as a lever in the customer journey. For some organizations, training is primarily a lever to improve customer retention, renewals and upsells. Other companies care more about reducing costs, or increasing product usage. All of these outcomes have financial benefits.

When looking at customer renewals, a key question to ask is whether training is a driver for customer health. One way to assess this is by calculating if customers who take training renew at a higher rate than those who don’t. The training course enrollment data and contract renewal data are needed to perform this calculation. Other relevant questions are, given the following accounts that expanded their contract value, what percentage took training?

Or, given the following accounts that churned this year, what percentage took training?

While many companies create an ROI story around training revenue, there is also a compelling case to be made around reduced costs. For example, a strong proactive training program will often aid in support ticket deflection.

Tracking the number of support tickets per customer or support tickets per topic over time can help demonstrate the impact of a training program that addresses these areas.

Another key area, especially for software companies, is product usage.

Increased product usage can lead to increased value for the customer, so it’s important to assess if training helps drive usage. To measure this, capture customer behavior with user behavior software or plug ins. Then, compare time spent using the product with learner session time (time spent in training). Ideally, a well-trained user will spend more time in the product.

Once these analyses are complete, it becomes simple to determine if there is a correlation between training and these positive or negative outcomes.

If a high percentage of customers who take training don’t renew, it’s cause to take another look at content and make sure it addresses the right knowledge gaps and value adds. From an executive perspective, failure to address the business need may be acceptable, so long as it’s evident that a data-driven approach is in play to course correct. It’s better to invest in making training meaningful and impactful than to continue to spend valuable resources on a program that’s ineffective. Whichever way things go, when presenting data to an executive audience, it’s useful to elevate the conversation to true business outcomes and align on expectations.

COURSE-LEVEL DATA Course-level data, or training engagement metrics are a training manager’s best friend. They help assess if the program is reaching the right people, if the content is engaging, and most of all, the data is often easiest to come by. The audience for this level of data tends to use it to inform their tactics for marketing coursework, onboarding customers and creating new courses.

Since every organization has different needs, it’s important to establish baselines from past performance, or if the program is new, take time to collect initial training data and see how different tactics affect outcomes.

The major metrics at play here are course registrations and completions, and student satisfaction.

Registration and completion data provides insight into which topics interest learners, or how discoverable courses are. If a customer registers for a course, but drops out, it’s possible

TRAINING DATA SHOULD BE A ROUTINE PART OF ANY TRAINING PROFESSIONAL’S WORKFLOW. that the training content wasn’t engaging or relevant. Additionally, this information can help optimize for course length, if drop off is consistent on longer courses.

Smile sheets, or student satisfaction surveys, determine whether participants find training favorable, engaging and relevant to their jobs. There are many possible questions to include on these surveys. Using a net promoter score (NPS) question can help align training program reviews with those of the company as a whole. They ask a simple question: How likely are you to recommend this training to a colleague? NPS is globally benchmarked, and many companies and training programs include it on surveys, so it’s useful for comparison.

CONTENT-LEVEL DATA A training program is of course only as effective as the content within it, and data can help with evaluating content assets, as well as prioritizing content creation and revision. Unfortunately, training professionals often “set and forget” their content. It can be hard to find the time to revisit and rework existing content with the variety of other day-to-day priorities. Since content can become quickly outdated, it’s advantageous to use data-driven approaches to prioritizing content revisions and creation.

The best place to begin is with the low-hanging fruit. Data can be used to determine what content is needed.

Start by analyzing the company’s most popular knowledge-based articles, as well as the most popular topics for user education support tickets. Then, create content that addresses those needs. This should prompt quick improvement in those areas.

Next, use data to keep the most popular content fresh. Look at registration and completion rates at the course level to understand where to focus revision efforts. This way, the company is putting its best foot forward for frequently viewed content.

CONCLUSION In this world of ever-increasing data, customer training programs can benefit in a multitude of ways through data analysis. From executive proposals for starting new initiatives, to instructional designer content prioritization, data is powerful. A solid understanding of the audience for the data, the specific metrics needed, and the purpose of the analysis will help avoid the risk of being overwhelmed and uncover valuable insights.

Linda Schwaber-Cohen’s expertise lies in building and growing onboarding and training programs at software companies.

She currently serves as head of training at Skilljar, a Seattle-based customer training platform and LMS. Email Linda.

CUSTOMER TRAINING DATA CHEAT SHEET Business-level data • Customer retention • Customer expansion • Training participation by account • Product usage data • Support tickets per account Course-level data • Course registrations and completions • Student satisfaction surveys • Training net promoter score Content-level data • Knowledge base traffic • Support ticket topic volume

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

| 19

www.niit.com

ADAPTIVE LEARNING

CAN ELEVATE CORPORATE TRAINING

BY NICK J. HOWE

Recent advances in virtual and augmented reality and in techniques such as gamification are creating excitement in corporate education, yet chief learning officers (CLOs) today still grapple with the same basic problems they’ve faced for more than 30 years: How to train employees and increase their skill levels effectively and efficiently.

Traditional online learning has been driven by cost and convenience, but efficacy of learning outcomes has been largely absent. What’s clearly needed is a better learning solution that satisfies the logistical need by offering courses “anytime, anywhere,” while also addressing the much larger issues of effectiveness and learner engagement.

Rather than rush to “cool yet unproven” solutions, CLOs should consider adaptive learning: a personalized, softwareenabled teaching approach that delivers proven outcomes. Research has consistently demonstrated the benefits of adaptive learning, especially in K-12 and post-secondary education, where it is widely deployed. For example, in a study of seven U.S. universities, adaptive learning increased passing and retention rates, as well as instructional efficiency.

Although adaptive learning is already deployed in most universities, and one-third of public school districts are

committed to adaptive learning, in the corporate space, it’s still an early adopter technology. Yet, judging by the attendees of the 2016 and 2017 Association for Talent Development (ATD) annual conferences, over the past year and a half there has been a dramatic uptick in awareness and acceptance of adaptive learning in both small and large enterprises. But there still seems to be some confusion about what adaptive learning means and what it’s capable of achieving.

As the marriage of computer science and cognitive research, adaptive learning delivers a personalized, online and tutorlike teaching experience at scale. For corporate learning, it offers tremendous potential to establish effective and time-efficient pathways to mastery that are unique to every individual, while improving business-based outcomes.

To showcase this potential, here are six ways adaptive learning can elevate corporate learning.

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

| 21

1

GREATER TIME EFFICIENCY – AND LESS TIME OFF THE JOB

A personalized adaptive approach can cut in half the amount of time it takes the typical learner to achieve mastery, compared to other learning approaches.

With one client project, for example, a two-and-a-half-day instructor-led course was converted to a series of adaptive learning modules. Most learners mastered the adaptive material in less than eight hours, and some achieved mastery in as little as four hours.

The reason is the personalized approach, which adapts to each learner. There is no need to reteach what people already know; instead, adaptive learning focuses on where they need to become competent. For workers in fields such as call centers, retail, or nursing, where time off the floor is critical, or for expensive resources like salespeople, improving time efficiency in training is crucial.

2

GREATER COMPETENCE MEANS BETTER OUTCOMES

Across every industry, there is a need to improve employee proficiency by identifying and addressing competency gaps. In the best-case scenario, employees are aware of what they don’t know – they are “consciously

ADAPTIVE LEARNING FOCUSES ON WHERE PEOPLE NEED TO BECOME COMPETENT. incompetent.” In the worst-case scenario, which is becoming more common, employees are unaware of the gaps in their understanding: they are “unconsciously incompetent.” Such ignorance can be very costly to the company and the satisfaction of its customers. Addressing conscious and unconscious incompetence is of the greatest importance when learning outcomes have clear consequences, such as driving revenue, improving safety, or addressing customer satisfaction.

Figure 1 depicts results of 317 salespeople who took an adaptive learning course. The blue lines to the right reflect correctly answered questions where learners were confident in their responses. The green bands are where they answered correctly, but were unsure. The lines to the left show where they struggled, knowingly or unknowingly. The yellow shows where learners didn’t know the answer and

knew they didn’t know it - they were “consciously incompetent.”

The problem area is in orange: Learners answered a question and said they were confident in their responses, but their answers were incorrect. Such “unconscious incompetence” is the source of many workplace errors and potentially serious ones. The best training course cannot be effective if it is not capable of identifying and remediating unconscious incompetence. Adaptive learning is unique in its ability to both identify and remediate for unconscious incompetence.

3

A QUESTION-BASED APPROACH GATHERS LEARNER DATA

Adaptive learning takes a question-based approach to learning, probing what the learner already knows and where they have gaps. The result is a large volume of very granular data, which makes it possible to analyze groups’ performance as a whole, in particular areas, or even on specific questions.

Adaptive learning also keeps track of what people learned, so if training needs to be updated, the course can be modified and made available to learners without worrying about material being redundant. Equally important, using a

FIGURE 1 Results of 317 learners in adaptive learning course.

-80%

-60%

-40%

-20%

0%

20%

Metacognitive Results

| 22

40%

60%

80%

100% n=317 Source: Area9 Learning

question-based approach helps build confidence along with competence as learners gain mastery and become surer of what they know.

4

PERSONALIZED LEARNING FOR A HETEROGENEOUS GROUP

Learners within any group are never the same. Tenure in position or in the company, as well as the skills, knowledge and experiences a person brings from previous jobs or the outside world, all make each individual unique.

Even individual learners are not the same day to day due to mood, health, the morning’s commute, even subtle choices such as drinking tea have been shown to affect learning and memory.

Adaptive learning is ideally suited to heterogeneous audiences – which really means all audiences. Adaptive learning adjusts to novices and experts alike, avoiding the dreaded “one size fits none” of traditional e-learning with its static content.

5

MOVING AWAY FROM “CHECK THE BOX” COMPLIANCE

In the corporate world, a subset of courses is often required to be taken repeatedly, year after year. Unfortunately, these tend to be dry and uninteresting from a content perspective. Compliance courses are perfect examples, despite them being critical to mitigate material risk to the company. Nonetheless, when people are forced to review dry content to simply “check the box” that they completed the course, very little learning typically happens, which undermines the original purpose of mitigating risk.

“Test-out” strategies allow employees who can prove they know the material to skip the course. But tests are gross approximations of the real world. And what if someone scores, say, 90 percent?

Are they forced to take the training, wasting their time as they cover material they already know – while hopefully still being engaged when they come to the material they don’t? Or is 90 percent “good enough?” How much risk is associated with the missing 10 percent?

THE KNOWING-DOING GAP The data generated from the adaptive learning process confirms learning has occurred. But there’s no guarantee the learner will apply this knowledge in the workplace. The next major advance in adaptive learning will be the ability to capture observational assessments of realworld, on-the-job performance and link

that back into the adaptive model – this is currently in the experimental stage. For now, there appears to be some indirect evidence of a link between confidence and transfer.

Anecdotally at least, the confidence instilled through the adaptive approach makes it more likely the person will apply knowledge and skills on the job.

Because adaptive learning’s questionbased approach involves the learner, even dry material becomes more engaging. It also allows people who are relatively proficient, thanks to taking repeated courses multiple times, to skip over what they’ve already mastered and focus only on what they don’t know.

By combining the assessment and the learning content into the adaptive engine, duplication is avoided - while remediating unconscious incompetence and the risk associated with it.

differently depending on the amount of content each learner was previously exposed to. Adaptive learning also provides the ability to incrementally author content, releasing the highestpriority subjects first and then adding new content to the system.

6

UPDATING TRAINING WHEN INFORMATION CHANGES FREQUENTLY

Traditional approaches to training are not well-suited to information that changes rapidly. In face-to-face instruction, the teacher can deliver the most up-to-date material. But what about those who went through the course previously?

Traditional online approaches do not accurately track what people have learned and could not adjust if they did, which makes it difficult to add new material without making learners go through everything again (wasting time and reducing engagement). Adding the new material as an addendum may work for those who have already taken the course, but it can confuse new learners.

To avoid such messiness, companies often limit the number of updates, but that delays new information getting out to the employees.

The solution, once again, is adaptive learning. When changes to the course are introduced, the system can differentiate between material a learner has already covered and new areas to be mastered.

In fact, two people could take the same course, and the system would behave

USING A QUESTION-BASED APPROACH HELPS BUILD CONFIDENCE ALONG WITH COMPETENCE. There are many other benefits to adaptive learning, but in closing, it’s important to note one in particular that is crucial for CLOs: line of sight. CLOs look for a “seat at the table” as an integral member of the executive team. That requires speaking the language of the business and demonstrating business impact. Measurement of training impact is notoriously difficult, and adaptive learning provides a clear advancement in this area. The combination of closely aligning learning objectives to business outcomes, along with granular evidence from the formative assessment process, gives CLOs the capability to effect and measure tangible change in the workforce, becoming a generator of business value.

Nick J. Howe is chief learning officer of Area9 Learning, a provider of adaptive learning technology across a range of industries, specializing in sectors particularly wellsuited to the adaptive approach. Email Nick.

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

| 23

Facilitating a Robust Social Learning Experience for Geographically Dispersed Teams

R

on, the senior vice president of sales, has just called to give you a heads up – the CEO is about to announce there has been a merger and the company’s product line will be expanded. All existing sales teams will be reconfigured, and team members will need training on the new product lines. The learning and development (L&D) team is responsible for training the sales staff and facilitating their transition into new, globally dispersed teams. You have two weeks to formulate a plan to train 1,200 individuals across five different time zones. Many of these individuals have never met face to face and will be working virtually most of the time. Sound familiar?

Whether your company is large or small, geographically dispersed teams are becoming more common, requiring L&D departments to develop effective training on content while facilitating virtual team building at the same time.

While there are many techniques to address these two items separately, social learning provides a highly flexible, low-cost way to tackle learning and team building together. In fact, teams may already be incorporating social learning techniques into their daily routine! This article will explore strategies you can use to promote engagement within virtual teams by integrating social learning experiences into traditional learning programs.

BY DAILA BOUFFORD

| 24

The challenges within virtual teams are well-known. Communication, or lack thereof, is a primary pain point.

Research on methods of communication has long promoted the importance of visual cues, such as body language, to truly understand what is being said.

Most virtual teams communicate via email or phone and are not able to pick up on these important visual cues.

Virtual teams also suffer from a lack of shared experience. A funny moment in the break room, a favorite restaurant or

a local cause to rally around all create shared experiences that build emotional connections within a team. Teams with limited face-to-face time have a greater degree of social distance, meaning they have trouble establishing an emotional connection with one another, which limits their ability to trust each other.

These inherent challenges must be overcome to ensure a successful and effective learning experience.

INTEGRATING SOCIAL LEARNING ACTIVITIES WITHIN TRADITIONAL e-LEARNING

Social learning typically refers to individuals learning from each other as they share an experience. It can occur online or in person and the most effective programs tend to incorporate a mix of the two environments. On a recent project, I designed a series of social learning components to augment a traditional e-learning course. The client was looking for a way to bring a geographically dispersed team together, strengthen their relationships and create a more engaging learning experience. Building on the traditional online discussion forum, we developed creative ways for the team to connect with one another. For example, during a lesson on active listening we asked the learners to post a fun image and caption it with one of the benefits of listening actively. Extra points were awarded for creativity in image selection. Not only did this get the learners to think about the content in a new way, it allowed them to develop an appreciation of one another’s sense of humor as they reacted to each other’s posts.

Integrating social learning activities into traditional e-learning has many benefits and can be a great way to develop relationships.

Although mentoring often comes to mind, social learning activities also develop

relationships between peers and even among business units. For example, an internal sabbatical program that allows individuals to be “loaned out” to a team in a different business unit can establish connections among teams

BE PROACTIVE IN CREATING CONNECTIONS BETWEEN LEARNING ACTIVITIES AND REALTIME WORK.

that normally don’t interface with each other. The individual being “loaned out” engages with the new team in their daily work while providing a fresh perspective based on his experience in a different part of the company.

This activity may require more face-toface time than others, but if done well, it can lay the foundation for lasting partnerships that provide benefits long after the sabbatical has ended.

OVERCOMING CHALLENGES WHEN ESTABLISHING SOCIAL LEARNING PROGRAMS

Social learning has received some criticism over the years. There are many instances where the final output doesn’t quite measure up to the stated objectives for the activity. When incorporating social learning activities, it helps to have a long-term view of success. There may be some quick wins, but the real value is in the learning

networks that are established as the result of social learning experience.

Facilitating social learning for geographically dispersed teams requires careful planning. The technology tools that enable discussion and sharing need to be user friendly, and are ideally something that the learner is already familiar with. If a new platform or software is needed, be sure to provide technical support and encouragement as learners become accustomed to using the tools.

To help managers see the value and promote social learning within their team, design activities in a way that allows learners to share their knowledge.

For example, ask the learner to arrange a one-on-one with their manager to discuss what they learned and how it can be applied to a current project.

Or, encourage the learner to share a challenge they are facing with their learning cohort, so the group can tackle it together. Be proactive in creating connections between learning activities and real-time work. This ensures your objectives are on track and is the best way to help learners retain the content.

THE INFLUENCE OF CULTURE WITHIN SOCIAL LEARNING

When working with global teams, it is essential to understand how culture affects the social learning experience.

“The Culture Map” by Erin Meyer is an excellent resource to better understand how communication norms and business practices differ across the globe. Culture affects how we manage conflict, how we perceive other’s actions and words and how we participate in shared learning experiences. For example, in group discussions, individuals from hierarchical cultures may be less inclined to speak up, especially if there are higher ranking members among the group. Identify

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

| 25

potential cultural barriers as you design the experience and create a plan for all learners to feel safe when participating.

Cultural differences can lead to conflict. Although difficult to manage, a conflict-based exercise can be a fun catalyst for learning. When used effectively, conflict allows learners to challenge their existing views, increase their cultural awareness and assess their initial understanding of the content. It is important to prepare for a conflict-based activity by building a strong foundation of trust so that learners feel comfortable in expressing their opinions. Using a conflict activity in a discussion board format can be troublesome, as a learner with an outside view may feel ostracized from the group. It is better to establish a team-based social learning activity that builds comradery while challenging common assumptions. See sidebar for an example of a conflict-based exercise.

OVERCOMING CHALLENGES WHEN WORKING WITH GLOBAL VIRTUAL TEAMS

As you design social learning activities, there are a few concerns to keep in mind. Although English is the

predominant language used within the business environment, it is important to recognize that it is a secondary language for many, which can impede participation in certain activities.

Consider how you can accommodate those who may struggle with a rapid

DESIGN ACTIVITIES IN A WAY THAT ALLOWS LEARNERS TO SHARE THEIR KNOWLEDGE.

pace of communication. Design activities that dial down the dominance of native speakers and dial up the engagement of those who are less fluent. Encourage participants to refrain from using euphemisms that may be hard to understand or even offensive to a culturally diverse population.

A CONFLICT-BASED LEARNING EXERCISE A team debate is a great example of a conflict learning exercise. Divide your cohort into two teams. Create an online networking space for each team using a tool such as Yammer, Slack, etc. Give the teams a week or two to collaborate and build up their case. Then, host a debate via video conference. As the debate takes place, team members can communicate supporting information

| 26

to each other via a chat tool, such as Skype. In this exercise, a geographically dispersed team dives deeper into the content through sharing perspectives with their peers. They can identify gaps in their learning and develop an appreciation for differing points of view.

The social activity creates a shared experience for the team to build on as they continue to work and learn together.

Although global teams benefit from a diversity of backgrounds and experiences, it is important that they are also able to rally around a unifying factor. Consider adding a fun activity at the start of the course to help teams develop their emotional connection with each other. One idea is to have the team come up with a mascot or team motto.

After the training is complete, this can serve as a reminder of what was learned or a commitment to performance. With each team training activity, you are adding to the team’s shared experience, helping them to create their own unique team culture.

Social learning serves as a bridge between formal learning and learning on the job. Individuals and teams should be encouraged to seek out social learning opportunities on their own, in addition to formal learning programs.

In my organization, our instructional design team gets together once a month on Skype to share insights from articles, conferences and webinars. These meetings ensure that we regularly take time to seek out new knowledge and hone our skills.

I’ve been part of a virtual team for eight years and have enjoyed participating in social learning activities that make me feel as if my colleagues were down the hall instead of half a world away. So, the next time you are faced with the dilemma of how to get a geographically dispersed team trained and communicating effectively, consider how social learning may play a role. The tools are easy to use, the design possibilities are endless, and the benefits are lifelong.

Daila Boufford is the manager of learning design at ansrsource. She has more than 15 years of professional experience in client service, content development and instructional design. Email Daila.

If You Want Your Best Employees Around in the Future, Give Them One Cornerstone’s software helps the world’s leading organizations to realize the potential of their people.

Find Out More at csod.com/training

creating training

appropriate to the role

By Ashley Korneagay, CPTM

Rob Liano once said: “Knowledge is power? No. Knowledge on its own is nothing, but the application of useful knowledge, now that is powerful.”

Adapting a role-based training curriculum produces training that can be consistently applied to an employee’s job.

Oftentimes, employee feedback reveals that they feel over-trained in various functions such as soft skills, and undertrained in other core competencies that directly pertain to their day-to-day role. Middlesex University conducted a study on work-based learning and discovered that 74 percent of the 4,300 workers felt that they weren’t achieving their full potential at work. The creation of training appropriate to the role ensures that employees are receiving training that will be applicable to them and in return will help them meet their full potential.

Employees possess a desire and a need for training that is pertinent to what they actually do. They want to be able to learn more about their jobs. A 2016 study by Udemy revealed that 44 percent of respondents cite a lack of learning opportunities as a reason they left their last job. That is a significant number of employees looking for opportunities outside of their current role. It is the role of the learning developer to create training appropriate to the employee’s role. Here are a few steps to developing role-based training.

KNOW YOUR AUDIENCE While general training provides a basic understanding, the trick is to design and deliver training content in a digestible manner so that employees can apply it to their roles. When designing rolebased training, it is wise to start with getting to know the audience. The learning developers should have a basic understanding of what the audience’s

| 28

roles and responsibilities are within the organization. This will help to establish the appropriate areas to cover as well as the desired outcome from the training.

Learning about the audience can be done through reviewing internal documents and procedures, conducting interviews, performing on-the-job observation, or even spending time in the employee’s shoes. The learning developer should gather and document the knowledge gained from these observations. The next step is analyzing this information and pulling from it everything that can and should be used for developing the training regimen. This knowledge will help ensure that the training is significant and impactful.

CREATE A CULTURE OF ACCOUNTABILITY A 24X7 learning survey revealed that only 12 percent of learners say they apply the skills from the training they receive to their job. This is where learning developers need to not just know their audience, but understand their role within the organization. Establish a specific action for the audience to take with them to establish employee accountability and ownership.

When employees see how they can apply their knowledge they continue to be engaged. Training should not just cover the “what’s in it for me,” but also educate employees on how this impacts them and what to do next. This leaves them with a better understanding of how they can apply what they learned to their jobs.

Adapting role-specific training does not necessarily mean reinventing the wheel or constantly creating custom content.

Organizations should have documented policies and procedures for their business practices. Training should always direct employees to applicable policies and

procedures. While much of the time, these documents may be sitting on the top shelf, it is time to take them down and blow off the dust. Acclimating these policies and procedures into the training content has multiple purposes. For starters, it keeps the documents living and breathing instead of becoming forgotten about. Often utilized policies and procedures are frequently viewed by key stakeholders and subject matter experts, which is prime motivation for keeping them up to date.

EMPLOYEES POSSESS A DESIRE AND A NEED FOR TRAINING THAT IS PERTINENT TO WHAT THEY ACTUALLY DO. ONLY PROVIDE APPLICABLE CONTENT One out of three employees say that “uninspiring content” is a barrier to learning. Training is most optimal when the content is customized to what the employee does on a day-to-day basis.

For example, is the training for a customer service employee who handles general customer inquiries the same training given to a collections employee making outbound calls to collect on a debt? Both employees are speaking to their company’s customers, but they each have their own procedures as their roles are different. Should an employee who works with welding equipment receive the same training as a machine operator? Both need to know about safety procedures and requirements, but more specifically, they should be trained

on what could happen during their shift and what to do in that scenario.

Similar job positions are often combined from a training perspective, but might not be the best approach. Role-based training should not be one-size-fitsall. Training needs to be tailored to the employee’s role so that it can be wholly applied on the job.

TRAINING DELIVERY When developing the training content, the material may change depending on the direction and overall tone of the training. This delivery execution is critical to how employees will retain and apply the material. In a growing age of e-learning, a virtual course may have the same (or greater) impact than a meeting that lasts for an hour.

According to eLearning Industry, corporate e-learning has increased by an astonishing 900 percent over the last 16 years. When working through scenarios, an interactive virtual classroom or live cohort may be the best choice. In a primarily Gen-X workforce, ATD reports that social learning approaches have a 75-to-1 ROI ratio over web-based training.

In addition to knowing the roles and responsibilities of the audience, take into consideration factors such as classroom size and participants’ location. These factors will also have an impact on deciding how to best conduct training.

Strong training content alone does not make a training session impactful. While the content is good, the delivery and presentation materials should have an equal impact. To tie the training content to the overall session, supporting training materials and activities keep the audience engaged. When generating the supporting training materials,

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

| 29

put the trainee “in the moment.” The training should speak as specifically as possible to that employee and his or her day-to-day tasks.

IMPLEMENT REINFORCEMENT AND EVALUATION The next step is to monitor and ensure that employees are applying what they have learned. Reinforcement helps to close any gaps and ensures that employees are knowledgeable and applying what they have learned.

Again, this assessment should be tailored to the employee’s role. During the “getting to know your audience” phase, it is important to determine how the audience is evaluated. The training reinforcement should equate to the employee’s normal evaluation.

As employees are evaluated on their performance after training, it should mirror how they are evaluated on their day-to-day performance.

STRONG TRAINING CONTENT ALONE DOES NOT MAKE A TRAINING SESSION IMPACTFUL.

CONCLUSION consider how the employee performs his or her job.

Aligning training materials with the employee’s role will in return help the employee apply what they have learned back on the job. If the employee’s role is primarily working independently at a desk performing a specific task, selfpaced e-learning may align best with how the employee prefers to learn.

Taking employees out of their comfort zone may be helpful in some situations, but if the audience changes their focus or becomes distracted then the training will be less beneficial to them. On the flip side, employees who work in sales and often interact with groups of people would benefit more from a classroom and/or group setting. This

| 30

training could include activities and interactions that promote group interaction and mirror their daily atmosphere. Having an instructor lead the training session will help to drive conversation and interactivity.

Providing applicable case studies are great reference materials on real-life examples. Working through scenarios and simulations often have a greater impact on professionals who are specialized in a specific skill, such as a heart surgeon, machine operator, manager or bank teller. These learning scenarios could be conducted individually or as a group, depending on the employee’s role. This is a great time to be innovative. With growing technology, virtual simulations could

The average employee only devotes 1 percent of their work week to training.

That equates to 4.8 minutes a day and 24 minutes a week. As learning professionals, it is critical to help employees make the most of that 1 percent. Role-based training provides employees with the tools and resources they need to not just do their job, but to perform it accurately. Role-applicable training can optimize an employee’s skill set and set them up for long-term success in their role.

Ashley Korneagay, CPTM, is a regulatory compliance training program specialist and administrator at Alliance Data. She is responsible for executing strategic training plans and delivering role-based training opportunities to compliance professionals and first line of defense employees. Email Ashley.

USING DEVELOPMENT MAPS TO ALIGN AND UNLOCK PERFORMANCE

BY KELLY SMITH & BRIAN BLECKE

| 32

LIKE

A ROAD MAP GUIDES YOU TO YOUR DESTINATION, A DEVELOPMENT MAP GUIDES EMPLOYEES ALONG THEIR DEVELOPMENT JOURNEY. SIMPLY STATED, A DEVELOPMENT MAP PROVIDES A PICTURE OF THE LEARNING AND DEVELOPMENT (L&D) EXPERIENCES NECESSARY TO MOVE FROM ONE LEVEL OF JOB PERFORMANCE TO THE NEXT.

Looked at another way, each level of performance is a common destination, and is an outcome to be achieved by all members of the role. And because each employee brings unique experiences, knowledge and skill to the role, development maps are adaptable to the employee, with the potential for unique, individualized journeys from one outcome to the next.

When designed well, development maps balance the various needs and goals of the business, the L&D function, and the employee. Creating a development map is an opportunity for business leaders and L&D to work together to look holistically at a role and to identify and optimize the sequence of learning needed to achieve levels of job performance. It is because of this performance context that decisions about learning and development are more balanced, more economical, more efficient and more valuable to the organization and to the employees.

ELEMENTS OF A DEVELOPMENT MAP

Development maps are different than traditional training plans or course catalogs because they move beyond formal training offered in isolation to establish a comprehensive roadmap

encompassing all of the developmental experiences needed to achieve a measurable performance outcome.

By naming and showing the relationship between various L&D experiences (i.e., formal learning, informal learning, on-the-job experiences, available resources, which may include jobaid libraries, intranet sites, content delivered via employee performance support systems, etc.), a development map provides a more complete and realistic roadmap to performance.

During the process of creating a development map, learning outcomes, learning modalities and learning technologies are evaluated, and decisions and tradeoffs are made to balance impact, effectiveness, and other factors in consideration of the entire development journey, which helps resist the inclination to chase the latest “shiny object.�

FOR A DEVELOPMENT MAP TO WORK EFFECTIVELY, IT SHOULD ALIGN WITH BUSINESS GOALS.

Optimizing the learning experience takes on different characteristics when contemplating a development journey stretched across a year or more versus when contemplating a solitary learning event. When tightly connecting performance outcomes to business goals, those considerations evolve yet again.

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

| 33

A development map may use different types of development events to achieve a performance outcome (e.g., informal learning, on-the-job training, formal instruction, or resources).

Well-designed development maps clearly anchor the developmental experience to performance and connect various knowledge and skill topics to job performance. Development maps answer the question, “When will I use what I’m learning?”

The first step to creating a development map is to effectively uncover and organize the performance outcomes to be achieved (from there you can define the development events needed to achieve those outcomes). For example, if an employee’s role requires them to troubleshoot, there may be multiple levels of performance to be achieved over some period of time.

Performance outcomes may differ in terms of the number of stages needed to achieve the required level of performance. In most roles, one performance outcome is often a building block to more advanced performance downstream. Figure 1 provides a conceptual view of a development map for a management role.

As employees move left-to-right across the development map, they encounter a variety of development experiences and assessment milestones.

If the work hasn’t been done to organize the developmental journey, employees are left on their own to sort out how to use multiple training paths, lists, or checklists to navigate their way through onboarding, compliance training, soft skills training, technical skills training, etc. It can be difficult to fit all the pieces together – taking more time than is necessary, negatively impacting employee engagement and satisfaction.

While development maps provide structure, they should allow for flexibility where warranted.

In the example provided in Figure 1,

| 34

Figure 1.

DEVELOPMENT MAP WITH MULTIPLE STAGES

STAGE 1 Entry-level

STAGE 2 Proficient in Role

STAGE 3 Advanced in Role

Managing Work Track To Next Leadership Role

Managing People Track Managing Results Track Off-Ramp to Parallel Leadership Roles

KEY Performance Outcome

Assessment Milestone

most employees would follow the progression from one performance outcome to the next – completing all the development events indicated.

However, using performance-based assessments can allow the learner to skip certain developmental events or jump past a performance outcome.

While Figure 1 provides a very simplified example, a development map is often graphical, making it a user-friendly tool that shows an entire development journey on a single page. Much like a blueprint for a home lays out the home’s structure, but doesn’t show the interior design of each room, a development map shows the overarching structure of the development journey, but doesn’t include all the instructional design details.

A development map shows relationships among development events using proximity, lines and arrows to display sequence and dependencies or connections among events.

BUSINESS ALIGNMENT

For a development map to work effectively, particularly for business leaders, it should align with business goals. When business leaders can trace the connection between their goals (both short- and long-term) and the performance outcomes depicted on the development map, it gives them confidence and bolsters their support of the training and the L&D function.

Establishing clearly defined performance outcomes and timelines is the important link back to the business goals.

When creating development maps, it is necessary to work with business stakeholders and top performers in the role to capture and understand: • Performance dimensions • Performance outcomes • Work outcomes • Processes and tasks

• Measures of quality, quantity and cost • Knowledge and skill requirements • Typical gaps Additionally, organizational performance and financial data should be used to more fully understand the performance outcomes and to correctly organize and sequence them. As xAPI and other business intelligence capabilities continue to mature, the process for creating and maintaining development maps will evolve.

Traditional training design processes (like ADDIE) don’t work well as a method for creating development maps. Like the process for interior design is different from the process of creating an architectural blueprint, development maps require a different approach. The focus of a development map is often the entire role and accounts for all the development and learning needs from day one to mastery of the role, and that scope requires the designer to approach the design in a different manner.

A development map allows the design team to engineer and sequence learning and development experiences – not just by linking one learning objective to the next, but by linking one aspect of the job to the next. This mindset encourages the designer to assemble the smaller building blocks of courses and experiences into larger models of learning and ultimately to performance outcomes.

In part, the process requires consideration of a wide range of developmental modalities – both formal and informal, as well as real world work experience. Along the way, tradeoff decisions must be made based on availability of resources, budget and projected return on investment.

Tricky though it is, development maps provide an excellent opportunity to judge the value of each aspect of performance in comparison to others.

Some aspects require investment

while others can remain in the “figure it out on their own” category.

BENEFITS OF A DEVELOPMENT MAP

Business leaders, the L&D function and the employee are all stakeholders who benefit from development maps.

For employees

A development map is a practical, simple to use tool that helps them stage and put a timeline to their growth in their role and in the organization. Because development maps demonstrate that the organization has done its homework, employees often see development maps as evidence of the organization’s commitment to their success and respond in kind with greater commitment and higher engagement.

DEVELOPMENT MAPS PROVIDE A MEANS AND A MECHANISM TO CHANGE THE GAME. For managers

For managers, a development map simplifies employee career and development planning activities by eliminating waste from the process and saving time. By using a comprehensive roadmap, managers and employees avoid scouring the organization for resources as they plan “what comes next” in the employee’s development journey.

Managers are able to help employees make turn-by-turn decisions to reach their destination.

tactical goals to important performance and learning outcomes.

Since performance outcomes can be traced to business goals and are evaluated using meaningful metrics, it is easier to demonstrate measurable business value. With learning outcomes set inside the context of performance, the impact of learning can more readily be determined. Invariably, when the goals and outcomes are tied to meaningful metrics and results, perceived and actual value goes up.

For L&D

Development maps also benefit the L&D function. First, they depict a common perspective shared with business stakeholders that ultimately simplifies communication with those business partners. Second, since the map can trace learning outcomes through performance outcomes to business goals, decisions regarding the importance and value of development activities can be assessed, ultimately guiding L&D decisions. And third, when dealing with ad hoc training requests throughout the year, development maps can be used to judge requests based on how well the request fits (or whether the request is redundant in some way). Development maps are tools that help L&D in their role as a trusted advisor to their business partners.

CONCLUSION

When designed well, development maps unlock and improve employee performance and bring into balance the goals of the business, L&D and the learner. Aligning around business and performance, development maps provide a means and a mechanism to change the game.

Kelly Smith and Brian Blecke are founding partners and performance consultants at Actio Learning. Email Kelly and Brian.

For business stakeholders

Business stakeholders can use the maps to trace the organization’s strategic and

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

| 35

FUTURE-PROOF TRAINING The need to demonstrate the business impact of learning is paramount in today’s data-driven, increasingly transparent world. Today’s fast-paced and often disruptive business climate makes it harder than ever to engage diverse skills, abilities and backgrounds and ensure organizational readiness to perform, let alone quantify it. Geographic dispersion often compounds the challenge.

To solve these problems, CLOs are turning to adaptive learning, or systems of artificial intelligence that optimize each learner’s experience in real time, enabling him or her to learn as efficiently and effectively as possible.

The technology creates a more humancentered vision of learning that is also more scientific and can be directly tied to business outcomes.

At their best, adaptive learning platforms provide CLOs with technology so finetuned and intelligent, it’s like having a one-on-one instructor for every learner.

Essentially, adaptive drives learning mastery in a personalized way at scale.

MASTERY, PERSONALIZATION, SCALE Mastery-based learning, the idea that learners’ progression through a course is dependent on mastery as opposed to seat time, lies at the core of adaptive learning. The concept is actually intrinsic to corporate life. Our progress up the corporate ladder is based on factors generally unrelated to seat time: achievement, initiative and networking.

Progress isn’t about the passage of time, it’s about focus and intensity. Adaptive learning applies this principle to learning: If mastery is what’s important in learning, then why not optimize for it?

Adaptive technology takes this concept of mastery one step further by personalizing it, so that each learner progresses not only according to his or her accuracy level but other factors like confidence and engagement as well.

To achieve this, adaptive learning incorporates algorithms. Different theories work together to give adaptive

| 36

learning its reflexes, the way an instructor might pivot strategies from learner to learner.

For example, metacognitive theory holds that people learn best when they know what they know and don’t know. Another principle often used in adaptive learning is the theory of deliberate practice, which suggests that understanding weaknesses helps refine and focus practice. To address this principle, an adaptive learning platform continually tailors content based on the individual learner’s weaknesses, saving time and focusing energy for maximum efficiency. At the same time, this theory can be tempered by the theory of fun for game design, which holds that learners are most engaged when they are challenged but not too challenged.

Adaptive learning can put this concept to work by continually adjusting the content to challenge but not overwhelm the learner.

Additionally, the concept of spaced repetition, which holds that to truly learn something, learners must commit it to long-term memory, and that the best time to do so is just as learners are about to forget it. Incorporating this theory, an adaptive platform can use data to predict when a learner is likely to lose a concept from short-term memory.

WITH ADAPTIVE LEARNING

BY CHRISTINA YU

It can then automatically reintroduce this content just before it slips away, solidifying it in the learner’s mind as long-term memory.

It’s called adaptive learning not just because it adapts seamlessly to a variety of learners, but also because it’s flexible enough to transform almost any content – technical, qualitative or quantitative – into an adaptive course. Whether it’s technical skills, accounting or public speaking, a successful adaptive learning platform can take any content and transform it into a modularized course that’s responsive to each learner’s specific needs and capabilities.

OUTCOMES FOR INDIVIDUALS VS. ORGANIZATIONS For individuals, the benefits are clear.

Adaptive learning improves outcomes, including proficiency rates, efficiency and engagement. Time and energy are allocated to the areas where a learner most needs to focus in order to make the greatest gains. Learning becomes a more satisfying process and engagement levels improve because learners always have a clear path forward. They’re presented with the content they need to see at the precise moment they need to see it.

For corporate organizations, the benefits are holistic and far-ranging. Adaptive technology drives training ROI and unlocks organizational performance. I would loosely break down the benefits into five categories.

PROGRESS ISN’T ABOUT THE PASSAGE OF TIME, IT’S ABOUT FOCUS AND INTENSITY.

Whatever the ontology or the authoring mechanisms, adaptive requires some granular mapping or breakdown of concepts that lays the foundation for true measurement.

Because it collects data on an atomic level, adaptive yields insight into learning down to the granular level of the learning objective and each learner’s interaction with it. This is data we can use to measure the process of learning and find patterns, tie learning to business outcomes, and most importantly, optimize each learner’s progress toward mastery. This happens in different ways depending on the platform, but is typically surfaced through reporting dashboards and helpful visualizations natively built into the adaptive program.

2 | MASTERY 1 | MEASUREMENT According to a CEB industry report, 36.7 percent of L&D teams say that “measuring learning impact” is their top priority. Learning teams need to articulate a next-generation vision of learning that is both human-centric in design and delivery and directly tied to business outcomes. Because adaptive learning requires a fine-grained approach, it achieves this naturally.

As described above, adaptive is inherently mastery-based, not seattime based. The technology, in other words, is optimized for mastery.

Accordingly, each learner in an adaptive program achieves 100 percent mastery of the specified objectives. In a situation where compliance is at stake, 70 percent mastery is different from 100 percent mastery. The gap can translate into misallocated funds or a major security breach.

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

| 37

CASE STUDY | TRACKING TIME TO COMPLETION

Figure 1.

10

HRS

5 AVERAGE = 3.4HRS

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 LEARNER

In traditional education, we’ve been accustomed to accepting the standard outcomes distribution curve, where some perform well, and others don’t.

Exemplary performance is usually reserved for those who are gifted in a subject or who possess great discipline.

(The very concept of grades and an allconsuming final assessment condition is to accept less from ourselves. A vs.

B students, for example.) But what if real mastery, rather than relative performance, were the goal for each individual? Adaptive learning puts each learner on the path to actually achieving total mastery. On an individual basis, this is a nice-to-have. On an organizational basis, when compliance and security are at stake, it’s a must-have.

3 | EFFICIENCY Personalized learning paths that show each learner only what they need to see at the moment they need to see it results in time saved. In Figure 1, the time-tocompletion for the one-size-fits-all course was originally five hours. With an adaptive course, in which each learner achieves mastery in their own way, the average time-to-completion became 3.4 hours, resulting in an immediate ROI of 32 percent in full-time hours of work recovered, the saved time redistributed toward other activities.

Not only is the course more effective (the learners on the right achieving mastery where they might not have otherwise),

| 38

it’s more efficient (the learners on the left achieved mastery in less time than they would have otherwise).

4 | ENGAGEMENT There’s a lot of talk in the corporate space about engagement. The perks (free dinner and yoga, massages in the office), the culture (whether it’s “radical honesty,” “ownership,” or “humility”) and the social recognition (likes, upvoting, badges) that produce engagement,

ADAPTIVE LEARNING IS INHERENTLY MASTERY-BASED, NOT SEAT-TIME BASED. the X-factor that is supposed to make an organization blossom and achieve relentless innovation and competitive advantage.

This is all beneficial, but I want to argue for something more straightforward. Can we motivate learners simply by showing them a clear path to attaining their goals

and by providing immediate feedback and targeted remediation along the way?

By focusing on the organization of content and optimizing each learner’s interaction with every piece of it, adaptive motivates learners by keeping them in a state of flow. It removes distraction and boredom and drives focus, like a cognitive “workout machine.”

5 | AGILITY Real-time intelligence for every stakeholder (learners, trainers, content authors and managers) results in more real-time action and responsiveness for each stakeholder.

Learners, for instance, can access reports on their performance at any given point and are able to adjust their practice accordingly. Trainers/managers grasp cohort dynamics through real-time analytics and know exactly what their learners are struggling with, so they can quickly adapt training. Authors understand what content works and doesn’t work, so they can continually refine content and keep it evergreen.

At its core, the fine-grained data-driven approach of adaptive delivers benefits to every stakeholder, holistically powering the organization and making it more agile.

Christina Yu is the head of marketing for McGraw-Hill Education Learning Science Platforms. Email Christina.

Cultural Considerations in a Non-Global Organization

BY CAROL E. BROGAN

| 40

C

ulture is tangible and intangible.

It is in what is seen, but more importantly it is in what is experienced in nearly every interaction that occurs.

Even in a seemingly mono-cultural group of people, cultural considerations must be taken into account when an instructional designer sits down at the computer, when a trainer enters the learning environment, when a coach or mentor interacts with others, and when a chief learning officer executes a strategy.

It has been ascertained that when cultural consideration is not given, diverse teams are not as likely to innovate as homogeneous teams are due to perceived or real cultural incongruity.

There are few homogeneous groups within organizations, even for those operating at local levels. Intentionally considering cultural differences and accentuating the strengths will increase innovation and engagement, ultimately making for a stronger learning culture and greater business returns.

What is really meant by cultural consideration, and for that matter, intercultural interactions? A large part of each person, especially as it relates to behaviors in the workplace, is cultural and related to worldviews and cultural drivers. As much as segments of society may try to categorize and separate people, cultures are not uniform and differences within cultures are sometimes profound.

In the standard sense of the definition, cultural contexts include national, ethnic, generational, organizational, gender and sexual orientation-based, institutional, and a variety of other contexts and sub-contexts. National cultures exert a powerful influence, but so do generational, disciplinary and institutional cultures, and their effects are difficult to discern, analyze and disentangle. Embedded in the larger cultural contexts are micro-cultures that form based on commonalities and similarities, yet may be divisive across the spectrum. Individuals within cultures differ and bring different

personality traits that bear on any cross-cultural interaction.

Attributes of the diversity found within society include values placed on education, religion, individuality and status, just to name a few. People bring backgrounds grounded in their cultural upbringing. Things like goals, aspirations and ways of interacting that individuals feel are positive, may conflict with others’ goals and ways of interacting. Personal motivations are deeply influenced by, if not caused by,

People bring backgrounds grounded in their cultural upbringing. cultural factors and nuances. No wonder even within a non-global workforce culture needs to be considered!

OPPORTUNITIES TO ACCENTUATE VARIOUS VIEWS IN THE LEARNING CULTURE Interestingly, cultural values, characteristics and preferences that develop with ethnic or national upbringing tend to merge, adapt, or alter according to predominate organizational or local cultural values, but they do not disappear. They remain underlying and are impactful.

Learning professionals must accept and acknowledge these and find ways to utilize, and in the best situations, accentuate them for the positive.

Learning professionals must listen and observe when walking through the hallways, sitting in their cubicles or workspace, eating in the common

area. Most importantly, observing others must take place in the learning environment, whether live or virtual.

Professionals must check for accuracy of their own interpretations of the observations. Resources abound on predominate cultural traits, values and attributes. Even asking questions of the learners directly is an invaluable tool in assessing cultural influences.

The development of trust can bridge differences. People tend to trust the familiar, so leverage the power of ingroup bias. The tendency for people to categorize others into “in-groups,” according to social categorization theory, lends itself to activities that seek out commonalities creating positive bias. When facilitating a learning session, create an environment of inclusiveness where commonalities are identified, and differences are perceived as an asset rather than a liability. This can be accomplished through case studies, if/ then scenarios, and other relational learning.

Learning professionals must always conduct these with the consideration of individualistic and collective learning values.

Being culturally intelligent and considerate requires that learning professionals develop a flexible set of skills, including the ability to listen and pay attention, reflect on the meaning of underlying behavior, seek out relevant information and advice, and adapt resourcefully. It requires being aware of, and willing to challenge, one’s own cultural assumptions. If a learning professional has the focus of aspects in the learning environment being planned and reliable, yet the learners respond more to flexibility and less structure, the professional must consider how the impact may affect the intended outcomes. If the organizational culture is competitive, should the learning opportunities confirm or encourage competitiveness, or should they place higher value on cooperative growth? Only the learning professional who observes interactions, is aware of their own assumptions and values, and has knowledge of the

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

| 41

cultures imbedded in the individuals and the organization will successfully navigate questions such as that.

THE BUSINESS CASE FOR ENGAGING IN MEANINGFUL DIVERSE INTERACTIONS Learning professionals, through onboarding and continuous quality learning offerings, influence not only the culture within an organization, but impact the business development of that organization. When the talent develops the ability to interact with others of diverse cultures, opportunities for personal and professional fulfillment abound. When those who rely on learning professionals, and the learning professionals themselves, have some understanding of different cultures, but also have problem-solving strategies and effective adaptations that work across or within different cultural contexts, business growth is a direct result.

Cultural considerations in the learning environment support multicultural team effectiveness. Multicultural team members’ abilities to interrelate translates to abilities to take the perspectives of a diverse customer base.

Learning professionals must observe others in the learning environment, whether live or virtual.

These abilities and team effectiveness drive profitability and cost-savings.

Success in culturally diverse markets can be linked directly back to engaging successfully in meaningful diverse interactions.

FINAL CULTURAL CONSIDERATIONS o Distinct cultural groups are not merely linguistic, matters of race or about specific ethnicity. o Each individual or perceived cultural group is different, and their unique cultural, psychological and economic personalities demand subtle, tailored responses. o The commitment to integrate diversity and cultural competence into an organization will increasingly pay big dividends in the coming years. o Integrating cultural competence into an organization does not happen by accident.

Even if an organization does not operate on a global scale, the cultural interactions within the working environment can

impact the innovation and development of the talent. As learning professionals come across diverse opinions, it becomes important to understand how and in what ways culture shapes individuals, teams and the organization, and how they in turn shape learning.

Carol E. Brogan has a Master of Science in international administration and is an CQ Certified Advanced professional by the Cultural Intelligence Center, LLC. She currently works with others on their cultural and professional development in the diverse city of Miami, Florida. Email Carol.

Questions for Cultural Considerations • What are the cultural characteristics of the members of the target learners?

• What are the types of technology used by the learners? What is their usage of and access to that technology?

• What gestures, clothing and forms of physical contact have negative connotations for any participating learners?

• If translation of language is required, what timing and logistical issues may need to be dealt with?

• If the learners’ first language is not that of the facilitator or instructor, but translation is not requested, what terminology, words or phrases, must be adapted to accommodate the internal understanding for the learners?

• How explicit do the instructions need to be? How much guidance is needed for clarity for all learners?

• What colloquialisms or idioms are imbedded in the script that may be regional and not easily translated or interpreted?

• Does time for socialization and relationship building need to be factored in to learning events and meetings?

• Do any resources or articles intended for use have aspects of, or suggest an ethnocentric bias toward a particular culture?

| 42

• What role does status play in the culture of the organization or of the values of the individuals involved in the interactions? How does that impact planning and activities?

• Is the type of approach, from very simple didactic to complex constructivist, appropriate to the cultural representation of the learners?

• Are adaptations to accommodate individuals needed or essential to accomplish the goals or objectives?

Or can individuals adapt or ignore the differences?

• Ultimately, are the needs of the learners, the workforce, and the content, methods and media aligned?

Linking Competency Models to Modern Learning Strategies

By Matt Donovan

With the rise of digital disruptions and “next generation� talent management strategies, nearly all HR and learning and development (L&D) teams are being pushed to work in concert together to create seamless employee learning, performance and development experiences. I truly believe that this is a great thing – the real beneficiary here is the employee/learner/performer (whichever term you prefer). However, the collaborative efforts between HR and L&D have brought out some long-standing differences between the two functions.

| 44

One of the pain points across these two similar yet different functions has been around the incorporation of competency frameworks in modern learning experiences. While HR teams have historically seen the value of strong competency models, performance improvement consultants and learning professionals have struggled with seeing how competency models support the design and development of relevant, modern and learner-centric curricula. The result is a passionate debate over how to link research-based

competency maps into the design of authentic and valid performancecentric learning curricula.

Two Views on the Performer Essentially, competency models and learner/performer-centric outcomes are two paths to define and benchmark the same performer population. To better understand the different paths, it is important to grasp the fundamental premise for each. Figure 1 (see page 46) compares features of a competency

IF HR AND L&D TEAMS CAN COLLABORATE, THE LEARNER WILL VALUE FROM A RESEARCH-BASED AND ROLE-RELEVANT LEARNING JOURNEY. model/map and modern learning experiences using an outcomes-based approach. It is important to note that when I refer to competency models, I am referring to those that are anchored in research across a range of organizations and have some degree of statistical analysis behind it.

in defining associated competencies for roles, which include large groups of performers. They tend to be blueprints outlining the desired state for role performance in general terms of required knowledge, skills and abilities. They are designed to support the organization in:

Key Elements of Competency Models

• Creating an objective framework for defining/assessing performance in a role

Competency models and maps are designed to support the organization

• Assessing the readiness of individuals in a population to perform in a role

• Providing general development guidelines for individual improvement A key goal of a standardized competency framework is to establish a common language around what “good looks like” within a generic role or functional context. This common language is often used to reduce the subjectivity for the review, evaluation and development of employees in a role. Refer to sidebar on page 47 for descriptions of key concepts.

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

| 45

Key Elements of a Modern Learning Experience Using Outcomes Modern learning experiences using outcomes are focused with the learner in mind. Learner/performance-centric curricula includes learning interventions/ experiences that enable learners to close specific performance gaps between current state and desired state for role performance. They are designed to support the learner/performer with detailed relevant insights and actions in creating specific outputs, including: • Gaining the knowledge required to perform critical tasks and activities in the role,

• Developing the skills required to perform critical tasks and activities in the role,

a mix of activities that meet a range of needs and the social learning roles.

• Establishing and refining abilities critical to execution of tasks and activities in a role, and

Bringing It All Together: Building the Best Modern Learning Experiences

• Leveraging a system of performance support.

Performance consultants and learning professionals approach role excellence by identifying the critical performance outcomes with the outputs of top performers as the source of the data set.

Then, to build a curriculum that meets the needs of the modern workforce, the learning interventions must include

Both approaches can provide valuable results when it comes to designing modern learning experiences that you can gather value from. While the competency model can help organizations determine the readiness of its employees and identify areas to build skill sets, it does not provide the detailed, specific inputs required for a relevant modern learning experience.

FIGURE 1 COMPETENCY MODEL/MAP

MODERN LEARNING EXPERIENCES USING OUTCOMES

The organization, talent recruitment, individual’s managers and individual learners/performers

The organization, individual’s managers and individual learners/performers

Focus

Competencies

Outcomes in terms of critical performance outputs

Scale

Large-scale application to roles across all functional units in the organization.

Focused application targeting a learner population within a shared role or need.

• Create an objective framework for defining/assessing performance in a role.

• Gain the knowledge required to perform critical tasks and activities in the role.

• Assess the readiness of individuals in a population to perform in a role.

• Develop the skills required to perform critical tasks and activities in the role.

• Diagnose which competencies an organization should focus on and competencies that may be lacking overall in an organization.

• Establish and refine abilities critical to execution of tasks and activities in a role.

• Provide general development guidelines for individual improvement.

• Provide a pervasive learning ecosystem that meets a range of moments of learning need.

Intended Audience

Purpose

Source of Analysis

Key Deliverables

Analysis of large sets of data looking at generalized roles and performers across multiple organizations. Usually does not include variations for different components in the organizational value chain (e.g., manufacturing, sales, research, etc.).

Analysis of the organization’s top “A” performers, determining key performance outputs and performance criteria. This is usually done at the group or function level. Lessons learned can be applied to other functions. Empathetic interviewing of the “B” performers to help shape the journey, including moments of learning need, social learning roles and the performance support network.

• Competency model/map.

• A list of outcomes (performance indicators) that include tasks performed to achieve those outcomes.

• Diagnostic and results for individuals highlighting areas for improvement.

• Diagnostic and results for organizations highlighting key competencies to invest in.

| 46

• Leverage a system of performance support.

• A series of personalized learning experiences that meet a range of needs based on a certain role.

• An avenue to build mobile, social, networked learning experiences for those new to role, gaining proficiency, or growing as an expert within a role.

When creating a learner/performercentric curriculum, start with the top performers and the performer’s outcomes, but always keep sight of the structure of the competency model and identified competency gaps. In the end, a successful curriculum will be evidence-based and built on a foundation of specific, descriptive, learner-centricity and a clear description of what good looks like from the relevant performers.

MODERN LEARNING EXPERIENCES USING OUTCOMES ARE FOCUSED WITH THE LEARNER IN MIND. The following are key best practices to consider: • As a learning professional, take time to learn as much as possible about the competency framework/ model and the diagnostic tools and research that backs it.

• Leverage insights from the competency framework and diagnostics as potential inputs into the learner journey, a reflection on key competencies and potential areas to focus.

• Ensure your modern learning experiences further defines role excellence by focusing on job-relevant accomplishments and create relevant scenarios/performance narratives.

• Throughout the experience, create learner-centric bridges from general competency descriptions to very

specific job accomplishments within a particular role.

• Where applicable, provide guidance for the learners on how to use the competency model in supporting their own development.

• Create/curate additional resources to support learner remediation or stretch goals.

• Provide clear mapping and coaching guidelines for managers to support continuous growth.

• Provide practice and feedback systems that deliver learners with targeted insight into their own development.

Conclusion Competency models play a critical role in supporting the design of modern learning experiences but should not be the only framework in guiding the design of the learning solution. Because competency maps lack detail about performance outcomes and tasks for a particular role, they can miss the mark for developing relevant learning experiences. A performance-centric focus on outcomes is necessary to meet specific learner needs.

The key is realizing that competency maps can add value to augment the modern learning experience, and an analysis around specific targeted performance is critical to create the most relevant modern learning experience.

If HR and L&D teams can collaborate together, rather than argue, the learner will value from a research-based and role-relevant learning journey that is designed to create measurable business results.

KEY CONCEPTS • COMPETENCE: Ability of an individual to perform his or her role in a proper manner or being adequately qualified to perform a specific role – typically anchors to the basic requirements.

• COMPETENCY: Description of the requisite knowledge, skills and abilities to effectively perform a job role – typically anchors to performance excellence.

• COMPETENCY MAP/MODEL: An off-theshelf model often based on an analysis of quantitative, qualitative and market-based data across organizations and industries.

Many frameworks are modular in nature and are comprised of 35-50 competencies, which can be grouped and/or clustered.

• COMPETENCY DIAGNOSTIC: An associated employee diagnostic administered to employees in the covered roles/functions. In more rigorous competency maps/frameworks, they can identify the competencies most linked to success for the role/function.

• PERFORMANCE/OUTCOME: The accomplishment of a certain task measured against known standards of completeness (e.g., accuracy, cost or speed).

• PERFORMANCE OUTPUT: The observable deliverables or asset(s) created during the accomplishment of a certain task.

• SECRET SAUCE: Very specific actions or tasks that a top performer does to achieve success. This may or may not be consciously understood by the top performer, but is critical.

• LEARNER JOURNEY: An experience map that begins with the learner/performer’s perspective, focuses on key performance points, incorporates a range of moments of learning need, and a range of social learning roles to meet the needs.

Matt Donovan is the vice president of GP Strategies Learning Solutions Group, where he leads global learning design, development and delivery organization for Fortune 500 companies. Email Matt.

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

| 47

DEVELOPING TRAINING WITH DIVERSITY AND INCLUSION IN MIND GREAT BRITAIN

SOUTH KOREA

BRAZIL

A colleague designed a virtual course that checked all the boxes: length was great, it was engaging, interactive and it was informative. Perhaps you even learned a thing or two. The one thing it was missing was a diverse set of characters and experiences. No one looked like you, behaved like you or like many of your peers. The main character was a white male. No women, no people of color, no differently abled individuals, or underprivileged persons. Not only was he a white male, you guessed it, he was also the leader. Now, there is nothing wrong with this per se, however, once the training was concluded, you felt slightly disconnected from what you experienced. You felt as if your experiences were not taken into consideration. You discuss this with your colleague. They nod in agreement, but you are not sure they fully understand your concerns.

ION NAVAJO NAT

INDIA

NY

PARIS

PHILLIPPINES

GERMA

ICELAND

IRAN

BY TIFFANY MCLEAN | 48

Oprah Winfrey often talks about what she refers to as “teachable moments” – moments when something bad occurs and you can turn that bad situation into something positive.

Basically, the lesson in the message. Diversity and inclusion (D&I) is one of those teachable moments. As learning and development (L&D) professionals, we have probably had an experience one time or another that is similar to the below scenario.

MEXICO

Incorporating diversity and inclusion is perhaps one of the most important roles of an L&D professional. The learning experience is varied and should be represented as such in training programs. We so often focus on developing programs that offer a consistent message, language, look and feel that we often forget the most important aspect of all design: the people.

WHAT IS DIVERSITY & INCLUSION? Diversity and inclusion has different meanings to different people, but let’s all work with this definition for the purposes of this discussion. Bersin defines diversity and inclusion as “the variety of people and ideas within a company, and the creation of an environment in which people feel involved, respected, valued, connected and able to bring their “authentic” selves (e.g., their ideas, backgrounds, and perspectives) to the team and to the business.”

Note that Bersin’s definition did not mention race or gender. This is vitally important as oftentimes that is exactly what people tend to think of regarding diversity and inclusion. Organizations want to make sure they can mark the checkbox. Woman? Check. Asian? Check.

LGBTQ? Check. However, it is so much more. Diversity and inclusion, especially in the L&D space, is about ensuring that there are “checks and balances” in what we do.

It is imperative that L&D practitioners consider multiple factors when preparing for a course. For example,

when creating a course, ensure that you truly keep your audience in mind.

Consider factors such as course content, any activities that may be a part of the course as well as awareness of their background and skills. Taking time to consider these items will help to ensure that an inclusive environment has been created and your audience will be better able to focus on the material as well as provide value to the organization.

Just as there are multiple branches of government to ensure the balance of democracy, there needs to be for the sake of organizational stability “checks and balances” to the designing of learning programs. Learning has never been a one-size-fits-all affair and the incorporation of diversity and inclusion is no exception.

as measurement. Fresh perspectives are needed for all aspects of learning, including just-in-time solutions, creative and innovative delivery models, and unique modes of assessment.

L&D professionals should ensure that diversity and inclusion is reflected in our learning programs. As such, diversity and inclusion needs to be a part of all learning activities as it is an essential part of any workplace and critical to fostering a diverse and inclusive organizational life. How, where and when it occurs is dependent on identifying both formal and informal solutions for planning, access, execution, reinforcement, as well

Now that we have some idea of what it looks like, let’s think about what diversity and inclusion feels like. Successful programs implemented by L&D should introduce and encourage new ideas, increase participants’ awareness, and offer valuable skills and suggestions to develop and enrich an employee’s implicit biases. It should also provide an inclusive space and opportunity to engage with these issues and grapple with their complexities. It should feel open and welcoming, a place equally as dedicated to exploration and support as it is to L&D.

DIVERSITY AND INCLUSION IS MORE THAN A CHECKBOX Diversity and inclusion encompasses more than race and gender. D&I addresses the variety of people and ideas within a company, and the creation of an environment in which people feel involved, respected, valued, connected and able to bring their ideas, backgrounds and perspectives to the team and business.

To build a more inclusive culture, L&D professionals need to keep their audience in mind when developing a training course. Consider factors such as course content, activities, as well as the background and skill set of the learners. Training courses should reflect the organization’s people, culture and values. Taking time to consider these items will help to create a more inclusive environment.

WE OFTEN FORGET THE MOST IMPORTANT ASPECT OF ALL DESIGN: THE PEOPLE.

INCORPORATING D&I INTO TRAINING So how does a modern learning leader ensure that diversity and inclusion is not an afterthought in the work we do? Here are a few things to keep in mind.

Develop a strategy. Work with your organization to commit to creating an inclusion strategy. This will help to elevate L&D’s seat at the table as

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

| 49

this shows a willingness to partner and consult. This provides the perfect opportunity for learning leaders to help guide the strategy and direction of the organization. This partnership also helps to ensure that diversity and inclusion are tied to learning objectives as well as the financials for your organization.

LEARNING HAS NEVER BEEN A ONE-SIZE-FITS-ALL AFFAIR AND THE INCORPORATION OF D&I IS NO EXCEPTION. Review your onboarding experience.

The usual onboarding focuses on corporate culture and values. How do you highlight that to new employees?

Consider ways to include diversity and inclusion thinking regarding how onboarding is presented.

• Can all employees regardless of socioeconomic status access the information? This is especially important for virtual onboarding experiences.

Create well-rounded leadership development programs. Developing leaders is a high priority for organizations, as many studies have shown. This is an area where most L&D teams excel. When creating these programs perhaps use a different lens, not just based on race and gender, but focus on other areas of acquired diversity. Keep in mind, for example, educational background, social status and socioeconomic status, amongst other things. This will help to ensure a well-rounded program that will cater to all participants.

Look for ways to address diversity and inclusion. Include diversity and inclusion as part of employee engagement and organizational health surveys. This is where partnering with the executive team and other departments can really benefit L&D. Track the talent pipeline. Assess the success of your leadership development programs by tracking the diversity of the talent pipeline over time. Note a specific increase in the number of minorities promoted for example.

Review turnover numbers. Work with your talent acquisition teams and review turnover numbers. Are there particular groups of employees who are leaving the organization? If the answer is yes, then review your programs and make appropriate changes.

• Visually, do the images capture the true makeup of your organization?

OUTCOMES

• Host an employee panel where diverse populations can speak to how they learned to navigate the company culture.

Now let’s take one last look at our colleague’s course from the beginning of the article with updates that incorporate diversity and inclusion. The length was great, it was engaging, it was interactive

| 50

and it was informative. You learned even more this time. You congratulate your colleague on a job well done. Then you stop. You reflect on this new course and realize that although this course was the same, it looked different, it felt different. You then realize why. The course was offered in different modalities, online and with an instructor-led training component. It had multiple generations of people represented, the leader was a Muslim woman who partnered with the white male in the original version, and they were peers. Wait … did you recall seeing your company’s diversity and inclusion mission statement in the background of one of the scenes?

This course ultimately becomes the showcase for how to implement a diversity and inclusion program into L&D at your organization.

The results are impressive. Your organization sees an increase in morale, in open dialogue and the financials begin to improve. Diversity and inclusion has gone from a teachable moment to a moment where the true business value is realized and lives are positively impacted. Your seat at the table has been established as you saw a problem, proposed a solution and ultimately tied learning objectives to the overall business strategy, which in turn, positively impacted the bottom line. As L&D professionals, isn’t that truly the goal? As we continue to focus on developing new programs, new initiatives, do not forget the most important part: the people.

Tiffany McLean is the learning and development manager at General RV Center, where she develops learning strategy for more than 1,500 employees in six states and 12 locations. Email Tiffany.

GLOBAL TRAINING

TAILORED LEARNING. MEASURABLE RESULTS. High-consequence environments require the right decision – the first time, every time. Raytheon Professional Services’ tailored learning solutions align with your critical goals to produce dramatically improved outcomes and measurable results for your workforce and organization.

rps.com @RaytheonRPS

SPEED TO PROFICIENCY

Raytheon Professional Services

BOTTOM-LINE PERFORMANCE

© 2017 Raytheon Company. All rights reserved.

CASEBOOK

ELEVATING

NEW LEADERSHIP

BY CHRISTI MOOT

Building strong leadership teams can make or break a company. Investing in managers to help them inspire action, instill company values, increase retention, and ensure every employee is fired up to come to work can generate exponential value for the business. This investment can create a multiplier effect – a powerful cornerstone that is essential for an organization to be successful.

With over 500 global employees, AdRoll strives to build the most powerful marketing platform through performance, usability and openness.

The company’s core values are embodied by its spirit animals (see Figure 1 on page 53). The owl resonates most with the values of the training and development team, as it represents the value of “Hire great people and help them grow.”

Growing our people, keeping them on an Everest-steep learning curve at all times, making sure they feel challenged, motivated, pushed out of their comfort zone, and they have resources to help them learn and grow – that is the crux of our job in training. It’s a mission that is backed by our company values, and that comes through in every project and initiative we work on.

THE PROBLEM Over the last several years, AdRoll has offered strong people development for the greater organization, and leadership development programs for its managers.

However, we noticed a gap in our offerings in early 2017: While we serve

| 52

our current managers well, we’ve lacked a curated, purposeful path for new managers who get promoted or join the company. Our initial solution for newly promoted managers was to enroll them into any upcoming leadership trainings; however, having them join programs mid-cohort felt disjointed and ineffective.

We needed a direct path for new managers – a cohort program where new managers could build strong relationships with leadership across departments. Where we could assess skills, provide foundational new management training, and build from there.

This is what we set out to solve for in Q2 of 2017. The project team consisted of two representatives from the people team and two representatives from the training team. We wanted to build a program that was deployed twice a year, after each promotion cycle. We also needed this to be a global solution that would work for all regions. Our goal was to see a clear improvement in manager skill sets after they completed the program.

THE ANALYSIS Research that went into building the new manager training paths was extremely comprehensive. We gathered intel in the form of qualitative assessments from HR Business Partners (HRBPs), who have had ongoing dialogue with every manager for several years, and we looked for any themes that emerged from the conversations.

The project team interviewed seasoned (and newer) managers on what they’d like to develop within themselves.

Themes of strategic decision making, motivating teams to perform, and retaining top talent were top findings from this research.

We went deep into industry benchmarking and research. What are similar companies doing? What’s working/not working at other companies our size, with similar cultures? Having a resource group for this type of datageneration really helps. I have a San Francisco-based monthly meet-up with peers across quite a few tech companies that is always a valuable use of our time. We learned that we weren’t alone in this gap; not many companies have established new manager ramp programs that are separate from ongoing leadership development.

Lastly, we wanted full executive buy-in for the new program, and to get their point of view on how new managers can best drive value for AdRoll. We presented a strawman proposal to our exec team, and they were fully behind the idea. They shared some of our company’s top initiatives for 2017 so that we could build a program that would help new managers drive results around these top goals.

Doing robust and varied research helped give us a strong baseline before diving into program creation. We huddled in a room (and over video conference),

made sure we had plenty of whiteboard space, and set to work transforming all the inputs and feedback we’d collected into a meaningful program for new managers.

THE SOLUTION Here’s what we came up with: • Timing: A three-month program, run twice each year was chosen to ensure an expedited ramp (one quarter), but would also give managers time to form relationships in their cohort, absorb material in a meaningful way, apply new skills and come back with challenges/feedback.

• Type of training: Blended learning would occur through articles, e-learning (built internally and through Udemy), webinars, live in-person sessions, book clubs, roundtables and exec panels.

managers to use to discuss trainings, things they’ve tried, what’s worked/ what hasn’t worked, ask questions and crowdsource ideas. This gave us space to post new articles, blogs, book recommendations, podcasts, TedTalks, and any additional relevant content.

The content itself is the meat of the program, so it was critical we nailed that piece.

The paths were input into our LMS (AbsorbLMS) so that new managers had easy, clear access to their learning path as soon as they joined AdRoll as a manager, or got promoted.

We formally rolled out the new manager learning path in July 2017, so that all new managers promoted in that cycle could partake. We also

• ROI: We’d measure success of the program through survey feedback (Did managers like it?), self-assessments built in throughout the path (Do managers feel like they are getting better at management skills?), and team/peer feedback (Do manager’s reports agree with manager self-assessments? Do HRBPs see measured improvements?).

BUILDING STRONG LEADERSHIP TEAMS CAN MAKE OR BREAK A COMPANY.

We also wanted to create a forum for ongoing dialogue between the new managers in the cohort. AdRoll uses Slack, a messaging platform that allows you to build customized and specific channels for different purposes, as a primary means of communication.

We built a Slack channel for new

retro-enrolled any new managers from January to July of that year, so we had a big first class. During pre-rollout, we went big on communication. We made announcements to the entire company via email, Slack posts and posters

FIGURE 1

around the office, so that people were aware that this was a resource for new managers, and an investment AdRoll was making in future leadership. We made sure our recruiting team had full insight into the program, so that they could use this as a value prop when recruiting new leadership talent into our company.

THE OUTCOME After several months of heavy lifting, it was extremely exciting to launch in July. Rolling out this new manager learning path was a huge success for the organization. It demonstrated that we were investing in our people.

It provided directive, relevant, hightouch learning to a group that can make a huge impact on our company. It established relationships that lasted far beyond the three-month training period.

Feedback was and continues to be wildly positive. The training and people teams continue to learn and iterate to ensure we are offering a best-in-class learning experience for every new manager at AdRoll. We’re proud of what we built to fill a gap, and are excited to keep elevating this path as we learn more.

Christi Moot is the head of training at AdRoll, where she creates globally scalable organizational training strategies across sales training, people development, leadership development, new hire onboarding, and LMS/vendor-management.

Email Christi.

OUR SPIRIT ANIMALS EMBODY OUR CORE VALUES

Owl

Jellyfish

Bee

Terrier

Monkey

Beaver

Hire great people and help each other grow

Be open and authentic

Build innovative products that work for lots of people

Do right by the customer and community

Take work seriously but not ourselves

Do more with less

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

| 53

MEASURING I M P A C T

GAUGING THE SUCCESS OF

SUBSCRIPTION-BASED EDUCATION

BY MILIND GURJAR

Professional survival depends on education in today’s world of rapidly accelerating transformation. At the same time, for organizations to stay competitive, they must have a way to measure the success of that education.

Instructional training is typically delivered as a transactional service, limited to occasions when individuals need to acquire new skills. But the pace of change today, especially in technical disciplines, has become so rapid that employees now need continuous education. This need for ongoing education has driven the growth of subscription-based training.

THREE EDUCATIONAL COMPONENTS BASED ON LEARNER NEEDS The first component is technology adoption, on which a company decides to deploy some exciting new technology. The IT team must have the right skills before they can start. So, they enroll in a class to learn new skills.

| 54

When the skills are acquired, the need for training is fulfilled.

The second component involves ongoing management. For centuries, employees learned trades through apprenticeships. When their training was complete, they had all the knowledge needed to practice their trade for the rest of their professional life.

However, the rapid pace of technological change renders this system obsolete.

Software and hardware are updated, security patches and protocols are implemented, and systems are expanded and integrated. It’s easy to imagine an IT professional needing to update their skills with information that has changed since their initial training.

In the midst of doing their job, they need an occasional lifeline to help them stay productive.

The third component represents another product of our rapidly

changing work environment: career development. To stay competitive, companies need employees’ skills to be up to date. Individuals work to keep their skills current to reduce career risk.

Others see new skill acquisition as a path to career advancement.

The three components align with the different models for education delivery.

For the most part, technology adoption aligns with the transactional model. A defined set of skills is learned in order to complete a project. The other two components of education, ongoing management and career development, require continuous learning, with constant access to skills, information and knowledge. By nature, these are better fitted to a subscription model.

MEASURING SUCCESS OF EDUCATION SUBSCRIPTIONS Educational theorists have suggested that training can be evaluated on five levels: • Reaction: how learners feel about the learning experience or training.

follow up with a learner six months later to measure real-world training effects – especially in a subscription model where the timing of training may be different for each individual.

Measuring outcomes is also useful. One outcome to measure in IT training is increased employee productivity. The second is improved uptime. It makes sense that there should be a correlation between training and productivity.

Increased network uptime might seem like a simple measure, but it’s not always easy to establish causality. If there is an improvement in uptime, was it because of training? Or was an expert hired? Or was it because a vendor fixed a critical bug?

AT THE END, ONLY THE LEARNER KNOWS WHETHER THE TRAINING HAS MET THEIR NEED.

• Learning: how the learner’s knowledge has increased after training.

• Application: how much the learner is applying what they learned on the job.

• Outcomes: the effect of the learner’s new skills on the business environment.

• ROI: the financial return on the training investment.

The easiest level to measure is surely the first one, reaction. It’s simple to ask, “How effective did you think this training was? How useful was the course material?” This provides a reasonably useful yardstick on training effectiveness. The other four levels are more difficult to measure. It’s hard to

However, it is likely that the best method for measuring educational success is to return to the three components of education and evaluate training with the needs of the learner and of the company in mind. For technology adoption, two measurement approaches stand out. The first is time to value. If a trained team can complete a project in four months, while an untrained team takes six to eight, this is a clear demonstration of value. Another approach is to compare error rates on new projects between trained and untrained individuals. What does it cost to find errors and fix them later?

How much money can the company

save by reducing errors? This may be difficult to measure with precision, but no one would argue that training doesn’t reduce mistakes.

LOOKING AT THE NEEDS OF INDIVIDUALS As for the ongoing management and career development components, it’s important to consider the specific needs of the individual receiving the training. Subscription-based education is typically much more focused on the individual than instructional learning.

Individuals frequently make their own decisions about what they want to learn. Often, they proceed at their own pace. At the end, only the learner knows whether the training has met their need.

The individual must be the primary focus of effective evaluation of subscription-based learning. It must answer the questions: What were the individual’s initial goals and expectations? Has the training helped them meet these goals? What elements of the training were most effective and least effective in meeting these goals? Ultimately, how has the training helped the individual on their journey to success?

What makes subscription-based learning so timely and valuable is that it provides learning content tailored to the individual learner’s needs in an on-demand format. Consequently, measuring this type of training’s effectiveness cannot be reduced to a one-size-fits-all assessment. Instead, it comes down to gauging effectiveness at the individual level.

As the senior director and general manager for Learning@Cisco, Milind Gurjar is responsible for training Cisco’s top-tier customers, channel partners and employees worldwide. Email Milind.

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

| 55

Recycled & Reused

Repurposing Training Content for Digital Delivery.

Read the Report

SECRETS OF SOURCING DOUG HARWARD

ADAPTABILITY

Much has been written about the values and advantages of adaptive learning and testing. Both use computer-aided technology to allocate and deliver content based on learners’ needs. And both are sound strategies for delivering content and evaluating a learner’s abilities. These approaches to adaptability are based on the notion that adaptability is a function of technology – meaning that through machine learning, we can leverage information to adapt to the differences in learner styles and abilities.

I would suggest we think of adaptability in an alternative way as well. Let’s consider how an individual can and should adapt themselves to changing conditions in their workplace, and how they adapt the use of information while learning so that it becomes more personal, or more relevant to them.

According to research by Dr. Anders Ericsson, a psychology professor and neuroscientist at Florida State University, adaptability is a skill set found in all of us.

He argues that all humans have the gift to adapt the human brain and body based on their experiences and changing conditions in their environment. His research has found that individuals who demonstrate extraordinary performance in their chosen profession also demonstrate the ability to be adaptive, to learn from experiences, good and bad, and get better through deliberate efforts.

We all have the ability to adapt, but some learn how to be more adaptive than others and can achieve much higher performance. In other words, adaptability

is a skill set that can be learned and developed. We learn to be adaptive by learning how to process training or the consumption of formal instruction and day-to-day experiences. The more deliberate and purposeful we are about how we turn this information into behavior change can determine how successful we can be in our profession.

ADAPTABILITY IS A SKILL SET THAT CAN BE LEARNED AND DEVELOPED. In his book, “Peak: Secrets from the New Science of Expertise,” Dr. Ericsson shows that individuals who best leverage their ability to adapt by being purposeful and deliberate in getting better at what they do, consistently improve at levels greater and faster than those who haphazardly go through the motions of repetition, doing tasks over and over with no clear approach to getting better.

As simple as this sounds, the concept of deliberate practice has huge implications for learning and development professionals.

Corporate training organizations have traditionally been guilty of assuming that their responsibility towards improving an individual’s performance happened within the classroom or structured learning environment. And assuming how the individual applied skills or knowledge from the training course on the job was the responsibility of the learner.

What we can take from this research is that the more deliberate we as training

professionals are at preparing the learner to adapt in workplace conditions, and how we shift our focus from being a course provider to a performance support organization can and will have huge implications to the value we provide to the business.

THREE LESSONS From where I sit, there are three very simple lessons we should consider adding to job training in our organizations. First, let’s become deliberate about how we teach adaptability. Adaptability is a skill set that deals with knowing how to assess and diagnose on-the-job situations that may be out of the norm, how to determine alternatives that may be suitable solutions, and how to be deliberate about improving through repetition.

Secondly, we must consider how we design and formalize on-the-job training.

We all recognize that employees learn more on the job than in the classroom, so let’s assume the responsibility of teaching workers how to get better on the job. Consider on-the-job training as a formal experience.

Lastly, consider how you can teach the process of deliberate practice. The more an individual learns to be deliberate, and how we personalize our own learning and performance improvement, can not only be a game changer for us on the job, but a life-changing experience as well.

Doug Harward is CEO of Training Industry, Inc. and a former learning leader in the hightech industry. Email Doug.

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

| 57

JUNE 26-28, 2018 // RALEIGH, NC

#TICE2018

REGISTER TODAY! EARLY BIRD DISCOUNT ENDS MARCH 2018.

THE RIGHT SIZE. THE RIGHT FOCUS. FOR THE LEARNING LEADER. 3

DAYS

4

KEYNOTES

400+ LEARNING

PROFESSIONALS

40+

WORKSHOPS

PERSPECTIVES, COLLABORATION AND SOLUTIONS TO MAKE YOU A MORE EFFECTIVE LEADER OF LEARNING AND DEVELOPMENT. Register at www.TICE2018.com.

LEARNER MINDSET MICHELLE EGGLESTON

LEARNING IS A PERSONAL EXPERIENCE

Being selfish is typically frowned upon in most areas of our lives. The adage, “There’s no I in team” comes to mind when an individual is ridiculed for exhibiting a self-centered attitude. But in training, these rules don’t exactly apply. In fact, there are two I’s in training.

When you think about it, learning is a personal experience. It’s about self-exploration, gaining knowledge, building confidence and expanding skill sets. Regardless of how it occurs or whether it’s for personal or professional growth, learning is about the individual.

LEARNING IS ABOUT THE INDIVIDUAL. In corporate training, we must realize that each employee is unique, and these differences will influence outcomes. As an example, consider one department at your organization where multiple people perform the same job. Now, think about the backgrounds of these individuals (e.g., education, experience, age, etc.).

Then, consider their weaknesses and strengths. And finally, consider their interests and long-term goals.

While these individuals may perform the same job, they are dynamically different people who will progress at different speeds and interpret information differently. In this context, we can see why a one-size-fits-all training program would fail to meet the needs of all the employees.

THE RISE OF PERSONALIZED LEARNING Technology is transforming our lives – both inside and outside of the workplace. Today’s learners have grown accustom to personalized experiences in their day-to-day lives. From consumer websites recommending products based on previous searches to GPS tracking offering a recommendation based on your specific location, it is only natural for employees to expect that same level of customization in training.

Modern learners want to be at the center of the learning experience. They are concerned with the specific skills they need to perform their job and focused on the speed in which they can achieve growth. Before sinking their time into countless hours of training, learners want to understand what’s in it for them and how their role is making an impact in other areas of the business.

Personalized learning is engaging and impactful because it provides the relevancy learners want and that organizations need to meet business goals. For example, adaptive learning technologies can monitor the progress of employees, providing learning and development (L&D) with data and insight into their performance.

The technology uses a questionbased approach to check learner comprehension and enables the learner to place-out of content they may have already mastered. This ensures the learner is not wasting their time on content they already know and

allows them to focus challenging information.

on

more

COMING TOGETHER The learner is just one side of the equation. The priorities of the business must also be taken into consideration when developing corporate training initiatives. When you get down to it, high-performing organizations develop training programs that reflect the needs of the business. L&D must translate the organization’s mission, values, goals and objectives into training programs that will maximize the performance of its employees to deliver on the intended outcomes.

While the learner and the business operate with individual goals in mind, L&D must serve as a “liaison” advocating for the needs and expectations of both audiences. Training is the catalyst to driving organizational performance. It could be argued that L&D should view those two I’s in training as representative of the learner and the business. One is irrelevant without the other.

Without strategically aligned training programs, organizations will likely fall short in preparing their employees to meet their goals. By acknowledging that learning is a personal experience, L&D can bring relevancy to training that will benefit the learner as well as the business.

Michelle Eggleston is the editorial director at Training Industry, Inc. Email Michelle.

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

| 59

ARE YOU BEING HEARD?

Ever feel like senior management isn’t really hearing you?

Start by speaking their language. Earn your Certified Professional in Training Management (CPTM™) credential and build alignment between L&D and business goals.

Learn more at cptm.trainingindustry.com.

WHAT’S NEXT IN TECH ERIC SHARP

3

CONSIDERATIONS WHEN PERSONALIZING LEARNING

The past decade has rapidly increased our access to personalized experiences in every part of our lives. Heading to lunch? An app of your choosing will be there to recommend restaurants based on your past experience. Leaving your house for the day? A prominent map app will make an educated (and most likely correct) guess at where you’re heading based on the day of the week, time of day, etc.

As our use of technology becomes more and more personalized, we should be clear in assessing the

WITHOUT A “BIG WHY,” ENGAGEMENT WILL BE SHALLOW. potential of personalization in learning.

It is important to not overlook the context and overarching end goal of learning personalization, aligning our employees, our initiatives and our technology accordingly.

THE BIG WHY At our company, we often talk about our “big why,” the motivating and inspirational reason behind our mission and vision. The “big why” allows all of us to passionately take part in driving forward the mission and vision we share.

Some of the examples of personalization I mentioned above, and others in

areas like music, entertainment and tourism are useful, but are missing a “big why.” Without a “big why,” interest and engagement will be short term and shallow.

Fortunately, in learning, the “big why” is consistent and easy to call out. Every employee is interested in their career and the development of their skills. We need to emphasize the “big why” when we provide personalized learning so that employees are always reminded of their long-term goal. We believe that each employee should have a skill plan that contains personalized learning, giving them direction and bringing them closer to their career goals.

CONTEXT MATTERS The personalized skill plan is great at guiding and directing employees to progress in their career goals. However, if a skill plan is something I, as an employee, have to explicitly visit on my own, it will have limited traction. In other words, what good is personalization if the user never sees it? Personalized learning should be surfaced to the employee in an efficient and low-energy way.

In addition, our team has seen a higher level of activity on learning items that contain the context as to why the user is seeing it. For example, I am much more likely to watch a video if I know it was also watched or recommended by a coworker. The context makes all of the difference!

TECHNOLOGY Everyone is talking about putting the employee at the center of the learning universe, but how do you put that into practice? Now that we all agree that personalization and adaptive learning are key to putting the learner at the center, how do we make that happen? I would suggest an examination of your current technology is the best place to start.

Technology can have a dramatic effect on your ability to personalize your learning landscape. Machine learning can be leveraged to draw correlations from similar employees across your business and surface the personalized learning to them. Technology capable of analyzing the complexities and variations of employee learning and presenting a personalized, engaging learning experience is transformational in an organization. Do your due diligence on the technology you use; buzz words are not enough!

I am pleasantly surprised at the level of engagement our employees have when you place a personalized skill plan in front of them (reiterating their “big why”) and then create frictionless ways for them to find the context of learning presented to them. Bring personalization to the forefront of your efforts and you will see increased employee satisfaction and employee engagement, which can transform your company culture.

Eric Sharp is the co-founder and chief technology officer at Degreed. Email Eric.

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

| 61

CLOSING D E A L S

MANAGEMENT COACHING PLATFORM USES AI TO ENGAGE TEAMS AND DEVELOP LEADERS

BY TARYN OESCH

Simon Rakosi, David Mendlewicz and Marcus Perezi-Tormos saw a problem: New leaders, promoted early in their careers, lacked the training in leadership and soft skills to prepare them to manage teams and people effectively.

“We saw leadership training as a huge white space in the market,” says Rakosi.

That’s why they created Butterfly, a management coaching platform that uses artificial intelligence to collect real-time feedback from employees.

Managers use that feedback, according to Rakosi, the COO, “to gain insight on factors that can enable them to make

NO GENERATION OF EMPLOYEES FEELS THEY GET WHAT THEY NEED FROM THEIR LEADERS. better people decisions, have a bird’s eye view of the mood of their organization, and show employees that happiness is a top priority.” The feedback, which is anonymous, is collected weekly or biweekly through a pulse survey based on engagement factors identified by advisors from The University of Oxford and Columbia University. Alex, the platform’s artificial intelligence (AI) leadership coach, “gets to know managers over time and provides custom and ongoing leadership training via digestible content.”

| 62

Butterfly, which was named a top-five messaging app by the Webby Awards last year, announced a $ 2.4M seed round in October, led by venture capital firm Daphni, to “evolve Butterfly’s platform and expand the team to support increased client and product demand” based on current partners, who Rakosi says “represent Fortune 500 organizations across various industries and regions.”

ADAPTIVE LEADERSHIP COACHING FOR MILLENNIAL MANAGERS “Education is really only powerful to the extent that it’s relevant,” says Rakosi.

Effective manager coaching should be relevant, personalized, iterative and ongoing. AI helps organizations meet these coaching needs at a scale previously impossible.

Artificial intelligence is the ability of a computer or robot to perform tasks like identifying patterns in data, speech recognition and decision-making, which previously needed a human to complete. It’s used in adaptive learning platforms to personalize content based on individual learner data, such as team member feedback or answers to assessment questions.

Butterfly’s platform is especially targeted at millennial managers, who are entering leadership ranks at increasing rates as baby boomers retire. Those millennials are used to real-time communication like texting and instant messaging, and looking to use those tools in the workplace. They also value immediate feedback and are very coachable – but

they want their coaching to be based on data. However, according to a 2012 Harvard Business Review article, the average manager becomes a supervisor around age 30 but doesn’t receive leadership training until age 42.

A 2016 study by PDT and Leading Dimensions Consulting found that millennials tend to prefer a coaching style in their own leadership. They enjoy collaborative cultures and succeed at gaining consensus among stakeholders and, according to PDT managing director Paul Findlay, “may cultivate a more transparent culture with leaders who are more inclined to seek feedback and buy-in.” However, Findlay also says, “Millennial leaders may risk settling into a ‘one-size-fits-all’ approach to leadership” due to their ability to adapt to the fast pace of business. Using a platform like Butterfly may help by providing instant feedback and teaching leaders how to make adjustments based on that feedback.

Ultimately, a recent Life Meets Work study found, no generation of employees feels they get what they need from their leaders. That includes millennials but also Generation Z, Generation X and baby boomers. It’s clear that we all need better managers who understand their employees and know how to lead them well. Perhaps by using new technologies like artificial intelligence to provide feedback and adapt learning, we can enter 2018 with leaders who are better prepared to guide employees and organizations to a successful future.

Tary n Oesch is an editor at Training Industry, Inc. Email Taryn.

C O M PA N Y N E W S

ACQ UI S I T I O N S A N D PA RTN E R SHIPS VitalSource, the leading platform for institutions, campus stores, publishers, and companies to create and deliver educational and training content, has acquired the assets of Seattlebased corporate learning-technology leader, Intrepid Learning. Intrepid Learning’s and VitalSource’s corporate and professional business platforms will launch under the brand Intrepid

by VitalSource™.

Instructure, a leading software-as-aservice (SaaS) technology company in education and learning and performance management, has acquired Practice, an applied video microlearning solution used to sustain continuous learning cultures and drive innovation.

HighGround, a transformational employee engagement platform provider, has acquired Yoi Corporation, a leader in predictive situational coaching. The acquisition places HighGround at the forefront of continuous performance management, arming customers with a new approach for enhancing employee coaching, learning and performance.

Corel Corporation has acquired ClearSlide, the sales engagement platform leader that enables sales, marketing and service teams to drive more successful customer interactions.

With this acquisition, Corel will invest in the development of ClearSlide’s cloudbased platform, fueling its technology leadership and enabling ClearSlide to grow its business in new markets.

Kaplan Professional Education (Kaplan Professional), a unit of global education provider Kaplan, Inc., signed an agreement to acquire the College for Financial Planning (CFFP) from Apollo Education Group, enhancing its leadership role in educating and preparing professionals in the financial services industry globally by expanding its already superb selection of offerings.

Callidus Software, a global leader in cloud-based sales, marketing, learning and customer experience solutions, announced that it has acquired Learning Seat, a cutting-edge provider of adaptive training and compliance learning content. The acquisition enhances CallidusCloud Litmos Learning Platform by adding nearly 500 courses.

INDUSTRY NE WS INDEX TO QUANTIFY AND BENCHMARK SALES READINESS

MindTickle, the company with a comprehensive enablement platform and a trusted by global sales leaders to ensure always-ready sales teams, launched the Sales Capability Index (SCI), a capability score that, for the first time in the industry, provides a holistic, quantified assessment of sales reps and team readiness, while also producing a leading indicator of their expected performance.

LEARNING SCIENCE RESEARCH

Fuse Universal, market leading nextgeneration learning solutions provider, will collaborate with the computer science department at University College London (UCL) to explore machine learning driven personalized

learning experiences. The project’s aim is to look into human learning trends, to drive the development of novel e-learning software systems that provide a learning environment and corporate training within organizations.

NEW SELF-REVIEW EVALUATION

15Five, the performance management platform, announced a new quarterly performance review feature, the BestSelf Review, designed to replace annual employee evaluation. This researchbacked performance management software facilitates quarterly or biannual performance check-ins where employees and their managers reflect on progress over the previous quarter, and work together to map each employee’s career trajectory and development path.

VIRTUAL REALITY MICROSIMULATION EXPERIENCE

Aquinas, the premiere learning reinforcement software company, unveiled a cutting edge virtual reality simulation product. Dubbed “Microsimulation Engine,” the product was recently previewed at the NYVR Expo, where Aquinas co-led the creation of the conference and advised on the overall production.

INTERESTED IN SUBMITTING COMPANY NEWS? PLEASE SEND TO EDITOR@TRAININGINDUSTRY.COM

T R A I N I N G I N DUSTR Y MA GAZ INE - ADAPTING LEARNING 20 1 8 I WWW. T RAI NINGINDU S T RY . C OM/ MAGAZ I NE

| 63