Current Practices in 360 Degree Feedback: A Benchmark Study of North American Companies, 4th Edition

째 360

Current Practices in 360 Degree Feedback

A Benchmark Study of North American Companies

In referencing this study please use the following reference: 3D Group (2013). Current Practices in 360 Degree Feedback: A Benchmark Study of North American Companies.

Emeryville, CA: 3D Group.

Copyright Š 2013 by Data Driven Decisions, Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without prior written permission of the publisher.

Printed in the United States of America.

ISBN-13: 978-0615748351 (3D Group)

ISBN-10: 061574835X

Par t i c ip a t i n g Org a n i z a t io n s

by I ndust r y

Financial

Local & State Government

Pharmaceutical

Credit Union of Texas

Brevard County Government

Abbott Laboratories

Davis Selected Advisers

City of Lafayette

Portola Pharmaceuticals, Inc.

Mortgage Industry Advisory Corporation

State of Tennessee, Department of Environment and Conservation

Real Estate

Raymond James Financial Advertising & Marketing

Classified Ventures, LLC North American Membership Group, Inc. PR Newswire Association, LLC

REW Insurance & Financial Services, LLC Ronald Blue & Co. Ryan Anonymous Food & Beverage

Aerospace & Aviation Services

MillerCoors

Air Serv Corporation

PepsiCo

The Boeing Company

Silver Oak Cellars

Agricultural

Odom’s TN Pride Sausage, Inc.

The Wine Group Wiskerchen Cheese, Inc.

Monsanto Company Automotive

Cooper Standard Ewald Automotive Group Harte Auto Group Prestolite Performance, LLC

Illinois Cancer Care Kaiser Permanente

Prime Therapeutics, LLC

RoadSafe Traffic Systems, Inc. Quinn Company Wick Building Systems Anonymous Consulting

Catapult Leadership Communications, LLC Heart and Head Leadership, LLC Management Research Group Moffat & Nichol PMC Social Policy Research Associates Anonymous Defense Contracting

ADS, Inc. Energy & Utility

Ontario Power Generation Tennessee Valley Authority Anonymous (2) Entertainment

Clientel Anonymous

Manufacturing

3M Advanced Circuits AlloSource

Pembroke Regional Hospital

Higher Education

REI Spec’s Family Partners

Corning Incorporated

Anonymous (2)

Ex-Tech Plastics, Inc. FMC Technologies Holland Company Honeywell Safety Products Hubbell Incorporated JRB Attachments, LLC KX Technologies, LLC Liberty Safe and Security Products, Inc. Murata Power Solutions

Stanford University, VPUE

Racor, division of Parker Hannifin Corporation

Alliance Management Hyatt Hotels and Resorts Lone Star Parking Management Marriott International, Inc. Omni Hotels Papa Johns International, Inc. Universal Orlando Resort Warner Center Marriott Woodland Hills Wild Dunes Resort World Travel Holdings IT Consulting

Intelliware Development, Inc. VMC Consulting Inc. International Organization

The World Bank Group

Truck Bodies & Equipment International, Inc.

Dell DevonWay, Inc. PCTEL, Inc. Riverbed Technology VMC Consulting, Inc. XO Group, Inc. Anonymous Telecommunication

Cox Communications, Inc. Team Fishel Transportation

Briggs Equipment, Inc. CSX Corporation, Inc.

News Media

FedEx

Associated Press

National Express Corporation

Topeka Capital-Journal

SkyWest Airlines

Nonprofit

E-Z Rent-A-Car

Anonymous (3)

Bill & Melinda Gates Foundation

Waste Disposal

David Suzuki Foundation

Waste Management, Inc.

GreatSchools, Inc. Highline Public Schools

OTHER

The Institute for Human and Social Development

G&K Services, Inc. Calista Corporation

Irvine Housing Opportunities, Inc.

Anonymous (58)

KIPP New Orleans KIPP St. Louis KIPP Washington D.C. LL Global, Inc.

Allstate Insurance Company

National Committee for Quality Assurance

Mutual of Omaha Insurance Company

Vera Institute of Justice

Safety Insurance Company

Anonymous

Woodmen of the World Life Insurance Society

Technology

Anonymous (3)

MNM Insurance

Orchard Supply Hardware

The Cookson Company, Inc.

Samuel Merritt University

American Blue Ribbon Holdings

Edwin Watts Golf Shops, LLC

Charter Steel

Mennie Machine Company, Inc.

Hospitality

Costco Wholesale Corporation

The Pharmacy Express of C&K Express, LLC

Georgia Perimeter College

Anonymous (3)

Brownells, Inc.

Amcor Rigid Plastics

ETS-Lindgren

Construction

Parsons Corporation

Altrec

Franciscan Missionaries of Our Lady Health System

Miraca Life Sciences, Inc.

McHugh Construction

Retail

Riverside Logistics

Cristal Global

Hope Haven, Inc.

Cushman & Wakefield, Inc.

Flextronics Global Services

Healthcare

Elesys North America, Inc.

INTREN

Logistics

Preface This study is the fourth of its kind that 3D Group has conducted. We completed the first 360 Degree Feedback Benchmark Study over a decade ago when our consultants recognized that academicbased “best practices” were not always aligned with the needs of our clients. Often, what sounded good on paper just didn’t fit the organizational realities our clients were facing. This led us to wonder, to what extent were companies that were implementing 360s choosing to veer away from or align with established best practices. We also realized, of course, that practitioners often ask about what choices other organizations are making when designing 360 degree feedback processes. While we could share insights from our 19 years of experience delivering 360 degree feedback with our own clients, we felt it would be more valuable to have a data-based answer to many of these questions that was not constrained by our own preferences (we named ourselves

Current Practices in 360 Degree Feedback: A Benchmark Study of North American Companies

Data Driven Decisions for a reason!). As a result, this study samples widely from companies across North America with fewer than 5% of the sample coming from our own clients.

Taking a step beyond the well-documented best practices, this study sheds light on the day-to-day realities of how companies actually design their 360 degree feedback processes. These results should provide a reliable reference point for practitioners as they make design decisions with respect

to their own 360 degree feedback programs. While it isn’t always a good idea to blindly follow the lead of other companies when building a tailored 360 process, it certainly is beneficial to understand how your own design choices fit with common trends in the industry.

While the current study focuses primarily on the practices of 211 companies, one significant value in these results lies in the trends over time. For each of the previous studies we have tweaked our methods and our questions slightly, but we have also retained enough critical questions each year to allow insight into trends within the 360 degree feedback industry over time. This year, we added new questions to address the changing landscape of technology in the assessment industry. Our new survey also adds questions to expand our knowledge of the coaching practices that are so integral to the 360 degree feedback follow-up. One of the biggest additions this year related to the ever-controversial discussion about whether 360 degree feedback should be used for “development only” or for “decision making.” With the help of David Bracken, who is perhaps the most vocal advocate nationally for the “decision making” side of the debate, we reframed the question (see page 27) to allow respondents to more clearly indicate how the tool is used.

This year we also significantly increased our sample size. Through collaboration with Cox eLearning Consultants, LLC, we were able to greatly expand the number of participating companies compared to our previous three studies. The 211 companies we surveyed came from a wide variety of industries with large and small companies represented. Given the expanded sample in this study, we are confident that the latest results accurately reflect current 360 degree feedback practices among companies based in North America.

We would like to acknowledge everyone involved in creating this research. The study was authored

by Dale Rose, Ph.D., Justine Lewis, and Micheline Magnotta, with significant contributions from many others among the 3D Group staff. The survey was administered by John Diller and Chloe Doan from 3D Group. An important collaboration with Dan Cox and Barbara Sanner from Cox eLearning Consultants helped significantly increase our sample pool and added a tremendous amount of data to our study.

2

3D Gro u p

TA B LE O F C O NTENTS 2 Preface 4 About 360 Degree Feedback 4 About This Study 5 Executive Summary 6 Sample and Organizational Characteristics 7 Getting Started Decision Point #1: What type of 360 degree feedback program?

Decision Point #2: What levels within the organization should our 360 feedback process target?

Decision Point #3: How often should the 360 degree feedback program be run?

9 Survey Development Decision Point #4: Develop surveys internally or buy a pre-validated, off-the-shelf survey?

Decision Point #5: What should the characteristics of the survey be?

13 Rater Selection Decision Point #6: Who should select the raters?

14 Pre-Survey Administration Decision Point #7: What should we do to prepare the feedback recipients and raters for the survey?

15

Completing Surveys and Generating Reports

18

Feedback Reports

Decision Point #8: What mode should be used for data collection?

Decision Point #9: How will surveys be scored and reports generated?

Decision Point #10: Who should score the surveys and generate the reports?

Decision Point #11: Should reports contain normative data?

Decision Point #12: Should there be a minimum number of raters required?

Decision Point #13: How should open-ended questions be reported?

Decision Point #14: Should the feedback report include item variation?

Decision Point #15: Should the feedback report include a list of rater names?

Decision Point #16: Who receives a copy of the feedback report?

23 Developmental Support Decision Point #17: How should the report be delivered to the feedback recipient?

Decision Point #18: What type of development support should be available to feedback recipients?

25

360 Degree Feedback Coaching Decision Point #19: What type of training or certification should 360 degree feedback coaches have?

Decision Point #20: Should the 360 degree feedback coaches be internal staff or hired externally?

Decision Point #21: How should the coaching be delivered to feedback recipients?

27 Using 360 Degree Feedback Results Decision Point #22: Should the 360 degree feedback data be used for development purposes, administrative purposes or a combination of the two?

29 Attitudes about 360 Degree Feedback 29 Looking Back 30 Looking Forward 31 Conclusion 32 Research Methods 32 References

About 360 Degree Feedback Popular programs in human resources, talent management and leadership development seem to come and go, but few initiatives have had impact that has lasted as long as 360 degree feedback.

360 degree feedback has been proven time and time again as one of the most effective tools for providing insight into co-worker perceptions. Because leaders rely on getting work done through others, having an accurate understanding of co-worker perceptions is an essential ingredient to leadership success. 360 degree feedback can be used in a vast array of formats, from providing a single leader with development insights, to acting as a large-scale mechanism for organizational change and alignment.

In a typical 360 degree feedback program, an employee is given feedback via surveys distributed to his or her boss, peers, customers, supervisor, and direct reports. These surveys contain questions about the employee’s behavior at work. Open-ended comments may be requested (for example, “Name one thing this associate could do better at work”) to add richness and detail to the numerical ratings. Human Resources or an outside consultant summarizes the data into a feedback report that is given to the employee who is the subject of the survey, the feedback recipient. Qualified coaches may be available to confidentially discuss the report and assist feedback recipients with skill improvement and development planning.

Participating in a 360 feedback program evokes many emotional responses, including curiosity (“What do my co-workers think of me?”), joy (“Wow, the extra effort I put in has been noticed!”), fear (“Will I get fired over this?”), self-awareness (“I hadn’t thought about my behavior like this before”), and even anger (“I can’t believe they said that about me!”). How, then, do organizations harness the power of this information and the emotional responses it provokes to maximize individual and organizational effectiveness?

Three fundamental assumptions are implicit in 360 degree feedback programs: (1) awareness of one’s strengths and development needs can be a valuable tool to improve individual effectiveness; (2) awareness of strengths and development needs will result when employees hear honest, accurate feedback about their behavior; and (3) the most valuable feedback about job-related behavior comes from co-workers and others who observe that behavior firsthand.

About This Stud y This 360 degree feedback benchmark study investigates trends and common practices in 360 degree feedback across North America. There exists a great deal of research on “best practices” for 360 degree feedback.

Competing with the plethora of advice is the reality of what organizations are actually doing and how they perceive the success of the program.

Current Practices in 360 Degree Feedback: A Benchmark Study Of North American Companies provides a comprehensive description of the range of 360 degree feedback programs used by North American companies.

This report describes 211 programs in the United States and Canada across 30 different industries. The report comprises data received from respondents who are Human Resource professionals and business leaders

4

3D Gro u p

who design and implement 360 degree feedback programs. While 3D Group has conducted four such studies over the last decade, in this study you will find fresh information about 360 degree feedback practices, analyses of trends over time and an expansion of our investigation of coaching trends, overall attitudes and support for 360 degree feedback programs.

To make our findings useful to everyone from the academic sector to the first-time user, we have structured the study around a series of “decision points.” These decision points guide the reader through the intricacies of designing a 360 degree feedback process from designing survey content to selecting raters to providing follow up support to feedback recipients.

Executive Summary 3D Group interviewed 360 degree feedback program managers from 211 companies of all sizes based in the United States and Canada representing over 30 industries.

Interview questions addressed the size of the program, sources of the survey content, selection of raters, changes to the program processes, development support and the future of the 360 degree feedback program in the organization. We administered an in-depth electronic survey in order to profile the wide assortment of 360 degree feedback programs now in use. Our findings reveal vast differences between programs in survey content, methods used to select raters, use of the feedback and assumptions used in the decision-making process about 360 degree feedback programs. Some of the key findings are highlighted below.

3 The importance of rater anonymity was strongly indicated in two ways:

1) The minimum number of raters is increasing and 2) 52% of organizations report that rater anonymity is essential.

3 Coaching feedback recipients about how to use their 360 degree feedback

continues to be very common, and coaching is felt to be a highly valuable component of the process.

3 360 degree feedback continues to be a developmental tool, but it is clear

that feedback is also being used to drive important talent decisions.

3 Companies continue to use paper surveys despite the apparent efficiency

of electronic surveys.

3 Surveys are getting shorter.

3 Feedback results are displayed in increasingly graphical ways.

3 The 5-point scale has become the industry standard with over 85% of

companies using this type scale.

3 Companies are paying increased attention to rater training.

3 Comparisons to company norms and previous years’ scores are on the rise.

3 Human Resources Departments are more involved than ever. They are now

more often picking raters and receiving reports.

3 6 0 F e ed b a c k Be n c h m a r k R e port

5

Sample and Organizational Characteristics This study recruited Human Resources and business professionals from a wide variety of organizations across North America. Respondents were self-selected and included companies that provided data for our 2002, 2004 and 2009 studies. Participants were selfidentified as Human Resources and business professionals who

table 1

were active practitioners of 360 degree feedback.

On which continents do your managers work?

Of the 211 valid respondents, 15 of these came from 360 degree feedback practitioners in Canada; the remaining 196 were from

North America

companies headquartered in the continental United States.

While some responses were received from international professionals in China, India and Europe, these responses were excluded. Respondents were asked, “On which continents do your managers work?” and all the respondents had managers working in North America (see Table 1).

Nearly half (45%) of the sample had been using 360 degree

100%

Europe Asia

21% 18%

South America

14%

Africa Australia

10% 10%

feedback for 1 to 5 years (see Figure 1). One-quarter of the organizations were newer to 360 degree feedback, with one year of experience. The remaining 30% of respondents said they had been doing 360 degree feedback within their organization for more than 6 years.

All business sizes were well-represented in this study. The average number of employees per organization in the sample was 14,651, with a median of 800. As Figure 2 illustrates, there was a relatively even distribution in terms of organization size.

figure 1

figure 2

How long has your organization been conducting this 360 degree feedback process?

How many employees does your company employ?

50%

25%

45%

30%

6

3D Gro u p

+ 01 0,0

10 1-

10

00 0,0

00 ,00 10

01

-1

0,0

00 1,0

1,0

16 or more years

1-

11-15 years

0

6-10 years

5%

0%

50

1-5 years

5%

50

Less than 1 year

3%

1-

0%

6%

10

21%

10%

15%

10%

0

25%

17%

15%

10

20%

15%

26%

23%

20%

1-

40%

30%

Getting Started When organizations design a 360 degree feedback process, they typically begin by making some program-level decisions. These decisions include overarching questions such as, “Who should our 360 degree feedback program target?,” “Should 360s be run whenever we perceive a need or should they be regularly scheduled?” And “What type of 360 degree feedback program is right for our organization?”

decision point

#1: What type of 360 degree feedback program?

There are four types of 360 degree feedback programs that organizations typically use:

1 Individual: The 360 degree feedback process is made available to individuals on an

as-needed basis.

2 Training and Development: The 360 degree feedback program is part of a larger training

and/or development program for employees.

3 Department wide: Specific departments or sub-units of the organization (e.g., Sales

Department) go through the 360 degree feedback process.

4 Enterprise-wide: All or nearly all of the managers within the organization participate in

the 360 degree feedback process.

As illustrated in Figure 3, many organizations have more than one type of 360 degree feedback process. The most common use by far was for individuals, with training also being quite popular.

While Department-wide and Enterprise-wide programs were less popular, it is interesting to note that even companies that had enterprise-wide processes also concurrently used other methods.

Thus, companies seem to utilize 360 degree feedback programs in an expansive range of settings for a wide range of uses.

figure 3

Which types of 360 programs does your organization implement?

80% 70%

78%

60% 50% 40%

45%

30% 20%

17%

10% 0%

Individual

Training and Development

Department-wide

23% Enterprise-wide

3 6 0 F e ed b a c k Be n c h m a r k R e port

7

decision point

#2: What levels within the organization should our 360 feedback process target?

Almost half (48%) of organizations targeted managers with their 360 degree feedback program and 43% of respondents targeted executives (see Figure 4). In 2009, there was a shift in 360 degree feedback programs toward including all employees, and in 2013 this trend continued. Specifically, in 2004, 14% of respondents targeted all employees with their 360 degree feedback programs (3D Group, 2009). This number increased to 20% in 2009 and reached 24% in 2013. While the 360 degree feedback process is typically designed for upper management, this shift illustrates an increasing awareness of the power of 360 degree feedback to develop human capital across the organization. figure 4

What level(s) does your 360 program target?

I wish the process was ❝

presented throughout

48%

Managers

43%

Executives

the company versus given to only a select few.

34%

All Leaders

❠

24%

All Employees Individual Contributors

15% 0%

decision point

If I could go back and change ❝

one thing about our 360

program I would just like to do it more often.

❠

5%

10%

15%

20%

25%

40%

45%

50%

A slight majority of organizations administer 360 degree feedback surveys on a set schedule, with 55% of organizations administering their 360 survey annually or every two years. As Table 2 illustrates, the percentage of organizations that have annual 360s has remained relatively constant from 2002 to 2013. A large percentage, 38% of companies, still administer 360s on an as-needed basis (see Figure 5). One company reported that they did 360s more than once a year.

figure 5

2002

2004

2009

2013

As needed

47%

36%

28%

38%

Annually

42%

38%

41%

44%

40%

Every two years

11%

6%

28%

11%

30%

— 20%

3%

8%

Other

35%

#3: How often should the 360 degree feedback program be run?

table 2

30%

How often are 360 surveys administered?

45% 35%

44% 38%

25% 20% 15% 10%

11%

5% 0%

8

3D Gro u p

As needed

Annually

Every two years

8% Other

Survey Development As organizations embark upon the 360 degree feedback process, one of the first questions is what behaviors the survey will include. The following questions about survey development prove to be instrumental in the overall success of the 360 degree feedback program: “Should we develop our survey internally or buy an off-the-shelf solution?” “What should the survey characteristics be?”

Addressing both of these decisions requires weighing organizational needs against available resources.

decision point

#4: Develop surveys internally or buy a pre-validated, off-the-shelf survey?

When asked who developed their 360 degree feedback survey, about a

❝

I should have used a multi-

method approach. In other words, I’ve found that doing both an

off-the-shelf 360 combined with

qualitative interviews with key

stake-holders yields more effect

and specific feedback.

❠

third (33%) of organizations developed their survey internally with a committee or team (see Figure 6). 42% of companies engaged the support of a consultant either to support their internal survey development or to design the survey entirely. Only about a fifth (20%) of those polled purchased an off-the-shelf product. This year’s study demonstrated that the trends for survey design remained relatively constant. In 2009, 32% developed their surveys internally by committee or team, 26% used an internal team along with the help of a consultant, 22% used an off-the-shelf tool and 18% had a consultant develop the survey entirely (3D Group, 2009).

figure 6

Who developed the survey and report format you use?

Other

5%

Committee/team (internal)

20%

Bought ”off-the-shelf”

22%

Consultant

33%

Internal committee/ team with a consultant

20%

Of the respondents who bought an off the shelf tool, a substantial majority (71%) said they did not customize the content. Only 29% of respondents added customization to the survey they purchased.

This represents a significant shift from 2009 when 43% choose to customize the content of their off-the-shelf tools. This shift could be explained by an increase in variety of off-the-shelf tools available in the marketplace, such as surveys designed for industry-specific roles (e.g., retail managers, or education leaders).

3 6 0 F e ed b a c k Be n c h m a r k R e port

9

If I could do it over again I would have ❝

created different versions for different

levels in the organization but with the same competency standards.

❠

decision point

#5: What should the characteristics of the survey be?

The majority (53%) of participants reported that they used the same survey for all participants (see Figure 7). For those who chose to use different surveys for different groups within their organization, the feedback recipient’s level in the organization often dictated the differing surveys.

Changing the survey based on the feedback recipient’s specific job function remains the least common practice.

figure 7

Was the survey content:

Different, depending on the participant’s level

28% 53%

Different, depending on the participant’s job

10

3D Gro u p

19%

The same for all participants

Survey length is a critical component of survey design, since a survey that is too short loses its validity and a survey that is too long can become cumbersome and time consuming for raters and feedback recipients. The shortest 360 degree feedback survey only contained 2 questions; the longest had 240 questions. The average survey length was 35 questions. Over half (53%) of organizations used 360 degree surveys with 11 to 40 questions (Figure 8). This demonstrates a significant decrease in the length of 360 degree feedback surveys from 2009. That is, the majority (42%) of surveys in 2009 contained between 41 and 69 items. Today, only 21% of surveys contain between 41 and 69 items. Even the number of surveys over 100 questions has decreased from 10% in 2009 to only 3% in 2013. figure 8

How many questions does the survey contain?

60%

2009

53%

50%

2013

40%

42% 33%

30% 20% 10% 0%

21%

17%

6%

8% up to 10

11 to 40

41 to 69

6%

10% 3%

70 to100

over 100

93% of 360 degree feedback surveys include some form of a quantified rating scale. While the question of the “ideal� scale was once quite common, it appears that the 5-point scale has become the clear winner with 87% of companies using a 5-point rating scale. The second most popular rating scale was the 4-point scale but that only represented 6% of respondents (Figure 9). Compared to 2009, when the 7-point scale was more common (16%), and 4-point and 10-point scales represented more of the sample (4% each), the shift toward the 5-point scale becomes clear. The good news about these findings is that research has long pointed out the utility of the 5-point scale and so now this scale is not only considered a best practice given the research, it appears to also be the most popular rating scale for 360 degree feedback. figure 9

How many points are on your rating scale?

3-point scale 4-point scale 6% 1%

1%

5-point scale

87%

1% 2% 2%

5-point scale 6-point scale 7-point scale 9-point scale 10-point scale

3 6 0 F e ed b a c k Be n c h m a r k R e port

11

In terms of the type of rating scale, half of organizations (49%) use an agreement rating scale, such as “Strongly Agree” to “Strongly Disagree” (Figure 10). The second most popular type of rating scale was an effectiveness scale, which was represented in 31% of responses. One third (33%) of the “Other” responses wrote in a rating scale that included expectations, such as “meets expectations.” figure 10

Does your rating scale focus on: Agreement (such as ”strongly agree” to ”strongly disagree”)

49%

Effectiveness (such as ”not effective” to ”very effective”)

31%

Frequency (such as ”always” or ”daily”)

23%

Importance (such as ”very important” for the job)

16% 7%

Other 0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

In addition to rating scales, another important component of survey development is the use of open-ended questions. Open-ended questions offer raters the opportunity to explain their quantitative ratings and when done well they can add richness to the survey results (Rose, et al., 2004).

While open-ended questions can add considerable detail to 360 degree feedback responses, there is still a trade-off in terms of the time it takes to complete them. As a result, open-ended questions, however, are not always used, and the number of open-ended questions varies quite a bit. This study found that companies use anywhere from zero to 67 open-ended questions on their 360 degree feedback surveys. The mean number of open-ended questions was 4.27, with the median of 3 and the standard deviation of 7.4. 65% of respondents said their 360 degree feedback surveys have between 1 and 5 open-ended questions (see Figure 11). From 2009 to 2013, there has been a significant increase from 2% to 15% of 360 degree feedback surveys that have no open ended questions. figure 11

How many open-ended questions are included?

25%

22%

20% 15%

20% 16%

15%

10%

11% 7%

5% 0%

12

3D Gro u p

0

1

7% 2

3

4

3% 5

6 to 10 over 10

Rater Selection Choosing who will complete the survey, rater selection, is an exceedingly important aspect of the 360 degree feedback process. Not only do the raters selected have a big impact on the results, but the selection of raters can influence how the feedback recipient views their results and how much of an impact the 360 process can have on the feedback recipients’ professional development. A feedback recipient who feels the chosen raters provided biased or unfair feedback is unlikely to take the results seriously. A well-balanced group of raters, and some control over rater selection, will positively affect whether the feedback recipient is compelled to make behavioral changes based on their feedback results.

In theory, anyone who has had the opportunity to observe the feedback recipients’ behavior on the job can provide useful feedback as a rater. In practice, raters typically include members from all or some of the following categories: supervisors, peers, direct reports, business partners, customers, as well as the feedback recipient themselves. There are a variety of opinions as to the usefulness of self-ratings; however, one common viewpoint is that self-awareness is an important aspect of creating behavioral change in feedback recipients. Being able to compare self ratings to non-self ratings can provide a useful window into an individual’s level of self-awareness.

decision point

#6: Who should select the raters?

Some 360 degree feedback experts believe that raters should be chosen solely by the

❝

feedback recipient in order to ensure a sense of ownership and buy-in from the recipient.

IfIf II could could do do itit over over again Iwould wouldinclude include more more raters. raters.

Others argue that to ensure a well-balanced representation of their performance, an

❠ ❠

objective third party or manager needs to choose the raters. It appears that the most common choice among organizations is to do a combination of the two approaches by asking the feedback recipient to select the raters with the managers’ approval (see Figure 12). However, the percentage of organizations that have used this method has declined from 61% since 2009. Instead, Human Resources and managers seem to be playing a larger role in selecting raters in 2013.

figure 12

Who chooses raters?

70% 60%

2009 2013

61%

50% 40%

42%

30%

25%

20% 10% 0%

4% Participant with manager approval

HR (based on organizational chart)

31% 17% Participant only

2% 10% Manager only

3 6 0 F e ed b a c k Be n c h m a r k R e port

13

Pre-Survey Administration With 360 degree feedback, the process is just as important as the results. The way an organization sets up the entire 360 degree process will influence not only the feed-

I wish we had better ❝

training on the process

for all involved.❠

back that recipients receive but it will also have tremendous effects on the validity of the results. Without a clear understanding of the purpose and process, 360 feedback recipients are unlikely to trust or act on the survey’s findings. Likewise, without proper training, raters may not be forthright in their answers and are unlikely to provide the constructive feedback that is so important to a leader’s success. Without rater training to minimize biases, these biases can taint the accuracy of the results.

decision point

#7: What should we do to prepare the feedback recipients and raters for the survey?

Most organizations seem to understand the value of readiness, so they invest considerable time and resources into ensuring the feedback recipients and raters are prepared for the 360 degree feedback processes. 40% of respondents said feedback recipients were introduced to the program through a formal orientation by their human resources department or other internal group (see Figure 13). While 34% of organizations polled said they introduce the program through written communication (a letter or email) sent to feedback recipients. A remarkable 33% of organizations reported that they introduced the program to feedback recipients in a one-on-one setting. This finding in particular illustrates how much emphasis, time and resources organizations allocate to the pre-survey administration. Of the 3% of respondents who selected “other,” most respondents (71%) cited an internal website or other eModule.

Rater training is also a valuable component of pre-survey administration. Rater training typically consists of making raters aware of the purpose of 360 degree feedback and the various types of rater errors and rater biases (e.g., the leniency effect or the halo effect) that commonly occur when making observations of behavior. From 2009 to 2013, there has been a significant drop in the number of organizations that provide no rater training, from 61% to 26% respectively. This demonstrates an increased understanding on the part of practitioners as to the value of rater training.

Of those who do provide rater training, the most popular method is still text included in the survey (45%); although in-person rater training has gained in popularity from 13% to 35% over the last four years (see Figure 14). figure 13

figure 14

How do you introduce the program to 360 feedback recipients?

How do you train your raters to complete surveys?

Formal orientation by HR or other internal employee

40%

Discussed during group meeting

3D Gro u p

5%

10%

15%

20%

25%

30%

35%

40%

29% 2009

6% 5% 0%

10%

56%

35%

19% 14%

Teleconference

6% 0%

6%

Web training

29%

Formal orientation by vendor/consultant

13%

Email

33%

One-on-one

45%

In-person training

34%

Described in letter or email

14

Text included in the survey

2013 20%

30%

40%

50%

60%

Completing Surveys and Generating Reports With technological advances, what used to be one of the most complex aspects of 360 degree feedback can now be boiled down to three simple questions: (1) What mode should be used for data collection?

(2) How should surveys be scored and reports generated?

(3) Who should score surveys and generate reports?

decision point

#8: What mode should be used for data collection?

There are two main ways in which organizations collect 360 degree feedback survey data: “traditional� paper surveys and popular electronic surveys. The use of electronic surveys has become an attractive alternative for 360 degree feedback program managers because of the ease of administration, real-time scoring and flexibility. Using the Internet and other electronic means for data collection, processing and analysis provides significant time saving over the traditional paper methods. For this reason the majority (70%) of organizations use only electronic 360 feedback surveys (see Figure 15).

There are three significant security issues that 360 degree feedback administrators must take into account when conducting web-based surveys. These issues include: (1) making sure the ratings come from the correct raters; (2) protecting the data from corruption; and (3) ensuring that results are only shared with the designated recipients.

These issues can be easily resolved with most professionally-developed electronic 360 survey programs; however, some organizations still create and implement their surveys in house, which might explain the 29% of organizations that still use paper surveys either in collaboration with electronic surveys or as a sole means of data collection. Also, employees in some job roles, such as construction, utility workers or truck drivers, may not have ready access to company email. figure 15

How do raters complete surveys?

86%

Internet only

Both paper and Internet

Paper only

70% 12% 17% 2009

0%

2013

12% 0%

50%

100%

3 6 0 F e ed b a c k Be n c h m a r k R e port

15

decision point

#9: How will surveys be scored and reports generated?

Depending on the number of feedback recipients, analyzing data and generating 360 degree feedback reports can be daunting. Many organizations turn to outside vendors or software programs to complete this process. There are many options available, including full-service 360 degree feedback providers that have experts to handle all of the nuances of the 360 process, as well as desktop software programs that allow program managers to do the administration themselves. The study found that 65% of companies use a web-based reporting tool, while 17% use desktop office software and most likely generate reports in house (Figure 16). Interestingly, 30% of those who reported “other” volunteered that reports were supplied by a vendor, indicating that they were unclear on what tools the vendor used to score and generate the reports. Another 18% of those who replied “other” reported they computed the scores and created final reports manually. figure 16

How are surveys scored and reports generated?

Other

10%

Desktop office software

18% 65%

Desktop 360-specific software

16

3D Gro u p

7%

Automated on-line (web-based reporting)

decision point

#10: Who should score the surveys and generate the reports?

Some companies use internal resources to handle the technical aspects of scoring and generating reports, while others use an outside vendor to assist in this process. Using an outside vendor for this aspect of the 360 process is often preferable because it lends objectivity, anonymity and confidentiality to the results (see Timmreck and Wentworth, 2000). Furthermore, many program managers report that having the data sent to a third party gives their program the benefit of real and perceived data security, not to mention the major benefit of saving their own administration time.

The majority (56%) of organizations recognize the benefits involved with hiring a vendor or external consultant to score and generate the 360 degree feedback reports (Figure 17). Looking back through 2002, 2004, and 2009 the majority of 360 practitioners have consistently reported using a vendor for this aspect of the 360 degree feedback process. figure 17

Who scores the survey and generates reports?

Other

6%

Internal – Human Resources

38%

56%

Vendor/consultant

3 6 0 F e ed b a c k Be n c h m a r k R e port

17

Feedback Reports To obtain the maximum development benefits for feedback recipients it is important that reports are clear, relevant and reliable. One should not

We should have hired someone ❝

from outside to manage the process.

Our department is too small!❠

need a Ph.D. to interpret the report. On the other hand, a 360 degree feedback report should not be so simple and trivial that it leaves out valuable information. In addition to reporting scale scores by overall ratings and rater groups, there are several other options to consider when deciding what to include in the feedback report, such as reporting normative data and item-level scores.

decision point

#11: Should reports contain normative data?

One consideration of what to include in the final 360 degree feedback report is whether to present standardized normative data. Practitioners of 360 degree feedback vary on their opinions of the usefulness of normative data. Norms are usually average scores of both scales and individual questions. They should provide a meaningful comparison of individuals who have completed the 360 survey recently. They can be based on the average within the organization or based on an external sample. 360 practitioners who argue against the use of norms believe the added comparative information detracts from the primary purpose of the feedback: identifying individual areas for growth. Proponents of using norms argue that it is important to have these benchmarks to help interpret the data (see Yukl and Lepsinger, 1995). In addition, including norms on the feedback report makes it easier for feedback recipients to determine whether a rating is favorable (i.e., is a “3.9” rating high or low?). Norms may be especially important for 360 degree feedback interpretation since ratings tend to be positively skewed (see English, Rose & McClellan, 2009). For instance in some situations, a 3.9 on a 5-point scale would seem like a positive score, but if the norm for that question is 4.4, perhaps that item should still be considered an area for improvement.

In general, the use of norms is increasing in popularity, as there was a significant increase in the number of organizations that use norms

It’s important to be consistent with ❝

the questions from year to year to

have better comparative data.❠

from 2009 to 2013 (see Figure 18). In particular, the use of individual comparative norms from year to year has increased from 6% to 26%.

Organizational norms continue to be the most prevalent type of norms used in 360 degree feedback.

figure 18

What standard measure, if any, do you have for feedback recipients to compare their results to?

6%

Previous year’s scores Do not have a standard measure for comparison

26%

26% 0%

3D Gro u p

2013

54%

30%

Organization Norms

18

2009

20% 19%

National Norms

10%

20%

34% 30%

40%

50%

60%

decision point

#12: Should there be a minimum number of raters required?

Providing raters with anonymity is one of the most important things a 360 degree feedback process can do to ensure accurate ratings. Since most organizations ensure rater privacy, an important question for consideration is the minimum number of raters per category that is required in order to present results. This becomes more critical as the number of raters for a particular group decreases.

For instance, with no minimum threshold, if only one direct report replies to the survey invitation, the feedback recipient can clearly see exactly how that rater responded to the survey. In order to protect the anonymity of raters, most organizations define a decision rule for how many raters must respond from a particular rater group for that group’s data to appear in the report. Typically, if the number of raters is below this minimum threshold then those ratings get rolled into another group. This issue in particular can prove challenging for small or flat companies with small numbers of direct reports or peers.

In the current study, rater anonymity became even more important to 360 degree feedback practitioners, with 32% of organizations requiring 4 or more raters per rater category, compared to only 10% in 2009. The majority of respondents still said they had a 3-rater minimum per category, but this majority decreased significantly in favor of even larger minimum numbers. figure 19

What, if any, is the minimum number of raters per rater category (e.g., peers, direct reports) for data to show in a report?

70%

2009

64%

60%

2013

50% 40%

38%

30% 20% 10% 0%

14%

11%

Two

Three

2% 10%

4% 10%

Four

Five

7%

4%

0% Six

5%

More than six

12%

19%

No minimum

3 6 0 F e ed b a c k Be n c h m a r k R e port

19

decision point

#13: How should open-ended questions be reported?

Open-ended questions have the power to add depth and detail to a 360 degree feedback report.

As illustrated in figure 20, most organizations report open-ended responses verbatim (64%); however, the number of organizations that are screening these responses before reporting them doubled between 2009 and 2013. In 2013, there was also a significant jump in organizations that analyzed the open-ended responses for themes, from 4% to 18%. This demonstrates an increased thoughtfulness around open-ended questions. figure 20

How are written comments processed?

90% 80%

2009

84%

2013

70% 60%

64%

50% 40% 30% 20%

15%

10% 0%

20

3D Gro u p

8% Reported verbatim

Screened

4%

18%

Analyzed for theme

decision point

#14: Should the feedback report include item variation?

There are numerous options when considering how to illustrate in any report how the responses vary (e.g., an average score of 3 resulting from a 1 and a 5 rating, instead of two ratings of 3). This variation can help the feedback recipient to better interpret the data from their 360 degree feedback report. Variability can be presented by showing raw scores or by illustrating results graphically to show frequency distributions, standard deviation, a range of scores, or a combination thereof.

Presenting a range of scores can be a good way to preserve rater anonymity and can provide more detailed information than reporting only averages. Although presenting an index of variation has some advantages over presenting the frequency, Dalessio and Vasilopouslos (2000) suggest presenting the range of scores only in organizations where individuals are comfortable with the 360 process. In addition, there are some concerns over the general comprehension and meaning of variation for feedback recipients. In some cases, variation is best left out to avoid overwhelming the feedback recipients with information.

Since 2002, 3D Group has tracked the use of variation in 360 degree feedback reports and has found a few trends, as represented in Table 3. There has been a steady decline in organizations that chose to leave out variation entirely, from nearly half of the organizations in 2002 showing no item variation to only 14% in 2013. Illustrating variation graphically was relatively rare but is now commonplace (increasing from 13% in 2002 to 37% in 2013). The increased popularity of graphic illustration could be due to the fact that it illuminates the variation in a way that is less overwhelming and easier to comprehend. figure 21

How, if at all, do you present variation in scores across raters?

37%

Illustrate Graphically Raw Scores – Frequency Distribution

33% 14%

None

13%

Standard Deviation 0%

5%

10%

15%

20%

25%

30%

35%

40%

table 3

Illustrate Graphically Raw Scores Frequency Distribution None Standard Deviation

2002

2004

2009

2013

13%

11%

25%

37%

7%

17%

31%

33%

47%

41%

42%

14%

7%

12%

19%

13%

3 6 0 F e ed b a c k Be n c h m a r k R e port

21

decision point

#15: Should the feedback report include a list of rater names?

Another consideration when designing a 360 degree feedback process is whether to include a list of rater names in the report. Including a list of invited raters can help remind the feedback recipient who was included in each group (“Did I include all my direct reports, or did I exclude the new hire?”), which can aid in interpreting the results. Rater names can help remind both feedback recipient and coach what types of raters comprise each rater group, which in turn assist in a clearer interpretation of the data. Despite their potential utility, the vast majority of organizations choose to leave rater names off the 360 degree feedback report entirely (82% in both 2009 and 2013). This choice is most likely intended to further protect the anonymity of the raters. Despite the value that knowing the source of feedback brings to help interpret the results, only 15% of companies included the names of either the actual respondents or those invited to respond (Figure 22).

decision point

#16: Who receives a copy of the feedback report?

Determining who should receive a copy of the feedback report might seem like a trivial question; however, this decision has far reaching implications. Some practitioners believe survey results should remain confidential, so they only share the results directly with the feedback recipients.

Other practitioners feel the 360 degree feedback will only create real change if the manager of the feedback recipient also receives results. When the manager or HR personnel are involved, it ensures that the feedback recipient is held accountable and results are not ignored (Bracken and Rose, 2011). At the same time, providing reports to the manager or Human Resources increases the likelihood that the results will be used for administrative purposes (e.g., promoting, firing, pay increase), whether or not that is the intent of the organization’s 360 degree feedback program.

Although a large majority (78%) of organizations still provide a copy of the 360 feedback report to feedback recipients, there has been a shift towards managers and HR receiving copies of reports and away from feedback recipients receiving copies of the reports (see Figure 23). In 2013, over half of the organizations give copies of 360 degree feedback reports to managers (59%) and HR departments (60%). Some organizations require that the feedback recipient shares the results with their supervisors, leaving it “up to the feedback recipient whether [to share a] summary or all.” figure 22

figure 23

Do you include names in the report, and if so which ones?

Who receives a copy of the report?

Participant

Respondent Rater names

7% 8%

85%

78% 39%

Participant’s Direct Manager

No names included

0%

3D Gro u p

60%

2009

23% 24%

360 coach

22

59%

37%

Human Resources

96%

20%

2013 40%

60%

80%

100%

Developmental Support decision point

#17: How should the report be delivered to the feedback recipient?

Most organizations treat 360 degree feedback reports as highly sensitive documents. Many organizations choose to make the actual experience of receiving the report more personal, while others choose a simpler automated method of electronic distribution. A small percent of organizations (4%) deliver the reports in a workshop setting (Figure 24). As in 2009, the electronic method of distribution remains the most prominent, with 43% of companies delivering 360 degree feedback reports via email or online. There was a large increase in the prevalence of reports being delivered by managers, just as there was a significant increase in direct managers receiving copies of the reports (Figures 23 and 24). 27% of organizations chose to have the reports delivered by the coach.

Other organizations distributed the reports “in person” or held a “meeting with HR and manager to review report, [then had the] report emailed to feedback recipient after the meeting.” figure 24

How is the 360 feedback report distributed to the participant?

Electronically (report sent via email or obtained online)

43%

57%

28% 27%

Given to participant by a facilitator or coach

6%

Given to participant‘s manager

20%

2009

2% 4%

Workshop setting 0%

10%

2013 20%

30%

40%

50%

60%

3 6 0 F e ed b a c k Be n c h m a r k R e port

23

decision point

#18: What type of development support should be available to feedback recipients?

One of the greatest sources of variation found in this study was the extent and variety of support offered to feedback recipients once they receive their report. Organizations have options ranging from simply sending the reports to the feedback recipient and leaving the professional development entirely to him/her or, at the other end of the continuum, providing structured coaching and scheduled development activities throughout the year.

As with previous studies, the majority of organizations (65%) still require feedback recipients to work with their manager and/or offer individual coaching (60%) to help the feedback recipient digest the report and develop a professional development plan. In Figure 25, it is clear that since 2009 those two types of development support are the most common. While workshops, action planning (both live and online), and resource libraries are also relatively commonplace, very few organizations offer no development support to 360 degree feedback recipients.

Companies that require feedback recipients to complete development plans reported only an 80% compliance level. Coaching was far more commonly adopted by feedback recipients. Companies that offer coaching to feedback recipients indicated that an average of 90% of those who were offered this type of support actually received the coaching. figure 25

What, if any, development support is available to participants?

69% 65%

Develop plan with manager

65% 60%

Individual coaching

39% 41%

Action planning

18%

Workshops/Training – live Workshops/Training – online

18%

Resource library

2%

None (report only) 0%

24

3D Gro u p

37%

12%

32% 26%

2009 2013

7%

10%

20%

30%

40%

50%

60%

70%

80%

360 Degree Feedback Coaching Over time, we have consistently seen the prevalence of coaching as part of the 360 degree feedback program. Due to the popularity of coaching, we included a section to investigate how organizations typically use coaching with 360 degree feedback. We were particularly interested in how coaches are trained and certified, how coaching is typically delivered and whether coaches tend to be outside consultants or internally sourced.

decision point

#19: What type of training or certification should 360 degree feedback coaches have?

Coaches must be formally certified on many off-the-shelf 360 degree feedback instruments, particularly when the reports are complex or the leadership model being used is highly nuanced.

Professionals also have available many coaching institutes that offer formal certification in executive coaching, most of which include at least some coverage of assessments such as 360 feedback.

Despite the micro-industry around certifying coaches, only 30% of companies use coaches that are formally certified on the instrument. In lieu of formal certification, half (49%) of 360 degree coaches have received some training for a specific 360 degree feedback instrument. A sizable number of companies (21%) use coaches for their 360 feedback debriefs that are not trained on the instrument. figure 26

What type of training on the instrument did your coaches receive?

Formally certified

30% 49% Trained on the

No training on the instrument

decision point

instrument

21%

#20: Should the 360 degree feedback coaches be internal staff or hired externally?

The decision to use internal or external coaches is one that can involve consideration of a program budget, competencies of internal staff and the importance of confidentiality regarding development activities. While internal coaching can limit costs, external coaching can provide real and perceived objectivity, authority, and confidentiality to a 360 degree feedback recipient. Also, HR departments that do not carry full-time Organizational Development (OD) professionals or coaches on staff often find they do not have the right skill-sets internally to provide high quality coaching to senior executives. Nonetheless, 69% of organizations use internal coaches, such as managers, HR professionals or training and development specialists, and 31% of organizations hire external coaches for 360 degree feedback coaching.

3 6 0 F e ed b a c k Be n c h m a r k R e port

25

decision point

#21: How should the coaching be delivered to feedback recipients?

Due to the sensitive nature of the conversations, many feedback recipients assume that coaching will be conducted during an in-person meeting; however, this kind of conversation can also be conducted remotely and doing so can have some benefits (Miller & Rose, 2008). Even with the improvements in video conferencing capabilities, online coaching is still not as popular as traditional in-person coaching or telephone coaching. The majority of organizations (65%) hold 360 degree feedback coaching in person, while a third of organizations deliver coaching by phone (see Figure 26). figure 26

Is your coaching delivered: Online Video Conference (such as Skype)

6%

By phone

26

3D Gro u p

29%

65%

In person

Using 360 Degree Feedback Results One of the most controversial topics around 360 degree feedback is the appropriate way for organizations to use the feedback results. Countless articles and books have been written to discuss the main question in this debate. Should feedback be tied to compensation and promotion, or should it be used purely for development purposes? Is there some way to ensure that the 360 degree feedback leads to quantifiable results without compromising the quality and honesty of the feedback?

decision point

#22: Should the 360 degree feedback data be used for development purposes, administrative purposes or a combination of the two?

Advocates for the use of feedback results exclusively for developmental purposes argue that ratings become tainted by bias if raters know the results will affect the recipient’s position or benefits in the organization. For example, an employee rating a co-worker who they feel deserves a raise or a promotion is likely to inflate their ratings if they know the rating will directly impact his or her chances of getting the increase. On the other hand, raters may be more apt to respond honestly if they know the feedback is intended to solely aid the feedback recipient’s professional development. Considerable research on the distortion of scores in these situations has been published over the years (Klimoski and Inks, 1990; London, Wohlers and Gallagher 1990; Kozlowski, Chao and Morrison, 1998).

Proponents of using results to influence administrative decisions feel that, without recognizable consequences for failing to act on results, feedback recipients can easily ignore undesirable results.

Therefore, if feedback recipients are not held accountable for making behavioral changes based on their feedback results, 360 degree feedback will not have an impact on performance and the bottom line. Dave Bracken (Bracken, 2012) has also argued that organizations lose considerable value if they don’t use 360 feedback results to make decisions about leaders.

With our newly re-written question about the use of 360 results, we found 37% of companies use the 360 degree feedback results for self-directed development decisions solely, where the feedback recipient owns the results and is independently responsible for making use of the feedback results (Figure 27). Another 32% of organizations use their 360 degree feedback for development planning decisions such as identifying training and development needs, annual development goals and succession planning. A very small percent of organizations (2%) use 360 degree feedback results only for personnel decisions such as pay, promotions and hiring. 29% of organizations use 360 degree feedback results for a combination of development and personnel decisions. figure 27

Which ONE of the following best describes the type of decisions that 360 results are used to make?

Both development and personnel decisions

29%

37%

Personnel decisions

Self-directed development decisions only

2%

Development planning decisions

32%

3 6 0 F e ed b a c k Be n c h m a r k R e port

27

When asked to go into more specifics about the use of 360 degree feedback results in their organizations, we found a tremendous range of answers from “motivation” to “termination.” Figure 28 details more specifically how 360 degree feedback is typically used, but it is important to note that development and career development have been the most popular use of 360 degree feedback results over time. Very few organizations directly link 360 degree feedback results to pay and promotion. A few more organizations indirectly link 360 results to pay and promotion but the practice is still relatively uncommon. figure 28

More specifically, how is the feedback used?

70%

Career development

60%

Development

47%

Performance management

36%

Link to training/curriculum planning

35%

Succession planning

27%

High potential identification Indirect link to promotion

8%

Indirect link to pay increases

8%

Direct link to pay increases

4%

Direct link to promotion

2% 0%

10%

20%

table 4

2004

2009

2013

Career development

69%

20%

70%

Development

72%

92%

60%

Performance management

43%

29%

47%

Link to training/curriculum planning

53%

8%

36%

Succession planning

46%

16%

35%

High potential identification

35%

20%

27%

Indirect link to promotion

27%

14%

8%

Indirect link to pay increases 27%

14%

8%

Direct link to pay increases

4%

16%

4%

Direct link to promotion

8%

8%

2%

28

3D Gro u p

30%

40%

50%

60%

70%

Attitudes about 360 Degree Feedback Having the trust and support of everyone involved (upper management, raters, and feedback recipients, etc.) makes a tremendous difference in the ease and effectiveness of a 360 degree feedback program. A high level of support gives credence to 360 degree feedback as a value-added business tool worth the investment. When feedback recipients trust the results of the 360 degree feedback process, they are more likely to make demonstrable efforts toward achieving their development goals.

Overall, upper management tends to be very supportive of 360 degree

71% of companies reported strong senior management support for their 360 degree feedback process.

feedback. In 2009, senior leadership in 89% of companies was supportive or very supportive of their 360 degree feedback process. In 2013, 71% of organizations reported they had supportive or very supportive upper management; however, 21% of respondents felt senior leadership was neither supportive nor unsupportive of the 360 degree feedback program.

Another, perhaps even more critical question about attitudes toward 360 degree feedback is whether feedback recipients trust the 360 process. We found the majority (55%) of feedback recipients had “complete trust in the 360 process,” although 15% of feedback recipients did not have trust in the process. This trust is critical in getting feedback recipients to respond to their feedback and can be fostered by 360 degree feedback practitioners through preparation, communication, confidentiality and objectivity.

Looking Back

Table 5

At the end of every survey, 360 program managers

What is one thing about the 360 process you wish you had done differently had you known better?

were asked if, in retrospect, there was anything

different they would have done with their 360 degree

2013

Satisfied – No Changes

24%

(24%) was “nothing,” and that they were completely

Increased Follow-up

13%

satisfied with their current 360 program (see Table

Change Survey

11%

5). The second-most popular response (13%) had to

More Training

10%

feedback program. The most common response

do with more or different follow-up. This included comments like, “ongoing management follow-up

9%

Administration and Improved Planning

with individual’s development plan,” and “better

Expanded Number of Feedback Recipients 6%

follow-up with development plans on a regular

Added or Improved Coaching

4%

basis.” 11% of respondents said they would have

More Upper Management Support

3%

Improved Communication

3%

respondents said they would provide “more training

Different Instrument

3%

to raters on how to deliver effective feedback,” or

Added Norms

3%

“better training on the process for all involved.”

Improved Rater Selection

3%

More Confidentiality

2%

Better Comment Processing

2%

Different Reports

2%

Better Timing

2%

made changes to the survey content. Training was also an important theme in the comments. Many

3 6 0 F e ed b a c k Be n c h m a r k R e port

29

Looking Forward Organizations appear optimistic about maintaining or expanding their 360 degree feedback programs in the future. Figure 29 illustrates that 74% of organizations expect to either keep their program the same or expand it to include more feedback recipients in the future. This is a positive sign that organizations value 360 degree feedback as a critical component of their development initiatives for employees. figure 29

In the coming year, what portion of participants will go through the 360 process compared to this year?

60% 50%

2009

56%

40%

2013

44%

30%

32%

30%

20% 10% 0%

The Same

More

6% 12%

6% 14%

Fewer

Don’t know

So how important are specific 360 design features to companies? The most critical aspects of a 360 degree feedback process is anonymity of raters (52%) and a report that is easy to read and interpret (51%). Another essential component of a powerful 360 is ensuring that the feedback recipient work with their supervisor to discuss the results (see Figure 30). Less essential, but still important, is coaching and a formal orientation for feedback recipients. Although many practitioners cited it as something they would like to add to their program in the future, formal orientation for raters is considered the least important component of their 360 feedback program. figure 30 100%

14%

90% 80% 70%

52%

39%

51%

41%

Essential

28%

Somewhat Important

35%

60%

39%

50% 40% 30% 20% 10% 0%

30

3D Gro u p

24%

38%

42%

36%

43% 17% 7% Anonymity of raters

27% 18%

16% —3% Coaching

6% Easy to read report

—3% Participant meeting boss to discuss results

15% Formal program orientation for raters

Important

6% Formal program orientation for participants

Not Important

Conclusion Overall, organizations are very satisfied with their 360 degree feedback programs, and senior leadership at these organizations has shown very strong support for the 360 process. As a result, most companies plan to retain or expand their current program in the coming year.

360 degree feedback continues to be used primarily as a development tool on an annual basis, although use for developmental decision making with more involvement from HR and a feedback recipients’ direct supervisor seems to be increasing. For instance, 360s are commonly used for training needs analysis, succession planning and identifying high potential employees.

Typically used for managers and executives, 360 degree feedback programs are often varied, based on feedback recipients’ level within the organization. For instance, follow-up coaching may be a ubiquitous part of the process with senior leaders, but is less common with individual contributors.

While many organizations do at least some aspects of the survey development in house, the majority of organizations engage the help of a consultant for at least a part of the process. It appears that surveys are getting shorter, with the majority of organizations using surveys that are no longer than 40 questions, but they almost always include at least one open-ended question. The 5-point Likert scale seems to have become a standard; however, the rating anchors are still quite varied.

Raters are most often selected by the feedback recipient with manager approval. Organizations are investing more heavily in orienting employees toward the program processes and goals, but online tools seem to be less common than face-to-face introductions.

When it comes to handling responses to the survey, organizations tend to use an outside vendor for scoring the results and generating reports because this provides a strong sense of objectivity and fairness.

It is clear that follow-up support for feedback participants and rater anonymity are among the most highly valued aspects of 360 degree feedback processes. While increased follow-up was commonly cited as an area for improvement, most organizations currently provide coaching support to feedback recipients and/or require specific action planning that involves the direct manager. Coaching is most often done internally by someone who is trained on the 360 degree instrument, but formal certification of coaches seems not to be the norm.

While 360 degree feedback continues to enjoy widespread popularity in companies of all sizes, both practitioners and researchers still have ample opportunity to innovate and improve program designs. Many of the current practices documented in this study align with well-researched “best practices.” Nonetheless, there remain areas where current practice contradicts research. This gap leaves an opportunity for researchers to find solutions that are more practical for companies to implement, but it also invites practitioners to continue finding ways to implement features that researchers have proven to be successful.

The most comprehensive conclusion from this study is that, as in the past, no two 360 feedback processes look alike. This diversity of design reflects the fact that each of the 211 companies surveyed designed their 360 process to fit the unique constraints, resources, and goals of their company. Despite the wide variability in design choices made by practitioners, 360 degree feedback in all of its forms continues to be one of the most valuable tools available to support excellence in leadership at companies across North America.

3 6 0 F e ed b a c k Be n c h m a r k R e port

31

Research Methods The survey questions were carefully designed by the 3D Group research team, which utilized many questions from previous surveys but also added and refined many questions for this current survey.

The data were collected using a wide distribution of electronic surveys. The survey consisted of 52 questions, including 8 open-ended questions. The possible response options varied from 5-point Likert rating scales, to multiple-responses sets (“select all that apply”), to single response sets (“select only one”). The mean time to complete the survey was 13.5 minutes and a standard deviation of 11.6 minutes. The questions represented in Figures 4, 7, 10, 13, 14, 18, 21, 23, 25, 26, and 28 were presented with the option to select all answers that apply.

Many of the open-ended questions asked for numerical answers. These numerical answers were analyzed using basic descriptive statistics. Open-ended responses were categorized and coded according to common themes. One round of initial content coding was conducted to group common responses. Then the codes were checked and recoded to ensure accuracy and relevance to the assigned categories.

Past participants and HR professionals throughout North America were solicited to participate in the study. Participation was incentivized in order to increase sample size. To control for unfounded responses, researchers developed rigorous validation strategies and only provided incentives to participants providing legitimate responses. The participant recruitment took place between July and November, 2012.

Refe rence s 3D Group (2009). Current Practices in 360-Degree Feedback: A Benchmark Study of North American Companies. 3D Group Technical Report #8326. Berkeley, CA: Data Driven Decisions, Inc. 3D Group (2004). Current Practices in 360-Degree Feedback: Overall Findings. 3D Group Technical Report #8285. Berkeley, CA: Data Driven Decisions, Inc. 3D Group (2002). Benchmark Study of North American 360-Degree Feedback Practices: 2002 Study. 3D Group Technical Report #8214.

Berkeley, CA: Data Driven Decisions, Inc. Bracken, D.W. (2012). http://dwbracken.wordpress.com/ 2012/04/15/what-is-a-decision/ Bracken, D.W., & Rose, D.S. (2011) When Does 360-Degree Feedback Create Behavior Change? And How Would We Know It When It Does? Journal of Business Psychology. 26(2), 183-192.

Dalessio, A., & Visilopoulos, N. (2000). Multi-source Feedback Reports: Content, formats and levels of analysis. In Bracken, D., Timmreck, C., & Church, A. (Eds.), The Handbook of Multi-source Feedback. San Francisco; Jossey-Bass.

English, A., Rose, D.S., & McLellan, J. (2009) Rating Scale Label Effects on Leniency Bias in 360-degree Feedback. Paper presented at the 24th Annual Meeting of the Society for Industrial Organizational Psychology. New Orleans, LA. Klimoski, R., & Inks, L. (1990). Accountability forces in performance appraisal. Organizational Behavior and Human Decision Process, (45)194-208.

32

3D Gro u p

Kozlowski, S.W.J., Chao, G.T., & Morrison R.F. (1998). Games raters play: Politics, strategies, and impression management in performance appraisal. In J.W. Smither (Ed.), Performance Appraisal: State of the Art in Practice. San Francisco, Jossey-Bass.

London, M., Wohlers, A.J., & Gallagher, P. (1990). 360-degree feedback surveys: A source of feedback to guide management development. Journal of Management Development, 9, 17-31.

Miller, B.E., & Rose, D.S. (2008). Remote Coaching: An Effective Alternative to Face-to-Face Coaching. Technical Report #8325.

Berkeley, CA: Data Driven Decisions, Inc. Rose, D.S. (2002) The Use and Abuse of Comments in 360-Degree Feedback. Paper Presented at the 17th Annual Convention of the Society for Industrial and Organizational Psychology, Toronto, Ontario.

Rose, D.S., Farrell, T., & Robinson, G.N. (2004). Are Narrative Comments in 360-Degree Feedback Useful or Useless? Technical Report #8253. Berkeley, CA: Data Driven Decisions, Inc. Timmreck, C.W., & Wentworth, T. (2000). Working with a Vendor for a Successful Project. In Bracken, D., Timmreck, C., & Church, A. (Eds.), The Handbook of Multi-source Feedback. San Francisco; Jossey-Bass.

Yukl, G., & Lepsinger, R. (1995). How to get the most out of 360 degree feedback. Training, 32, 45-50.

About 3D Group 3D Group is dedicated to helping businesses of all sizes use human resource assessments to increase the effectiveness of their people and programs. Our expert consultants provide high-impact, full service assessment solutions for 360 degree feedback, leadership development and employee engagement surveys. 3D Group researchers are widely published and regularly present findings to national audiences of business leaders and Human Resources professionals.

For more information about participating in upcoming studies or to learn more about our scientifically validated easy-to-use 360 degree feedback and employee engagement tools, contact 3D Group at 510.463.0333 or visit our website at www.3dgroup.net.

ISBN-13: 978-0615748351 (3D Group) ISBN-10: 061574835X