Feedback in Clinical Education

“Zij zullen trachten in het zweet des aanschijns ’t Onzegbaar zuivere nog eens te noemen En niets dan ’t schaduwrijk der beelden vinden.” (p.56) Uit: Bertus Aafjes, In den Beginne (1949).

Maar tot den mens heeft Hij gezegd: Zie, f 66 de vreze des Heeren is de wijsheid, en van het kwade te wijken is het verstand. Job 28:28.

f Ps. 111:10. Spr. 1:7; 9:10. 66 Alsof Hij zeide: De verborgen wijsheid, waardoor Ik de wereld en daarin de mensen regeer, is voor Mij; de wijsheid die Ik voor de mensen behouden heb, is, dat zij Mij vrezen naar het voorschrift van Mijn geopenbaarde wil. Zie Deut.

29 op vers 29.

Illustrations:

Sjaakkaashoek.tumblr.com, Nijmegen “The pattern on the front and back cover drawn in the border is based on the feedback model (figure 1.2) used in this book.

On the front cover I have made a tree structure of arteries and veins. On the back cover you see a representation of the human circulatory system with the heart in the circle, which is a feedback system that provides gas exchange in lungs and cells.

The small pictures at the top and bottom on the back refer to theory and practice, the books organized as a statistical graph and the pile of books. The small pictures on the left and right refer to feedback and communication situations in the clinical setting, feedback in the operation theater and communication at the bedside. I was inspired by “The Book of Kells”. Work of monks in monasteries and scientific research perhaps have some similarities.” Sjaak Kaashoek

Design:

Smart Printing Solutions, Gouda

Press:

Smart Printing Solutions, Gouda

ISBN:

978-90-393-6372-0

Copyright:

© J.M.M. van de Ridder, 2015

Feedback in clinical education Feedback in het klinisch onderwijs (met een samenvatting in het Nederlands)

Proefschrift ter verkrijging van de graad van doctor aan de Universiteit Utrecht op gezag van de rector magnificus, prof. dr. G.J. van der Zwaan, ingevolge het besluit van het college voor promoties in het openbaar te verdedigen op woensdag 1 juli 2015 des middags te 4.15 uur

door

Johanna Maria Monica van de Ridder geboren op 22 juli 1973 te Amersfoort

Promotoren: Prof. dr. Th.J. ten Cate Prof. dr. K.M. Stokking Prof. dr. W.C. McGaghie

Financial support of printing this thesis is gratefully acknowledged and was provided by the Center for Research and Development of Education University Medical Center Utrecht Utrecht, Communicatie In Gezondheidszorg CIGnificant De Bilt, Department of Education Albert Schweitzer hospital Dordrecht, Het Van Weel-Bethesda Ziekenhuis Dirksland, Nedermeijer Office Dordrecht, Stichting Appraisal & Assessment Sittard, Trainingacteurs & Bedrijfstheater Kapok Amsterdam, and Witvoet Auto Parts Lansing (IL) USA.

Opgedragen aan mijn dierbare vader en moeder Otto van de Ridder en IJke Maria van de Ridder-de Jong

Voorwoord Ziekenhuizen hebben een prominente plaats in mijn leven. Mijn wieg kwam letterlijk uit het Diaconessenhuis aan de Prinsenlaan in Dordrecht. In Dirksland groeide ik op naast het Van Weel-Bethesda Ziekenhuis, in een tijd dat je als kind nog met de ambulance mee mocht op zaterdag om een patiënt naar huis te brengen, met als hoogtepunt het rijden met zwaailicht en sirene. Ook onderwijs had al vroeg mijn hart; vanaf de lagere school wilde ik ‘juf worden’. Mijn interesse voor taal werd tijdens mijn MAVO- en HAVO-tijd verder aangewakkerd door het onderwijs in het vak Nederlands. Tijdens mijn studie Onderwijskunde ontdekte ik –mede door Karel Stokking- hoe spannend en uitdagend onderzoek doen is. In dit proefschrift ‘Feedback in Clinical Education’ komt alles bij elkaar: onderzoek doen in de medische context, naar het effect van taal, met als doel het leren te bevorderen.

De afgelopen jaren heb ik in veel feedbackdialogen geparticipeerd. Wat was het soms ingewikkeld om feedback te geven, feedback te krijgen, en met ontvangen feedback goed om te gaan. De theorie over feedback heeft me geholpen deze complexe interacties te doorgronden. Eén rode draad liep door al die processen heen: de intentie van de feedbackgevers om mij naar het eind van mijn promotietraject te krijgen. Naast het bedanken van mijn promotoren, collega’s, vrienden en familie, past het mij bovenal de Heere te danken voor de gezondheid, de lust en de kracht die Hij gaf om aan dit proefschrift te werken.

Olle, Karel and Bill, I respect the three of you for the large amount of patience you had with me. Olle, ik ben blij dat je me ooit heb gezegd dat je wel hield van mensen die zelfstandig konden denken. Ik voelde me dan minder bezwaard als ik soms al te zelfstandig dacht. ’t Was niet altijd makkelijk twee ‘stijfhoofdigen’ bij elkaar. Olle, ik bewonder je creativiteit en passie; ik heb veel ván en dóór je begeleiding geleerd. Karel, jouw preciesheid, grondigheid en kritische blik waardeer ik; ik ben er jaloers op. Jij hebt me geleerd om feedback -waar nodig- te relativeren. Ik dank je voor de wijze waarop je mij gevormd hebt! Bill, you are a role model for me when it comes to mentoring people new in the medical education field: Mrs. Fenstemacher, Lessons in Clarity and Grace, sharing ideas, discussing our research, I still can learn a lot from you. Thank you for being my mentor.

I wish to acknowledge Bob Gundlach for the insights in the writing process he has offered me. Bob, I’ll try to pass on the knowledge to others who struggle with writing.

Without the many hours spend in the Galter Library with Linda O’Dwyer teaching me how to search the literature systematically, the studies in this thesis would have been of lower quality. Linda, thank you for teaching me such valuable skills.

De heren Heerschap, Moeilijker en Slootweg leerden mij meer dan alleen Nederlands, Engels en Geschiedenis op de Prins Mauritsschool: u legde de basis voor verdere academische vorming, en u ging een band met uw leerlingen aan. Die lijn probeer ik door te trekken in mijn eigen onderwijs.

Joanna Freeke, Karin Schroten en Hanneke de Jong: ik dank jullie voor respectievelijk het corrigeren van de introductie en de conclusie, het dankwoord en de Nederlandse samenvatting. Willem van de Ridder, warme dank voor je steun bij het corrigeren en prettig dat je meedacht over de lay-out.

Sjaak Kaashoek, ik waardeer het erg dat je de mooie omslag van mijn proefschrift gemaakt hebt. Onze gesprekken over het verbeelden in woord en beeld hebben mij meer inzicht gegeven in de creatieve processen die zowel aan wetenschap als beeldende

kunst ten grondslag liggen. Dank voor je vriendschap.

De (oud)leden van de vakgroep Onderwijskunde, dank ik voor het meeleven tijdens mijn onderzoek. Mijn collega’s van het UMCU bedank ik voor de goede tijd; ik heb me bij jullie echt thuis gevoeld! Eugène Custers, Tom Fick, Marijke Sterman, Fred Tromp, Marjo Wijnen-Meijer en Marian Wolters, kamergenoten met wie ik langer dan een jaar een kamer deelde in het UMCU, dank ik voor de gezelligheid, het meeleven, de adviezen en... de tikken op de ruit.

My colleagues from the Feinberg School of Medicine have offered me a very valuable time in Chicago.

In het Albert Schweitzer ziekenhuis heb ik geleerd de feedbacktheorie pas echt naar de praktijk te vertalen: opleiders, arts-assistenten, Leerhuiscollega’s dank voor jullie kritische vragen, reflecties, belangstelling en meeleven.

Peter Plaisier, je bent het voorbeeld van een opleider met visie, die interesse heeft voor en betrokken wil zijn bij medisch-onderwijskundig (feedback) onderzoek; het was inspirerend om met je samen te werken! Rob Oostenbroek -ziekenhuismaatje, én andere helft van het onderwijsduo- het was me een waar genoegen om je in te leiden in de praktijk van het medisch onderwijs, dank voor het meedenken, de goede congressen en vooral voor je steun!

Tegen de Master- en Bachelorstudenten Onderwijskunde, te weten Ron Brendel, Claudia Peters, Gerdine Torn, Linda van Woerden, Ciska Berk, Lotte Govers, Atie van Putten, Joy de Vries, Cristel Wieman, Margo Habraken, Cecile Collast, Maaike Wynia, Marjan Hoitinga, Eveline Huiden, Janneke van Meteren, Ernst-Jan van Unen, Ellen Traas, Nicolette Guichelaar, Maarten Molendijk, Ditty van der Kroon, Britt van Hoek, Monica de Visser, Jorinde Beerens, Hafsa Zariouh en Jantine Heida, wil ik zeggen dat het een voorrecht was om met jullie verder over feedback na te denken, en om jullie te mogen begeleiden tijdens stages en afstudeeronderzoeken. Jullie feedback hielp mij om mijn begeleiderskwaliteiten verder te ontwikkelen.

Frans Grosfeld noem ik, omdat hij het was die me in de praktijk leerde om van wetenschap te genieten. Vandaar dat ik jou als paranimf naast me wil hebben.

Dé onderwijskundemeiden, de leesclub, en mijn vrienden wil ik bedanken voor de ontspanning: de gezellige maaltijden, de gesprekken over respectievelijk ons vak, de boeken en het leven.

Familie Van der Meij -Bas’ kamer, Nieks warmtematje-, oom IJp en tante Jannie -gezond eten, het bed-, Hanneke en Jos -de wijntjes, de kaa(r)sjes, de conversaties-, dank voor jullie hartelijkheid en niet aflatende interesse.

Jantine, Hendrik, Willemijn en Willem, dank voor jullie meeleven, dat jullie elk op jullie eigen, kenmerkende wijze toonden.

Pa en Ma, ik dank jullie voor de opvoeding die ik ontving. Jullie brachten mij groot met het NRC Handelsblad en het Reformatorisch Dagblad en leerden mij al jong de waarde van het woord kritisch te bezien. Ik ben vooral dankbaar dat ik opgroeide met Gods Woord en dat jullie me erop wezen -voor tijd en eeuwigheid- alles van de Heere te verwachten. ‘Indien iemand van u wijsheid ontbreekt, dat hij ze van God begere, Die een iegelijk mildelijk geeft en niet verwijt; en zij zal hem gegeven worden’ (Jakobus 1:5).

J. M. Monica van de Ridder Zeist, april 2015

Table of contents

Voorwoord Chapter 1

General introduction

Chapter 2

What is feedback in clinical education?

47

Chapter 3

Variables that affect the process and outcome of feedback, relevant for medical training: a meta-review

61

Chapter 4

Measuring trainee Perception of the value of Feedback in Clinical Settings (P-FiCS)

83

Chapter 5

Framing of feedback impact’s students’ satisfaction, self-efficacy and performance

103

Chapter 6

Feedback providers’ credibility impacts students’ satisfaction with feedback and delayed performance

119

Chapter 7

Conclusion and discussion

133

Appendices

163

Samenvatting

206

Short biography

11

213

General introduction

Chapter 1

12

1.1 Oral feedback in clinical education uring morning rounds in the surgery department a resident was visiting patients with a senior clerk and a junior clerk.

They visited a patient who had undergone an operation on the blood vessels of his leg the day before. The resident asked the senior clerk to perform a physical examination on the patient’s leg. The clerk put his fingers on the patient’s feet to feel pulses. He asked some questions and he touched the patient’s toes to feel the temperature. The resident observed the student and asked several questions. “What kind of information did you retrieve from feeling the temperature of the toes?” He suggested to him to maintain regular eye-contact with the patient while performing the examination. The resident also instructed the senior clerk to put his fingers at a different point on the feet to feel the pulse better. The junior clerk observed the situation.

Feedback context This example describes a feedback situation in a clinical setting. Ende wrote an influential article in 1983 about feedback in clinical medicine. He describes feedback as ‘information that a system uses to make adjustments to reaching a goal’ (p.777).1 Other descriptions of feedback are: ‘In technology, [the] process by which an electronic or mechanical control system monitors and regulates itself. […] Feedback works by returning part of the ‘output’ of the system to its ‘input’.2 The literature distinguishes several sources that provide feedback: the ‘self’, the task and others.2-3 Feedback from the self is the inner voice that reflects or critiques our actions, and corrects or ‘encourages’.3-5 Persons receive ‘task feedback’ when it becomes clear that certain tasks work or do not work, or when they work with an apparatus that ‘provides’ feedback.3-4 The third source is the people around us that provide feedback.

Feedback providers in the clinical setting have a variety of backgrounds: physicians, residents, and medical students.6-9 For multisource feedback co-workers -such as paramedics, nurses and psychologists- and patients are invited to give feedback.10-11 In specific training situations standardized patients also give feedback.12-13 The feedback recipients in the clinical setting are those in training: clerks, residents, interns, nurses or paramedics.9 In the lifelong learning tradition feedback is not only for those who are in training but also for colleagues and peers.14-17 In the medical clinical setting feedback can be provided on several occasions: on the ward during patient hand over, morning report, rounds, operating room, clinical presentations, clinical teaching, or in the outpatient clinic. Focus of attention are the different aspects of clinical consultation such as communication skills,18-19 or physical examination.20-22 Feedback can be given on the competencies defined by the Canadian Medical Education Directions for Specialists (CanMEDS) or by the US Accreditation Council for Graduate Medical Education (ACGME).23-24 Feedback can be used in a learning situation to improve a learners’ performance, attitude, or knowledge. When feedback is given between colleagues, ‘peer feedback’, the goal is to determine gaps in knowledge and performance, to improve work situations and interpersonal collaboration or interprofessional relations.14-17 There is variation in time between the performance of a task and giving feedback. Immediate feedback is given right after the performance, but at other times it may take time before the feed-

Focus In the three empirical studies (chapter 4, 5 and 6) the feedback is given by ‘others’: (simulated) clinicians. The feedback receivers are medical students and residents. The feedback setting is a (simulated) clinical environment. The purpose of the feedback is to improve the performance of a learner. The feedback message consists of verbal information which is provided orally. The content of the feedback is the learner’s performance of clinical activities. The feedback is directed towards the task and based on a checklist or spontaneous feedback situations. The effectiveness of feedback is measured in students perceptions, (self-efficacy, satisfaction, ideas about effective feedback) and performance of a physical examination task.

Relevance of effective feedback Humans reflect on their actions and critique themselves with the purpose to improve, encourage or correct their actions. Although we reflect or self-assess our actions, blind spots defy our perception. Humans are not good at self-assessment.35-38 Self-assessment is not the best method to improve performance. Keil summarizes this problem adequately (as cited in Eva, 2004) ‘How can I know what I don’t know if I don’t know what I don’t know?’39 People around us see our blind spots better than we do, and their feedback can be helpful to improve weaknesses we may not notice.

Task feedback teaches learners how a task can be performed. However, task feedback alone does not give explanations or suggestions about how to improve performance.

Human feedback can identify blind spots and explain and give suggestions. This is acknowledged by both feedback providers and feedback recipients in clinical education.

Feedback is an important learning and teaching tool in medical education with high educational impact.20-22, 26, 40-45 Two important factors constrain giving feedback: (a) lack of protected time to super-

13 General introduction

back is communicated, this is the delayed feedback situation.25 Feedback can be provided based on structured observations. Several forms and tools are used to elicit feedback, such as Mini Clinical Evaluation Exercise (Mini-CEX), Objective Structured Assessment of Technical Skills (OSATS), Direct Observations of Procedural Skills (DOPS), and feedback encounter cards.26-30 Detailed descriptions of these instruments are given in section 1.4. The instruments give the feedback provider specific observation points and make the observation focused. Often these feedback sessions are very explicit: both feedback provider and recipient are aware of it. In contrast, feedback is also given in unplanned, spontaneous occasions based on a ‘implicit’ observation.5 In these situations the feedback provider may use his/her own standards when providing feedback. Sometime feedback recipients are not aware of the feedback dialogue, which leads to an implicit feedback session.

The initiative for feedback can be taken by either the learner or the supervisor. When supervisors observe aspects of a learner’s performance which need to be improved they can decide to give the learner feedback. In certain situations the learner decides which topics (s)he would like to receive feedback and asks for it; the learner actively seeks feedback.31-33 Feedback content can be directed to the self, the task motivation and task learning.4 The verbal feedback message can be provided orally, in writing and using a combination of both. Feedback communication varies in time between several minutes and half an hour.5, 34

Chapter 1

14

vise, observe and give feedback,9, 46-49 and (b) lack of provider feedback skills. These factors affect feedback provided by ‘others’ in the clinical environment.21, 50-54 In the dynamic clinical environment, with time pressure, a heavy workload, fast decisionmaking, etc., workers face risks and emotions and have important responsibilities. To perform well in such an environment workers need to be competent and know their strengths and limitations. However, there is not much protected time to educate learners,55 so the available time has to be used effectively. If feedback is given well it can be effective and improve learner performance. Staff members and residents are trained for patient care, but not always in teaching.21, 52-53 If effective feedback is given to learners in medical education, patients, learners, and staff will profit.

When clerks and residents receive feedback about what went well, they learn which behaviour is correct and why it is correct. In this way the learner is reinforced to continue good practice. When points for improvement are discussed learners will discover what could be changed, why, and how. So, the learner gains more knowledge, skills and a professional attitude. When learning points are made explicitly, it is easier for the learner to set goals and to monitor his/her learning process.56-58 In comparison to learning by trial and error, learning with feedback can reduce the errors made in daily practice, save costs and also improve the pace in which learners learn compared to learning without feedback.

Well trained residents and clerks are an asset to the staff and co-workers. It is rewarding for feedback providers when they notice clerks and residents apply the received feedback and that they are willing to learn. This motivates staff in supervising the learners.

On the other hand, learners also like it when they feel staff are willing to teach and supervise them. In this case feedback functions as a motivator. When learners are able to apply the received feedback they get to know the habits, procedures and knowledge they need for working effectively. They can make a beneficial contribution to collaboration with their colleagues. This can make daily health care processes more efficient and this reduces workload.

Most importantly, these advantages should also lead to avoiding risk and errors, and improvement of the quality of patient care.

1.2 Purpose The relevance of feedback for students, staff and patients is clear because staff and students view feedback as a powerful learning and teaching tool.20-22, 26, 40-45 This tool, however, is only effective when feedback has a positive effect and improves performance. This assumes that the feedback recipient understands the feedback message and is willing and able to apply the feedback to daily practice. However, little is known about what makes the feedback process and feedback results effective. This is surprising for three reasons: (a) Feedback research has a long history, dating back to the last quarter of the Nineteenth century. Due to its long history, feedback research is influenced by several paradigms: Functionalism, Behaviourism, Cognitivism and Situated Learning and Constructivism (1.3.1). (b) Feedback is studied in a variety of fields. Beside the research in medical education, feedback is also studied in other Teaching and Learning fields, in Labor and Manage-

1. Concept Clarification The term feedback is used in many ways due to a long scholarly history, different paradigms, and different fields in which feedback research has been performed.

Conceptual and operational clarity is needed for research to advance. Using different feedback concepts interchangeably affects research outcomes. The purpose of this study is to explain different feedback concepts and provide a conceptual and operational definition of feedback.

2. Connecting feedback research Bringing feedback research from various scientific disciplines together provides an overview of differences and similarities in feedback research methods. Integration of the feedback literature from various social scientific disciplines creates an overview of the current state of feedback research. This overview is helpful to identify disciplines that need more research attention. Integration gives insight about different research methodologies and approaches that are used in different settings and can also address a research agenda.

Connecting feedback research may contribute to theory building on feedback effects.

In general both teachers and learners perceive feedback as a valuable learning tool. The effects of feedback in general are not very high. More insight into the feedback process and the variables that influence the feedback effect might help us understand why the results of feedback are not as powerful as we expect and explain variability in feedback results.

3. Comparing contexts The medical context differs from the educational context in a classroom, a psychotherapeutic context or a labor and management context. The medical education context is characterized by its fast pace, the importance of safety and high impact decisions. Feedback effects in alcohol therapy or classroom contexts, cannot be generalized to the medical education context. Work environment and task nature affects the context. Ideally studies should be repeated to determine whether the same effects are found in a medical context. To determine whether or not variables used in research in the non-medical context are also effective in the medical context, we carried out two experiments with communication variables.

15 General introduction

ment, Therapy and Communication. These fields often have their own theoretical approaches and models (1.3.2). (c) Feedback receives much attention in medical education, both in research and in daily practice (1.4).

A closer look at the literature shows much diversity in feedback research and a lack of integration. Different paradigms across fields are a hindrance for effective integration.

Research on feedback is often isolated and connections between fields and theories are missing: ‘recent FI [Feedback Intervention MvdR] research is carried out by isolated investigators who share either a theoretical or a paradigmatic orientation.’(p. 254).4 The purpose of this thesis is to integrate feedback literature from various paradigms and fields (chapter 2 and 3), and to explore variables that affect the feedback process and effect (chapter 4, 5 and 6). The thesis has four goals.

Chapter 1

16

4. Understanding feedback effectiveness Insight into variables that shape the feedback process and the feedback effect helps to explain why feedback is not always as effective as expected or perceived or why it has a negative effect. These insights can be used to develop feedback guidelines which are based on evidence rather than on ‘gut feeling’. Insight about variables that influence the feedback process and feedback effect might help those who develop feedback guidelines or teach how to provide effective feedback.

This insight might give feedback providers directions for improving feedback practice.

It puts in perspective that it is not only the feedback provider’s communication ability or techniques which makes the feedback effective. It shows that it is not always due to lack of recipient’s attention or motivation that results in feedback being less effective.

Feedback recipients need not be passive. They can improve feedback by active questioning for clarification and examples. They will explain why the given feedback is not always perceived as helpful, and why it is sometimes hard to apply.

Chapters 2 and 3 have a theoretical focus. A short historical overview of feedback research followed by an explanation of the different feedback fields and an overview of the last eight years of feedback research in medical education are given to increase understanding of these theoretical chapters.

1.3 Historical overview The term feedback was introduced to the social sciences in the middle of the 20th century.59-60 However, feedback research dates back to 1880. Despite this long tradition of feedback research, many questions are still unanswered. An explanation for this is that feedback research was performed from different scientific paradigms. Within each paradigm feedback research had a different focus, varying from the outcome of feedback (behaviorism), the feedback process itself (cognitivism), and interaction in the environment (situated learning and constructivism).

1.3.1 Paradigms Functionalism In the last quarter of the Nineteenth century researchers became interested in the effect of punishments and rewards on human behavior, a precursor of feedback research.61-62 When functionalism was the dominant paradigm in psychology, research was characterized by introspection and experiments.63 Learning was seen as an interaction of learners with their environment. With this mind set, learning was equated with a change in environment and successful learners were able to apply their experience in the past to new situations.64 In 1913, E.L. Thorndike (1874-1949) an American psychologist formulated the Law of Effect which has been important in feedback research:4, 65 “When a modifiable connection between a situation and a response is made and is accompanied or followed by a satisfying state of affairs, that connections strength is increased. When made and accompanied or followed by an annoying state of affairs, its strength is decreased. (p.4)”.66 The satisfying state of affairs was seen as the reward and the annoying state of affairs as the punishment, and both influence behavior.

Dissatisfaction with the many different types of introspection used in research and

their often contradictory outcomes made a place for the behaviorist view around 1920.67

Cognitivism Theories and concepts from linguistics (Chomsky), information theory (Shannon), artificial intelligence and experimental psychology (Broadbent) were used to answer questions about cognitivism.64, 67 Attention in learning changed from getting the ‘right answer’ in the behaviorist view, to using the ‘right process’ in the cognitivist view. The goal of learning was to acquire ‘expert’ rules which would be more efficient than ‘novice’ rules in problem solving. The second implication is the attention paid to individual learner problem-solving processes. Each learner has his/her own problem representation and their own problem-solving strategy.64 Criticism about the cognitive approach focused on the idea that thinking was a separate stage between sensory input and motor response. Pragmatic problems like inequality in education due to linguistic and cultural minorities and social aspects of thinking and acting drew attention to the importance of the social environment in the 1970s.64 Situated Learning and Constructivism Situated Learning and Constructivism are two schools of thought that can be seen as a reaction to the cognitive tradition. Learning and teaching outside a context are not possible. Contexts include a classroom, ward, or at the bedside. Thus in studying education, context and task should be related. The interaction between the learning process and context is important.67 Situated learning underlines the fact that knowledge is obtained in an external, social world. Knowledge is a product of the activity and the culture in which it is developed and used. Authentic activities – i.e. meaningful and purposeful actions- are important for learning. Cognitive apprenticeship supports learning by enabling students to acquire, develop and use cognitive tools in authentic domain activity.68 The term constructivism involves several sets of beliefs about the nature of reality (epistemological); the nature of mind, learning and cognition (psychological); and about the best way to support learning (educational).69 Constructivism argues that

General introduction

Behaviorism Direct external observations and minimal ambiguity characterize the research of the behaviorists. Behavioral researchers aimed to legitimize psychology as a science. They tried to use methods similar to those used in the physical sciences.64 The behaviorist focused on the question: What is a person’s response (R), given a certain stimulus (S) in the environment, and how can these relationships be strengthened? A person’s behavior received attention, while the working of the mind was largely ignored.63-64, 67 Learners are seen as comparatively passive, while instructors giving stimuli know what learners need. The environment (S) also determines the outcome (R). In the behaviorist view learning is not education but training.64 Within the behaviorist paradigm the focus of feedback was mainly on the feedback outcome or feedback effect. How the feedback was received and processed by the learner was mostly ignored.

The Second World War gave practical questions - e.g. how to avoid loss of attention in soldiers - which could not be answered with a behaviorist approach. These questions were not related to learning a new task (behavior) but to higher order mental processes, such as human information processing capacities, problem solving strategies, focus and attention span.63-64, 67 The behaviorist tradition had to make place for a cognitive approach around 1950-1960.

17

Chapter 1

18

knowledge resides in an individual’s internal state, not always apparent to others.67 Humans experience a real world, but every individual will give a different meaning to this experience. Because meaning is rooted in an individual’s experience, there is not only one correct meaning. Knowledge can therefore only be constructed by the learner.70 The learning environment is important because it should provide tools for engaging in this knowledge construction process.71-72 Choice of paradigms These different paradigms do not replace each other, but complement each other. That is also reflected in the studies in this thesis. In the theoretical chapters (2 and 3) elements of all the cited paradigms are visible.

The empirical studies have cognitivist elements, such as attention paid to the learning process, especially to a learner’s own perception. In chapter 4 attention is paid to the context and the learning environment; this chapter has elements of both the cognitivist and the constructivist approach. This thesis explores feedback recipients’ perceptions about the learning environment, but also on some behaviorist elements, for example in the experimental studies where the effects of certain stimuli on student’s behavior were tested. The fourth and fifth study address both behavioral and cognitivist traditions.

1.3.2 Research fields in social science after 1940 In 1943 the term feedback was introduced in the social sciences.59-60 Within social science feedback research can be broadly categorized in four fields: Teaching and Learning, Labor and Management, Therapy, and Communication.

Teaching and Learning Many studies focus on student reactions - perceptions or behavior- toward feedback. A student’s perceptions of feedback includes which aspects students view as valuable,73 or which aspects they think promote learning.74 After receiving feedback the performance changes. Examples include experiments on the effectiveness of different types of feedback on psycho-motor performance tasks,75-77 feedback in language learning,78 or feedback about classroom problem-behavior.79 Another group of studies is focused on computerized feedback. This feedback is given in Computer Assisted Learning (CAL) or Computer Assisted Instruction (CAI). The feedback is often standardized and provided by a computer as smileys, positive words, or team points indicating correct behavior. This feedback is not interpersonal.80-81 A third category of studies deals with feedback for evaluation purposes, e.g. of a teacher’s teaching skills, study programs and curricula.82-83 Labor and Management Feedback studies in labor and management are generally focused on behavioral changes by employers and employees and on organization development.84 The performance appraisal system, a tool to evaluate employers and employees, receives a lot of attention.

Research is focused on the reliability and validity of different methods and instruments for performance appraisals,85 and the influence of demographics, such as cultural background and race on performance evaluations.86-87 Other topics of interest are: Which circumstances and conditions increase the effectiveness of feedback on performance? How is an employee’s attitude towards the performance appraisal system affected by ratings?88 What is the influence of feedback on the feedback provider’s own

Therapy Studies in therapy, for example in psychotherapy97-101 or in alcohol misuse102 stress the importance of feedback as a tool which can lead to successful therapeutic outcomes.

Therapy studies focus on the effects of different feedback methods such as how viewing a video with one’s own performance influences affective reactions,98 or how different feedback types - generic information versus targeted, personalized feedback - affect the process of health behavior change.99 Other studies focus on therapist behavior and therapeutic outcomes, when therapists receive client’s progress information as feedback.100-101 The relationship between feedback provider and recipient and feedback communication aspects, such as the valence of the message, and the influence of the relationship on the acceptance of the feedback are important topics in this field.97 Communication Feedback research in communication is about how messages are transmitted. The reaction of the listener to these messages is the feedback. Shortly after the Second World War questions were raised about the influence of propaganda and mass communication on people. This was followed later by studies on communication between individuals, for example on the communication of bad news.103-105 The communication field has historically paid less attention to feedback than expected.106 Within the last 15 years the number of publications in the field of communication about feedback has increased. Research in this field focuses on learners’ perceptions of communication. Examples are the learners’ perceptions of the communication climate in which feedback and critique are given,107 the relationship between communication and the perception of time,108 students’ perceptions of the feedback provider and the feedback message,109 students’ sensitivity to the wording of the feedback message,110-111 participants’ satisfaction with interpersonal communication,112 and how student communication competence and speaker apprehension is impacted by privately watching their own videotaped speeches.113 Feedback perceptions are not only used as a dependent variable but also as an independent variable.

There is a line of research concentrating on the feedback provider’s verbal and nonverbal communication behavior during feedback as an independent variable.114-117 Two aspects received particular attention in these studies: teachers’ nonverbal immediacy (TNI)117 and face-threat mitigation (FTM).114-116 Immediacy is the perceived relationship between people. Nonverbal language such as

19 General introduction

behavior and the feedback recipient’s behavior?89 What is the influence of a feedback provider’s perceived intentions on subordinate reactions to feedback?90 Multi-source feedback (MSF) is also studied. Attention is paid to the reliability and validity of instruments, the differences in ratings between sources, feedback recipient reactions towards MSF, and the extent to which ratings of subordinates are comparable between different cultures.91-92 Additionally, the topic of actively seeking feedback receives some attention in this literature.31, 93-94 Less frequently considered is the influence of ‘upward’ feedback -feedback from a subordinate to a manager- on job performance,95 and how employees react in terms of motivation and satisfaction with the performance standards used in providing feedback.96

Chapter 1

20

head nods, and eye contact can express a good relationship. 117-119 The word ‘Face’ in FTM refers to the desired self-image an individual presents in interactions with others. FTM refers to the competence of communicators in using linguistic and or nonverbal signs that minimize face threats when communicating.116, 120-121 When feedback providers are good at FTM when giving feedback, students perceive that they are well mentored,116 recognise the feedback as fair and useful and considered the feedback provider credible.114, 120 Studies on the the influence of feedback sensitivity are related to student task performance, and affect for the teacher.110, 117 There are some research similarities despite the different fields. Studies are mainly feedback provider, recipient, and message oriented in all four groups. In general the feedback environment does not receive much attention. A change in behavior, knowledge or attitude is sought in all fields. The relationship between feedback provider, -whether teacher, therapist or employer- and the feedback recipient, -whether learner, client or employee- is viewed important.

Choice of perspective Interpersonal communication is essential in giving oral feedback. It is assumed that the effectiveness of oral feedback partly depends on how the feedback is communicated by the feedback provider, the dialogue between the feedback provider and the recipient, but also on how the feedback recipient receives this information. Therefore, attention is given to communication aspects of feedback such as the message, the sender, the receiver and the dialogue. Two models from communication theory are integrated: the Lasswell formula and ‘Mr. A talks to Mr. B’ to create a model which is helpful in approaching research from a communication perspective (1.5).

1.4 Feedback in medical education 2006-2013 Feedback receives much attention in the medical education literature. Four hundred thirty-nine articles, comments, short reports and letters in PubMED which had ‘feedback’ in their title were published over the last eight years. The diversity of feedback literature is large. Two categories of feedback literature are evident. First, most of the studies the term feedback referred to information provided to persons. The feedback was about the performance of other people. Within this category the literature was about the feedback process, methods to gather or provide feedback, and the effect of feedback.

In the second category the information was directed to an institute, a group of teachers, or program director. The information was not about personal performance, but about curricula, a program or course of studies. Sometimes a person’s performance was part of the study but this was always considered as part of the evaluation of a program or curricula.

1.4.1 Feedback process Within the articles that describe the feedback process, some focus on the feedback recipient, the feedback content or the feedback provider while others are more general.

In 2009 Archer gave a general overview of the state of science in medical education about feedback. He accentuates that ‘only feedback seen along a learning continuum within a culture of feedback’ is likely to be effective (p. 102). He describes how the type,

Feedback recipient Studies in this category focus on a feedback recipient’s perceptions of the feedback process, feedback seeking behavior, and a small number of studies pay attention to receiving feedback and the relationship between the feedback recipient’s acceptance of feedback and the recipient’s self-assessment.

Studies on feedback perception focus on the feedback recipient’s perception more than Feedback literature 2006-2013 PubMED: (feedback [title]) AND medical education; Limits: last eight years, English.

To improve human performance

To improve programs

Feedback process 1.

2.

3.

4.

General overview Feedback recipient Feedback content Feedback provider

Program Evaluation 1.

2.

Towards aspects of the feedback process Towards programs

Feedback methods 1.

2.

Feedback providing Feedback improvement

1.

2.

3.

Feedback effects Learning process Patients health General working

Figure 1.1. Categories of research topics within the feedback literature

21 General introduction

structure and timing of feedback are important, but he is still inconclusive as to how it impacts effectiveness. Furthermore, Archer gives an explanation about how the ‘self’ of the feedback recipient, the recipient’s acceptability and goal setting behaviour affects how the feedback is perceived. Archer argues for creating a feedback continuum where feedback consists of events which are related and a feedback culture which is focused on development of individual students rather than on problem identification.122 Norcini also stresses the importance of creating a feedback culture in which learners are active participants in the feedback process.123

Chapter 1

22

on the provider’s perceptions. A feedback recipient’s perceptions towards feedback providers, feedback content and the perceived qualities of good feedback receive attention. A study in the Emergency Department showed that faculty who provided feedback are more satisfied with the overall quality and the timeliness of the feedback than the residents who received the feedback.124 Perera et al. studied the mismatch between faculty perceptions and students expectations regarding feedback. Teachers believed they frequently provided feedback, but only 55% of the students agreed that feedback was given frequently. Students believe that clearly defined, task oriented, simple, and timely feedback provided by a content expert to be the most useful form of feedback.125 In the clinical clerkship medical students perceive feedback from specialists and residents as equally instructive. Students perceive feedback based on observation to be more instructive than feedback without observation.126 Medical students perceived peer feedback on oral anatomy presentations positively. Feedback about structure and presentation skills was perceived as fair compared to feedback on the content and the ability to answer a research question.127 A study on feedback from patients to junior doctors about their consultation skills showed that doctors perceived this feedback useful.

The feedback not only empowered the patients but junior doctors also perceived an educational impact. They become more aware of weaknesses in their consultation, and some even took communication workshops to improve.128 Another line of studies focuses on feedback recipients’ perceptions of MSF.129 Feedback seeking receives some attention in the literature. Seeking feedback is helpful for developing competencies.130 Feedback encounter cards are used to prompt learners to ask for feedback.131 Learning goal orientation, perceived feedback costs and benefits and gender are examples of factors that affect learner’s feedback seeking behavior.132-133 A few studies consider the relationship between the feedback recipient and reception of feedback.134-139 Studies on the relationship between feedback acceptance, a recipient’s self-assessment and reception of feedback, stress the need for expert feedback because learners are not able to self-assess their skills.136 When MSF is in line with the feedback recipient’s self-assessment, it makes the acceptance of the feedback easier.134-139 Enhancing reflection among feedback recipients will stimulate the acceptance and the use of feedback.134, 139 Lee et al. showed that there is a relationship between a recipient’s communication style, the clinical clerkship feedback they received and demographic differences, such as gender, ethnicity/race and generation.140 Feedback content From studies on feedback content we learn that the quality of written feedback varies.

Comments are often not specific enough, unclear and not focused on behaviour.141 Content analysis of written feedback on the Mini-CEX reveals that feedback providers in nearly 23% of the cases did not identify positive features of the performance and in about 50% of the cases an action plan was not discussed.142 A study in which written feedback on consultation skills from General Practitioners (GP) and General Practitioners in Training (GPiT) were compared, revealed that the quality of feedback reports from GPiT was low. The reports were often unstructured and contained reflective questions, remarks with explanations and suggestions for improvement were lacking.143 Sherbino compared the themes in written feedback on communication skills from faculty and residents. The themes in the feedback of faculty and residents differed significantly.144

Feedback provider Feedback providers often find providing feedback difficult.44, 146-147 They may avoid giving feedback because they fear an emotional response, a reduction in popularity or a destruction of the trainer-trainee relationship.146 The number of articles about giving effective feedback.148-160 illustrate this difficulty. What these articles have in common is that they describe feedback guidelines or models, e.g., a series of steps the feedback provider should use when giving ‘effective’ or ‘constructive’ feedback. Some studies provide guidelines based on evidence.158, 162 Others explain clearly behaviors feedback providers should pay attention to,i.e., why residents avoid feedback, and the possibility that feedback recipients may discredit the feedback they receive.146 A few authors compared different guidelines and explored what was favored by residents and students.161-162 However, in most articles information about the relative effectiveness of these guidelines compared to other guidelines, is lacking. Many guidelines seem to be based on ‘gut feelings’ and best practices.

Burr and Brodier go beyond giving guidelines. Their article explains that feedback providers should be aware of their own learning style and the learning style of the feedback recipient. If possible they should tailor their feedback towards the trainee’s learning style, so the likelihood of applying the feedback is maximized.163 However, evidence is missing to show whether this approach is truly effective.

Courses on feedback provision receive some attention in the literature.13, 164-165 Bokken and colleagues reviewed 49 studies on the feedback process from the perspective of standardized patients (SP). She noticed that clear standards for effective training of SP in giving feedback are not described.13 When students received a course on providing feedback to faculty, this resulted in immediate and long-term improvement of feedback skills.165 Feedback training also improved the skills of tutors. Tutors that were considered effective by students were considered to have improved after attending feedback training, but for unsatisfactory tutors it may take more time before a marked improvement is noted.164 Characteristics of feedback providers receive little attention in the literature. Menarchy explored which personal characteristics are related to good feedback skills.166 Faculty with good feedback skills were aware of and able to handle the emotional responses of learners, encouraged students to solve problems themselves, were able to handle conflicts and focused on the learner needs including expectations, the student – teacher relationship and goal setting. Another study focused on the differences in ‘concern’ ratings junior doctors received from various people within the hospital staff. Peers, administrators and managers were less likely to indicate areas of concern in junior doctors’ performance. Consultants and sisters reported their concerns more often.167 1.4.2 Feedback methods The literature on feedback methods describes tools for gathering the input for feedback and providing feedback.

23 General introduction

A study on oral feedback dialogues showed that dialogues are very teacher-centric and the feedback message referred mostly to the student’s factual cognitive processing and much less to any conceptual or theoretical cognitive processing.145

Chapter 1

24

Feedback providing tools MSF instruments168-171 and feedback card systems are examples of methods used to gather input for written feedback.131, 172 A microsurgery trainer with a quantitative feedback form about the time to perform and indicators for pressure, is an example of a tool that can give feedback.173 In contrast to the literature on guidelines the feedback methods are evaluated and aspects of validity, feasibility and reliability are discussed.

Feedback improvement tools A second category of articles describes tools that can improve feedback. The webenabled video feedback method is a tool to improve giving feedback. Experienced physicians receive a link with which they are able to view video-recordings of their consultation. After watching they receive feedback from an expert and receive a feedback report.18 Dattner and Lopreiata describe a Direct Observation Program introduced in a pediatric resident clinic. Faculty were asked to use Structured Clinical Observation (SCO) for 3-5 minutes during a patient encounter, to complete the checklist and to provide feedback to residents. After four months it turned out that each resident was observed at least four times and that feedback was more focused on the patient encounter rather than on the medical knowledge and decision making.174 These studies address the importance of feedback and suggest that it should be a specific part of doctor training.

1.4.3 Feedback effects Many studies on feedback in medical education focus on the effect of feedback on performance outcome within randomized controlled trials (RCTs), quasi-experiments and pre- and post-test studies. The goal of these studies is to explain the effects and interactions. Within this group of studies three directions can be distinguished.

Learning Studies that focus on how feedback can enhance learning can be divided as studies where feedback is given by others and through simulators or computers. These are human mediated and computer mediated feedback studies. Computer mediated feedback studies often focus on psychomotor skills.

Task feedback Haptic feedback or force feedback is an example of tactile feedback. It ‘refers to the sense of touch that a surgeon experiences […] while performing surgery’(p.1180).175 With haptic feedback the surgeon can feel the difference between skin, bone and different tissues. Tactile feedback is applied in virtual reality simulations of surgical procedures such as laparoscopy175-178 and ventriculostomy catheter placement.179 Haptic feedback is a form of task feedback.

Visual feedback is also used as a means to stimulate performance. Graphic feedback, bars or diagrams representing scores, are a form of visual feedback.180-181 But the visual information a learner receives when performing a task in a two or three dimensional virtual environment is also called visual feedback.182 When learners receive audio feedback a certain noise gives information.75 The message of this feedback is not words but another type of data.

Audiovisual feedback is used with mannequins when students practice dealing with cardiac arrests or basic life support.183-184 The learner receives visual feedback from chest compression depth and an audible tone

25 General introduction

from the chest compression rate from the mannequin.

Numerical feedback consists of scores that learners retrieve form a certain task. The scores are not represented in a graph but in numbers.185-187 Sometimes this score is provided by an electronic device. In the studies mentioned the feedback consisted of surgeon’s scores of risk estimation, or benchmark scores from tests.

A last category is programmed worded feedback. This feedback is written words, for example in Computer Assisted Instruction (CAI). Students may receive elaborate feedback consisting of a theoretical argument or a short theoretical explanation and the correct argument. This feedback is presented visually.188 A variation is that the feedback can be presented orally, e.g., by a fictitious doctor who gives an extra explanation, and in a knowledge of correct results (KCR) situation where the student heard only ‘rightwrong’ feedback.189 Feedback from others The most important characteristic of feedback from others is that humans are its source. This contrasts with task feedback where the task itself is the feedback source. Information in this type of feedback is mostly conveyed verbally, and tailored to the feedback recipient.

When learners receive feedback it is usually delivered orally, in a written form or a combination of both. Although in oral feedback hearing is important, what we observe from the non-verbal communication is even more important. From the information we receive in communication approximately 66% is derived from non-verbal cues.119, 190 So for oral feedback the recipients need to use most of their senses, except when this feedback is transmitted via computer or phone.

Verbal feedback can also be written down and provided on paper or via computer. The Clinical Encounter Card and notes are examples of methods in which the feedback is written down.28 Feedback from others can be given through different channels, e.g. the computer (emails) or the telephone. In MSF situations and also in teacher evaluation sessions several feedback providers fill-out electronic feedback forms, the feedback from the different participants is collected and then presented to the feedback recipient. Sometimes a meeting in person follows in which the feedback content is discussed with a coach, an educator or a consultant.191, 221 Most of the MSF is transmitted by computer.191-194 When oral feedback is given there is no guarantee that the feedback is better than other types of feedback. Oral feedback varies in the degree of specificity and tailoring to the feedback recipient. Also the tone of voice, the length of the feedback, the number of examples used or the explanations are important aspects of the feedback.

Experiments with oral feedback include receiving oral feedback versus no feedback,195-198 and receiving quantitative feedback on tests completed with feedback from a faculty member, compared to no feedback.199 Different feedback providers have also been compared e.g., peer feedback and faculty feedback,200 and student examiner feedback versus faculty examiner feedback on an Objective Structured Clinical Examination (OSCE).201 Several studies compare different feedback sources including task feedback versus feedback from a human expert, verbal feedback from an expert versus self-accessed computer-generated feedback,202 receiving intensive or limited feedback from an instructor versus viewing video tutorials about the task.203 Other experiments focus on the timing of feedback by comparing the effect of feedback if given directly after each task versus after a series of tasks.25, 204 Also the message content -feedback versus

Chapter 1

26

compliment-205 and the feedback intensity -group feedback compared to one-to-one feedback from an instructor206 are compared.

Patient health In a second group of studies the outcome is focused on the patient health. Patients are the subject of the study. Feedback is applied in situations of alcohol, drugs,207 tobacco use,208-210 or in managing diabetes care,211 and obesity.212 Feedback in these studies is given by means of an computer assistant that gives feedback with facial expressions and by expressing empathy, and compliments along with supporting self-efficacy and optimism. One example included an online diary,212 where patient lung age was reported in relation to their own age and the number of respiratory symptoms from smoking,210 or computer tailored feedback on smoking behavior based on questionnaires.208 In Wilsons’ study parents that smoke and have children with asthma received counseling sessions and used Cotinine values as feedback markers. Cotinine is a sign of passive smoking.209 Health care processes Another area in medical education where feedback is given is general working processes in daily health care. In this category feedback is provided to healthcare providers including consultants, faculty members and residents. The effects of this in daily practice are the focus of the care and practice, instead of the learning process. So it has a practical spin-off although it will never go without a personal learning process. A few examples of these daily working processes in health care are hand hygiene,213 safe passing of sharp instruments,214 reporting behavior of endoscopic findings,215 coding clinical visits,216 working with guidelines,217-218 counseling,219-220 and aspects of teaching and leadership.221-222 Examples of feedback interventions are MSF,193 numeric scores and narrative comments about teaching,222 one day individual video-feedback training for genetic counselors,219 feedback on role-play of counseling session,220 education and personal feedback on coding of clinic visits.

1.4.4. Program evaluation Studies on the effectiveness of feedback on a learner’s performance have in common that they are person centered, because they evaluate a learner. In other studies on the effectiveness of feedback a program, module, procedure or curriculum is evaluated. So, feedback in this instance is not meant primarily for personal improvement, but to increase the quality of programs, modules or curricula.223 A variety of methods are used to gather this information, e.g. questionnaires,19, 224, 228, 234, 236 (exit) interviews,234-235 and focus groups.9, 223 Although the literature in these occasions uses the term feedback when a program is evaluated, the term ‘evaluation’ would be better because the program is evaluated and can’t receive feedback. In this way further erosion of the term feedback can be avoided.

Feedback process A few studies in this category focused their evaluation on feedback tools or programs such as the use of Mini-CEX or feedback workshops.6, 12, 19, 225-227 General practitioners (GP) reviewed Significant Event Analysis from peers. The reviewers gave written feedback on potential learning needs and they confirmed good

Programs and procedures Other studies in this category were very diverse. Topics being evaluated ranged from evaluation of the surgery residency training application, research programs,229 didactic methods,230 online programs,231-233 appraisal and assessments,223 use of portfolio’s,223-234 teaching courses,235 and OSCE.236 Focus of thesis For oral feedback in medical education not all the cited categories need to be studied.

Literature about feedback methods, effectiveness of feedback on patients, general working procedures (not in a learning situation), and evaluation of programs (other than feedback programs) did not receive attention in the literature of this thesis. In figure 1.1 the topics written in black are relevant to this thesis.

1.5 Communication perspective This thesis uses a feedback model where the feedback process is represented and where the communication receives specific attention from the perspective of the feedback provider and from the feedback recipient.

A model represents a theory about a phenomenon. Models are an abstraction of reality and only relevant points are represented.237 Models have several functions: reduce a complex process to a more simple form, visualize a process which enhances understanding, and help organize and classify different elements in a process.238-240 By choosing a communication perspective to approach feedback, the feedback model needs to reflect that feedback is viewed as a form of communication instead of ‘technical information transmission’. The model needs to reflect elements important in communication: such as message, sender and recipient. Further, it needs to reflect psychological notions, because we focus on communication between people and not on transmitting of information between devices or devices and humans.

The feedback model shown in figure 1.2 reduces the complex mechanisms in interpersonal communication to a chronological description of what happens over time. In

27 General introduction

practices. GP’s perceived this process as educationally valuable but reported that the feedback instrument needed more attention.6 Perera compared students on their communication skills while half of the group received peer feedback and self-assessment in addition to SP feedback and supervisor feedback. The group with the additional selfassessment perceived their communication skills to be improved and their results were better than the control group. Fifteen percent of the students felt shy or uncomfortable when they received feedback from other students, but they agreed that their peers mentioned gaps which were not reported by the SP.19 Perera asked students to rate SP feedback skills before and after the SPs received training in giving feedback. According to students’ scores SPs provided better feedback after receiving the training.12 Studies from Walsh and Bydder evaluated the effect of a feedback workshop given to faculty. Walsh evaluated commitment to changes and implementation of change.

Based on self-reports from faculty it was concluded that faculty committed to changes in their own feedback encounters. Three months later most faculty reported that the changes had been implemented.226 Bydder evaluated more in-depth how the workshop was perceived and what should be changed a next time.227

Chapter 1

28

Phase A1 the feedback recipient performs a task which has certain standards. In phase B the feedback provider observes the task from Phase A1 and interprets task performance by comparing it to implicit or explicit standards. In Phase C the feedback provider communicates the feedback which is the difference between feedback recipient’s task performance and the standard. In Phase D the feedback recipient receives and interprets the feedback, and in Phase A2 the task performance is repeated.

In this feedback model the difference between feedback -the message communicated in phase C-, feedback process –Phase A, B, C, and D-, and feedback effect– the difference between A2 and A1 -, are visualized and clearly distinguished from each other.

Attention is paid to the reception, perception and interpretation of the message by the sender and the recipient. The four different perspectives from which our research question can be approached (provider, recipient, message, and context) are not directly visible in the model but subsumed within it.

The feedback model is an integration of two adjusted models: the ‘Lasswell formula’ from H.D. Lasswell (1902-1978)241 and W.A.L. Johnson’s (1906-1965) ‘Mr. A talks to Mr. B’.242-244 These models stem from the time shortly after the Second World War when interest in (mass) communication increased significantly.237 These models focus in particular on communication instead of general (technical) information transmission.

Furthermore, they take the whole communication process into account.

The contribution of Johnson’s model is the element of interpersonal communication.

It gives attention to the recipient’s reception, perception and interpretation of the feedback. The Lasswell formula, on the other hand, emphasizes the importance of the message and describes the elements present in any communication.241 The combination of the two adjusted models results in a matrix. The horizontal axis chronologically describes the communication process according to Johnson. The vertical axis presents some of the research components as described in the Lasswell formula: sender, recipient, and message.

The thirty cells represent the relationship between the components from Lasswell and

Figure 1.2. Feedback model which represents the feedback, the feedback process and the feedback effect

20 26

19 25

Feedback form

Feedback context

27

21

15

9

3

Provider

Observation & interpretation

28

22

16

10

4

Provider

Feedback communication

Phase C

29

23

17

11

5

Recipient

Reception & interpretation

Phase D

30

24

18

12

6

Recipient

Task performance2

Phase A2*

* The feedback effect is included in the second performance in phase A2. By comparing the first performance and the second performance after receiving feedback, the difference will be visible when the same type of task is carried out. The effect can be described as: Δ phase A1-A2

14

13

Feedback content

2

Recipient

Task performance1

8

1

Task & standard

7

Actor >

Activity >

Feedback Recipient

Feedback provider

Component

Phase >

Johnson’s Mr. A. talks to Mr. B. Phase A1 Phase B

General introduction

Lasswell Formula

Table 1.1. Organizing framework based on the Johnson’s Mr. A talks to Mr. B model and the Lasswell formula

29

Chapter 1

30

the phases from Johnson. Cell 1 represents the relationship between the feedback provider and the task and standards used in the recipient’s task performance. Cell 2 represents the relationship between the feedback provider and feedback recipient’s task performance. In this way different relationships in the feedback process can be described. The matrix serves as an organizational framework, a tool to classify and describe variables that influence the feedback process and the feedback outcome or both.

1.6 Research questions To achieve integration of feedback literature from the various paradigms and fields (chapter 2 and 3), and the exploration of variables that influence the feedback process and effect (chapter 4, 5 and 6), two research questions were formulated divided into several sub questions.

The first research question is about the variables that affect the feedback process and the feedback effect. The answer to question 1b should contribute to the clarification of the feedback concept. Answers to questions 1b and 1c should lead to an integration of the literature. Answers to questions 1b, 1c and 1d should also give insight into whether variables effective in a non-medical context are perceived to be effective in a medical context.

The second research question explores the impact of two variables that are common in medical education on several outcome variables: feedback framing and the feedback provider’s credibility. Feedback framing is about the way feedback providers formulate the feedback message i.e. are the good points or the points for improvement accentuated? Besides the actual wording of the message nonverbal behavior such as facial expression and tone of voice are also part of the framing.245-246 Different ways of framing feedback messages are common across different settings, including medical education.

Personality characteristics, training or the lack thereof,33 and feedback provider’s mood are suggested causes for different message framing.

In the clinical setting feedback recipients receive feedback from people with the same background but different levels of experience, such as junior and senior clerks, residents and medical specialists. Additionally they also receive feedback from other professionals such as nurses and paramedics. Not all feedback providers are perceived as equally credible by the feedback recipient.116, 247-249 In the second study the feedback provider’s credibility is considered.

The dependent variables used in both studies are the medical student’s satisfaction with the feedback, self-efficacy towards a physical examination task and performance scores during a physical examination task. These variables represent different levels of the Kirkpatrick hierarchy.250 Student satisfaction with feedback and their self-efficacy are both outcome measures on the lowest level of Kirkpatrick’s hierarchy: perception.

Student performance of the physical examination is an outcome measured on the second level: performance measure. Performance of the physical examination in a different setting with standardized patients in a different context is an outcome variables in the third level of Kirkpatrick’s hierarchy: transfer. The fourth level, organizational outcomes, is not included in this study.250 By manipulating two variables related to feedback communication and feedback provider, the answers to questions 2a and 2b will contribute to our further understanding of why feedback is not always effective.

1.7 Thesis outline Chapter 2 on the definition of feedback is descriptive.251 It reviews feedback definitions given in dictionaries, handbooks, encyclopedias, in social sciences and medical education journals. Furthermore, it describes the conceptual formulation of feedback, and the common approaches in social science and medical education. Definitions are compared and based on common elements both a conceptual and operational definition are proposed. Agreement on a conceptual definition of feedback is a prerequisite to carry out the meta-review described in the third chapter.

To better understand the feedback process and the feedback effect an overview of variables influencing the feedback process and feedback itself was made. Reviews and meta-analysis with the feedback as information concepts are included in this review.

Based on inclusion and exclusion criteria, out of 507 studies, forty-one English lan-

31 General introduction

1) Which variables influence the feedback process and the feedback effect?

To answer this first question four sub questions need to be answered: 1a. What are the operational and conceptual definitions of feedback? (chapter 2) 1b. Which variables influence the feedback process and the feedback effect according to reviews and meta-analysis? (chapter 3) 1c. In which ways do the variables (from question 1b) impact the feedback process and the feedback effect? (chapter 3) 1d. Which variables impact the feedback process and the feedback effect according to medical student and a resident perception? (chapter 4) 2) How do variables related to the communication of feedback and the feedback provider affect a student’s performance and perceptions?

2a. How does framing of the feedback message affect a medical student’s satisfaction with the feedback, self-efficacy about task performance, and task performance? (chapter 5) 2b. How does the feedback provider’s credibility affect a medical student’s satisfaction with the feedback, self-efficacy of task performance, and task performance? (chapter 6) Chapters 2 to 6 relate to the feedback model described in paragraph 1.5 as shown below (figure 1.2).

Chapter 2, on the definition of feedback touches all four phases but the focus is on the concept of feedback as information which can be provided by a feedback provider and received by a feedback recipient (phases C and D).

Chapter 3, about the variables that influence the feedback process and effect (sub question 1b and 1c), relates to all phases of the feedback model. The research question about the perception of feedback describes how several variables from different phases in the feedback process are perceived. Feedback perception is the focus of chapter 4, and therefore is classified as covering the phase about feedback interpretation etc. (phase D).

Research questions two and three focus on the independent variables of a feedback provider’s credibility and the framing of the message, and the dependent variables related to feedback communication (phase C). The dependent variables relate to the feedback effect, especially variables of performance. Variable self-efficacy and satisfaction are also effects of the feedback but it also affects how the message is received so it is part of phase D.

Chapter 1

32

guage meta-analyses and literature reviews were selected and included in this study.

Articles were derived from several databases, e.g., Education Resources Information Center (ERIC), Medline and PsycINFO, in the period between 1986 and 2012. The variables are categorized according to the organizational framework based on the Lasswell formula and Mr. A talks to Mr. B (table 1.1). The second part of this chapter describes the impact of these variables on the feedback process and the feedback effect. It offers explanations for why the impact of feedback is often small to moderate, why it can be both positive and negative, and why the size of the effect can vary widely. This chapter was prepared using reporting conventions described in the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) Statement for meta-analyses and systematic reviews.252 The studies in chapters 2 and 3 review and synthesize existing information, and are therefore secondary research. Secondary research interprets or analyzes an event or phenomenon and is one or more steps removed from the event.253-255 Features of secondary research applicable to these types of studies include they are systematic, reproducible and use clear search strategies; critically appraise contributory studies; select relevant outcomes from the primary research; and use valid methods to integrate the evidence.256 Although not explicitly stated in chapter 4, 5 and 6, notions from the theory of social perception are used. Assumptions in the theory of social perceptions are that when people perceive the world around them, the perceived stimuli activate memory. Knowledge and previous experience related the perceived stimuli is brought to mind. This automatically activated information shapes impressions, social judgments, feelings, intentions and behavior. This happens spontaneously, without awareness or intention, so people are unaware of such an influence. When people perceive behavior social knowledge is activated and also social judgment and behavior.257-259 The studies described in chapters 4, 5 and 6 are primary research.253-255 Data were collected with questionnaires and checklists.

The purpose of chapter 4 is to explore a feedback recipient’s perceptions of variables important in the feedback process in the clinical setting. This study is therefore more descriptive in nature.251 The variables we identified in the meta-review were used as input for development of a feedback perception scale. The study uses an eight step procedure to systematically develop an instrument as presented by Comrey and Gorsuch.260-261 Feedback recipients are junior and senior clerks, and residents from both the University Medical Center in Utrecht and the Feinberg School of Medicine in Chicago.

Furthermore we explored which other variables could influence recipient answers such as nationality, age, gender, seniority, and medical specialty. The results are reported according to guidelines given by Henson and Roberts.262 Phase A

Phase B

Phase C x

x

x

x

x

x

Chapter 2 Chapter 3 Chapter 4

Phase D

Effect x

x

Chapter 5

x

x

x

Chapter 6

x

x

x

Table 1.2. Relationship of chapters to the different phases in the feedback process

33 General introduction

The overview of the variables and the extent to which they are valued by medical students and residents serves as an input for the dependent variables used in experimental studies. Chapters 5 and 6 clarify the effects of feedback based on two variables related to feedback communication. Both studies used a single blind randomised controlled between-subject design.255, 263 Chapter 5 focuses on framing the feedback message in a positive or negative manner. Chapter 6 investigates how a feedback providers’ high or low credibility can influence the outcome.

We investigated the student physical examination self-efficacy and how it shapes satisfaction. Student performance of a physical examination task and subsequent self-evaluation were re-measured during a follow-up evaluation after approximately three weeks.

The last chapter concludes the thesis with a general discussion. A summary is given where the strengths and relative limitations are discussed. A preliminary research agenda is also proposed, particularly on the application of thesis results to the clinics for improvement of daily practice.

34

References

Chapter 1

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

Ende J. Feedback in clinical medical education. JAMA 1983;250(6):777-781.

Feedback. In: The World Encyclopedia [Internet]. New York: Oxford University Press. 2001 [updated 2012; cited 2013 December 27] Available from: http://www.oxfordreference.com. proxy.library.uu.nl/view/10.1093/acref/9780199546091.001.0001/acref-9780199546091-e4047?rskey=m5rd1V&result=3812.

Ilgen DR, Fisher CD, Taylor MS. Consequences of individual feedback on behavior in organizations. J Appl Psychol 1979;79(4):349-371.

Kluger AN, DeNisi A. Effects of feedback intervention on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychol Bull 1996;119(2):254284.

Branch WT, Paranjape A. Feedback and reflection: teaching methods for the clinical setting.

Acad Med 2002;77(12):1185–1188.

McKay J. Shephard A, Bowie P, Lough M. Acceptability and educational impact of a peer feedback model for significant event analysis. Med Educ 2008;42:1210-1217.

Shiner B, Green RL, Homa K, Watts BV, Groft A, Torrey WC, Oxman TE. Improving depression care in psychiatry resident psychopharmacology clinic: measurement, monitoring, feedback and education. Qual Saf Health Care 2010;19:234-238.

Wald HS, Davis SW, Reis SP, Monroe AD, Borkan JM. Reflecting on reflections: Enhancement of medical education curriculum with structured field notes and guided feedback.

Acad Med 2009;84(7):830-835.

Mazor KM, Holtman CM, Shchukin Y, Mee J, Katsufrakis PJ. The Relationship Between Direct Observation, Knowledge, and Feedback: Results of a National Survey. Acad Med 2011;86(10):S63–S68.

Violato C, Lockyer JM, Fidler H. Assessments of Pychiatrists in practice through multisource feedback. Can J Psychiat 2008;53(8):525-533.

Lockyer J, Blackmore D, Fidler H, Crutcher R, Salte B, Shaw K, Ward B, Wolfish N. A study of a multisource feedback system for international medical graduates holding defined licences. Med Educ 2006;40:340-347.

Perera J, Perera J, Abdullah J, Lee N. Training simulated patients: evalution of a training approach using self-assessment and peer/tutor feedback to improve performance. BMC Med Educ 2009;9:37. doi:10.1186/147206920-9-37.

Bokken L, Linssen T, Scherpbier A, van der Vleuten C, Rethans JJ. Feedback by simulated patients in undergraduate medical education: a systematic review of the literature. Med Educ 2009;43:202-210.

Landon BE, Normand S-LT, David Blumenthal D, Daley J. Physician clinical performance assessment prospects and barriers. JAMA 2003;290(9):1183-1189.

Overeem K, Faber MJ, Arah OA, Elwyn G, Lombarts KMJMH, Wollersheim HC, Grol RPTM. Doctor performance assessment in daily practise: does it help doctors or not? A systematic review. Med Educ 2007;41:1039–1049.

Newby D. Personal development plans: making them work, making them count. Adv Psychiatr Treat 2003;9:5-10.

Overeem K, Driessen EW, Arah OA, Elwyn G, Lombarts KMJMH, Wollersheim HC, Grol RPTM. Peer mentoring in doctor performance assessment: strategies, obstacles and benefits. Med Educ 2010;44:140–147.

Noordman J, Verhaak P, Van Dulmen S. Web-enabled video-feedback: A method to reflect on the communication skills of experienced physicians. Patient Educ Couns 2011;82(3):335340.

Perera J, Mohamadou G, Kaur S. The use of objective structured self-assessment and peerfeedback (OSSP) for learning communication skills: evaluation using a controlled trial. Adv Health Sci Educ Theory Pract 2010;15:185-193.

McGee SR, Irby DM. Teaching in the outpatient clinic. Practical tips. J Gen Intern Med 1997;12(S2):S34–S40.

Irby DM. Teaching and Learning in Ambulatory care settings: A thematic review of the literature. Acad Med 1995;70(10):898-931.

35 General introduction

22. Irby DM, Wilkerson L. Teaching rounds. Teaching when time is limited. BMJ 2008;336:384387.

23. Royal College of Physicians and Surgeons of Canada. CanMEDS: better standards, better physicians, better care. [Internet]. 2013 [cited 2013 December 28]. Available from: http:// www.royalcollege.ca/public/resources/aboutcanmeds 24. Singh R, Naughton B, Taylor JS, Koenigsberg MR, Anderson DR, McCausland LL, et al. A comprehensive collaborative patient safety residency curriculum to address the ACGME core competencies. Med Educ 2005;39:1195-1204.

25. Walsh CM, Ling SC, Wang CS, Carnahan H. Concurrent versus terminal feedback: it may be better to wait. Acad Med 2009;84(10):S54-S57.

26. Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31.

Med Teach 2007;29:855-871.

27. Wilkinson JR, Crossley JGM, Wragg A, Mills P, Cowan G,Wade W. Implementing workplacebased assessment across the medical specialties in the United Kingdom, Med Educ 2008;2:364–373.

28. Ozua PO, Reznik M, Greenberg L. Improving medical student feedback with a clinical encounter card. Ambul Pediatr 2007;7(6):449-452.

29. Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees. A systematic review. JAMA 2009; 302(12):1316-1326.

30. Miller A, Archer J. Impact of workplace based assessment on doctors’ education and performance: a systematic review. BMJ 2010;341:c5064.

31. Ashford SJ, Blatt R, VandeWalle D. Reflections on the looking glass: A review of research on feedback-seeking behavior in organizations. J Manage 2003;29(6):773–799.

32. Wolfe Morrison E, Bies RJ. Impression management in the feedback-seeking process: a literature review and research agenda. Acad Manage Rev 1991;16(3):522-541.

33. Anseel F, Lievens F, Levy PE. A self-motives perspective on feedback-seeking behavior: Linking organizational behavior and social psychology research. Int J Manage Rev 2007;9(3):211236.

34. Salerno SM, O’Malley PG, Pangaro LN, Wheeler GA, Moores LK, Jackson JL. Faculty Development Seminars Based on the One‐Minute Preceptor Improve Feedback in the Ambulatory Setting. J Gen Intern Med 2002;17(10)779-787.

35. Eva KW, Regehr G. Self-assessment in the health professions: a reformulation and research agenda. Acad Med 2005;80(10):S46-S54.

36. Rozenblit K, Keil F. The misunderstood limits of folk science: an illusion of explanatory depth. Cogn Sci 2002;25(5):521-562.

37. Dunning D, Heath C, Suls JM. Flawed Self-assessment. Implications for health, education and the workplace. Psychol Sci Public Interest 2004;5(3):69-106.

38. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier R. Accuracy of Physician Self-assessment compared with observed measures of competence. A systematic review. JAMA 2006;269(9):1094-1102.

39. Eva KW, Cunnington JPW, Reiter HI, Keane DR, Norman GR. How can I know what I don’t know? Poor self-assessment in a well defined domain. Adv Health Sci Educ 2004;9:211–224.

40. Dornan T. Experienced based learning. Learning clinical medicine in workplaces [PhD thesis]. Manchester: Tim Dornan; 2006.

41. Schultz KW, Kirby J, Delva D, Godwin M, Verma S, Birtwhistle R, Knapper C, Seguin R. Medical students’ and residents’ preferred site characteristics and preceptor behaviours for learning in the ambulatory setting: a cross-sectional survey. BMC Med Educ 2004;4:12 doi:10.1186/1472-6920-4-12.

42. Bowen JL, Irby DM. Assessing quality and costs of education in the ambulatory setting: a review of the literature. Acad Med 2002;77:621-680.

43. Liberman A, Liberman M, Steinert Y, McLeod P, Meterissian S. Surgery residents and attending surgeons have different perceptions of feedback. Med Teach 2005;27:470-472.

44. McIlwrick J, Nair B, Montgomery G. “How Am I doing?” Many problems but few solutions related to feedback delivery in undergraduate psychiatry education. Acad Psychiatr 2006;30:130-135.

Chapter 1

36

45. Hamilton KEStc, Coates V, Kelly B, Boore JRP, Cundell JH, Mcfetridge B, McGonigle M, Sinclair M. Performance assessment in health care providers: a critical review of evidence and current practice. J Nurs Manag 2007;15:773-791.

46. Holmboe ES, Hawkins RE, Huot SJ. Effects of Training in Direct Observation of Medical Residents’ Clinical Competence. Ann Intern Med 2004;140(11)874-881.

47. Kilminster SM, Jolly BC. Effective supervision in clinical practice settings: a literature review.

Med Educ 2000;34:827-840.

48. Norcini JJ, McKinley DW. Assessment methods in medical education. Teach Teach Educ 2007;23:239–250.

49. Holmboe E. Faculty and the observation of trainees’ clinical skills: problems and opportunities. Acad Med 2004;79(1):16-22.

50. Hewson GH, Little ML. Giving Feedback in Medical Education.Verification of Recommended Techniques. J Gen Intern Med 1998;13:111–116.

51. Keenan JM, Seim HC, Bland CJ, Altemeier TM. A workshop program to train volunteer community preceptors. Acad Med 1990;65:46–47.

52. Simon SR, Sousa PJ, MacBride SE. The importance of feedback training. Acad Med 1997;72(1):1-2.

53. Wipf JE, Pinsky LE, Burke W. Turning interns into senior residents: preparing residents for their teaching and leadership roles. Acad Med 1995;70(7):591-596.

54. Richardson BK. Feedback. Acad Emerg Med 2004;11(12):1283.e1-1283.e5.

55. Skeff KM, Bowen JL, Irby DM. Protecting time for teaching in the ambulatory care setting.

Acad Med 1997;72(8):694-697.

56. Locke EA, Latham G. Building a practically useful theory of goal setting and task motivation. A 35-year Odyssey. Am Psychol 2002;57(9):707-717.

57. Locke EA, Chah D-O, Harrison S, Lustgarten N. Separating the effects of goal specificity from goal level. Organ Behav Hum Decis Process 1989;43(2):270-287.

58. Locke EA, Latham G. New directions in goal setting theory. Curr Dir Psychol Sci 2006;15(5):265-268.

59. Rosenblueth A, Wiener N, Bigelow J. Behavior, purpose, and teleology. Philos Sci 1943;10(1):1824.

60. Richardson GP. Feedback Thought in Social Science and Systems Theory. Philadelphia: University of Pennsylvania Press 1991.

61. Annett J. Feedback and Human Behaviour. The effects of knowledge of results, incentives and reinforcement on learning and performance. Harmondsworth: Penguin Books Inc. 1969.

62. Postman L. The history and present status of the law of effect. Psychol Bull 1947;44:489-563.

63. Boon L. Geschiedenis van de psychologie [A history of psychology]. Meppel: Boom 1996.

64. Bredo E. The social construction of learning. In: Phye GD (ed). Handbook of academic learning. Construction of knowledge. Academic Press San Diego. USA, 1997. (p.3-45).

65. Kluger AN, DeNisi A. Feedback Interventions: Towards the understanding of a DoubleEdged Sword. Curr Dir Psychol Sci 1998;7(3):67-72.

66. E.L. Thorndike as cited in: Bangert-Drowns RL, Kulik CL, Kulik JA, Morgan MT. The instructional effects of feedback in testlike events. Rev Educ Res 1991;62:213-238.

67. Anderson JR. Cognitive psychology and its implications. New York: Freeman and Company: 1995.

68. Brown JS, Collins A, Duguid P. Situated cognition and the culture of learning. Educ Res 1989;18:32-42.

69. Kanselaar G, de Jong T, Andriessen J, Goodyear P. New Technologies. In: Simons RJ, van der Linden J, Duffy T (eds). New Learning. Kluwer academic publishers Dordrecht, 2000. (p.5581) 70. Anderson JR, Rede LM, Simon HA. Applications and misapplications of cognitive psychology to mathematics education. ERIC Clearninghouse: 1999.

71. Duffy TM, Jonassen DH. Constructivism: new implications for instructional technology. In: Educational Technology. Englewood Cliffs New York, 1991. (p.34-40) 72. Phye GD. Epilogue: classroom learning, looking ahead. In: Phye GD (ed). Handbook of academic learning. Construction of knowledge. Academic Press San Diego. USA, 1997. (p.593596)

37 General introduction

73. Holmes LE, Smith LJ. Student evaluations of faculty grading methods. J Educ Bus 2003;78(6):318-323.

74. Hedgcock J, Lefkowitz N. Some input on input: two analyses of student response to expert feedback in L2 Writing. Mod Lang J 1996;80:287-308.

75. Van Sickle KR, Gallagher AG, Smith CD. The effect of escalating feedback on the acquisition of psychomotor skills for laparoscopy. Surg Endosc 2007;21:220-224.

76. Van Sickle, Smith B, Mcclusky DA, Baghai M, Smith CD, Gallagher AG. Evaluation of a tensiometer to provide objective feedback in knot-tying performance. Am Surgeon 2005;71:10181023.

77. Silverman S. Communication and motor skill learning: what we learn from research in the gymnasium. Quest 1994;46:345-355.

78. Nicholas H, Lightbown PM, Spada N. Recasts as feedback to language learners. Lang Learn 2001;51:719-758.

79. Freeman KA, Dexter-Mazza ET. Using self-monitoring with and adolescent with disruptive classroom behavior. Preliminary analysis of the role of adult feedback. Behav Modif 2004;28:402-419.

80. Heywood G. Forms and functions of feedback in CAI and CAL. J Comput Assist Lear 1986;2:83-92.

81. Mathan SA, Koedinger KR. Fostering the intelligent novice: Learning form errors with metacognitive tutoring. Educ Psychol 2005;40:257-265.

82. Richardson JTE. Instruments for obtaining student feedback: a review of the literature. Assess Eval High Educ 2005;30:387-415.

83. Penny AR, Coe R. Effectiveness of consultation on student ratings feedback: a meta-analysis.

Rev Educ Res 2004;74:215-253.

84. Nadler DA. Feedback and organizational development: using data-based methods. Reading Massachusetts: Addison-Wesley Publishing Company 1977.

85. Giffin ME. Personnel research on testing, selection and performance appraisal. Learning form Personnel Research: 1963-1988. Public Pers Manag 1989;18:127-137.

86. Sackett PR, DuBois CLZ. Rater-ratee effects on performance evaluation: challenging metaanalytic conclusion. J Appl Psychol 1991;76:873-877.

87. Greller MM. Managing feedback systems to facilitate change in acquisitions: The introduction of a model and explanation of it’s application. Hum Resour Manag R 2003;13:647-673.

88. Pearce JL, Porter LW. Employee reactions to formal performance appraisal feedback. J Appl Psychol 1986;71:211-218.

89. Larson JR. The performance feedback process: a preliminary model. Organ Behav Hum Perform 1984;33:42-76.

90. Fedor DB, Eder RW, Buckley MR. The contributory effects of supervisor intentions on subordinate feedback responses. Organ Behav Hum Dec 1989;44: 396-414.

91. Gillespie TL. Internationalizing 360-degree feedback: are subordinate ratings comparable? J Bus Psychol 2005;19:361-382.

92. Facteau CL, Facteau JD, Schoel LC, Russell JEA, Poteet ML. Reactions of leaders to 360-degree feedback from subordinate to peers. Leadership Quart 1998;9:427-448.

93. Sully de Luque MF, Sommer SM. The impact of culture on feedback seeking behavior: an integrated model and propositions. Acad Manage Rev 2000;25:829-849.

94. VandeWalle D. A goal orientation model of feedback-seeking behavior. Hum Resource Manag Rev 2003;13:581-604.

95. Heslin PA, Latham GP. The effect of upward feedback on managerial behavior. Int Rev Appl Psychol 2004;53:23-37.

96. Bobko P, Colella A. Employee reactions to performance standards: a review and research propositions. Pers Psychol 1994;47:1-29.

97. Morran KD, Robinson FF, Stockton R. Feedback exchange in counseling groups: an analysis of message content and receiver acceptance as a function of leader versus member delivery, session and valence. J Couns Psychol 1985;32:57-67.

98. Raymond DD, Dowrick PW. Affective responses to seeing oneself for the first time on unedited videotape. Couns Psychol Q 1993;6:193-201.

Chapter 1

38

99. DiClimente CC, Marinilli AS, Singh M, Bellino LE. The role of feedback in the process of health behavior change. Am J Health Behavior 2001;25:217-227.

100. Whipple JL, Lambert MJ, Vermeersch DA, Smart DW, Nielsen SL, Hawkins EJ. Improving the effects of psychotherapy: The use of early identification of treatment failure and problem-solving strategies in routine practice. J Couns Psychol 2003;50:59-68.

101. Sapyta J, Riemer M, Bickman L. Feedback to clinicians: theory, research and practice. J Clin Psychol 2005;61(2):145-153.

102. Walters ST, Neighbors C. Feedback interventions for college alcohol misuse: What, why and for whom? Addict Behav 2005;30:1168-1182.

103. Rosen S, Tesser A. On reluctance to communicate undesirable information: The MUM effect. Sociometry 1970;33(3):253-263.

104. Gardiner JC. A synthesis of experimental studies of speech communication feedback. J Commun 1971;21:17-35.

105. Tesser A, Rosen S, Conlee MC. News valence and available recipient as determinants of news transmission. Sociometry 1972;35(4):619-628.

106. Frandsen, KD, Millis MA. On conceptual, theoretical and empirical treatments of feedback in human communication: fifteen years later. Comm Rep 1993;6:79-91.

107. Dannels DP, Housley GaffneyAL, Norris Martin K. Students’ talk about the climate of feedback interventions in the critique. Commun Educ 2011;60(1):95-114.

108. Ballard DI, Seibold DR. Communication‐related organizational structures and work group temporal experiences: the effects of coordination method, technology type, and feedback cycle on members’ construals and enactments of time. Communication Monogr 2004;71(1):127.

109. King PE, Schrodt P, Weisel JJ. The instructional feedback orientation scale: Conceptualizing and validating a new measure for assessing perceptions of instructional feedback. Commun Educ 2009;58(22):235-261.

110. Smith CD, King PE. Student feedback sensitivity and the efficacy of feedback interventions in public speaking performance improvement. Commun Educ 2004;53(3):203-216.

111. Waung M, Jones DR. The effect of feedback packaging on ratee reactions. J Appl Soc Psychol 2005;35:1630-1655.

112. Hecht ML, The conceptualization and measurement of interpersonal communication satisfaction. Hum Commun Res 1978;4(3):253-264.

113. Hinton JS, Kramer MW. The impact of self‐directed videotape feedback on students’ self‐reported levels of communication competence and apprehension. Commun Educ 1998;47(2):151-161.

114. Witt PL, Kerssen-Griep J. (2011): Instructional feedback I: The interaction of facework and immediacy on students’ perceptions of instructor credibility. Commun Educ 2011;60(1):7594.

115. Trees AR, Kerssen-Griep J, Hess JA. Earning influence by communicating respect: facework’s contributions to effective instructional feedback. Commun Educ 2009;58(3):397-416.

116. Kerssen-Griep J, Trees AR, Hess JA. Attentive facework during instructional feedback: key to perceiving mentorship and an optimal learning environment. Commun Educ 2008; 57(3):312332.

117. Martin L, Mottet TP. The effect of instructor nonverbal immediacy behaviors and feedback sensitivity on Hispanic students’ affective learning outcomes in ninth-grade writing conferences. Commun Educ 2011;60(1):1-19.

118. Mehrabian A. A semantic space for nonverbal behavior. J Consult Clin Psychol 1970;35(2):248257.

119. Burgoon JK, Guerrero LK, Floyd K. Nonverbal communication. US: Pearson Education 2010.

120. Trees AR, Manusov V. Managing face concerns in criticism: Integrating nonverbal behaviors as a dimension of politeness in female friendship dyads. Hum Commun Res 1998;24:564-683.

121. Brown P, Levinson SC. Politeness: Some universals in language use. Cambridge: Cambridge University Press 1987.

122. Archer JC. State of science in health professional education: effective feedback. Med Educ 2010;44:101-108.

123. Norcini J. The power of feedback [commentaries]. Med Educ 2010;44:16-17.

39 General introduction

124. Yarris LM, Linden JA, Hern HG, Lefebvre C, Nestler DM, Fu R, Choo E, LaMantia J, Brunett P. Attending and resident satisfaction in the Emergency Department. Acad Emerg Med 2006;16:S76-S81.

125. Perera J, Lee N, Win K, Perera J, Wijesuriya L. Formative feedback to students: the mismatch between faculty perceptions and student expectations. Med Teach 2008;30:395-399.

126. Van Hell, EA, Kuks JBM, Raat AN, van Lohuizen MT, Cohen-Schotanus J. Instructiveness of feedback during clerkships: Influence of supervisor, observation and student initiative. Med Teach 2009;31(1):45-50.

127. Gukas ID, Miles S, Heylings DJ, Leinster SJ. Medical students’ perception of peer feedback on an anatomy student-selected study module. Med Teach 2008;30:812-814.

128. Burford B, Greco M, Bedi A, Kergon C, Morrow G, Livingston M, Illing J. Does questionnaire-based patient feedback reflect the important qualities of clinical consultations? Context, benefits and risks. Patient Educ Couns 2011;84:e28-e36.

129. Tham K-Y. 360° feedback for emergency physicians in Singapore. Emerg Med J 2007;24:574575.

130. Cook V, Sharma A, Alstead E. Introduction to teaching for junior doctors 2: collecting feedback and developing expertise. Br J Hosp Med 2009;70(12):711-713.

131. Kogan, JR, Shea JA. Implementing feedback cards in core clerkships. Med Educ 2008;42:10711079.

132. Teunissen PW, Stapel DA, van der Vleuten C, Scherpbier A, Boor K, Scheele F. Who wants feedback? An investigation of the variables influencing residents’ feedback seeking behavior in relation to night shifts. Acad Med 2009;84(7):910-917.

133. Sinclair HK, Cleland JA. Undergraduate medial students: who seeks formative feedback?

Med Educ 2007;41:580-582.

134. Sargeant JM, Mann KV, van der Vleuten CP, Metsemakers JF. Reflection: a link between receiving and using assessment feedback. Adv Health Sci Educ Theory Pract 2009;14:399-410.

135. Lockyer JM, Violato C, Fidler HM. What multisource feedback factors influence physician self-assessments? A five year longitudinal study. Acad Med 2007;82(10):S77-S80.

136. Pandey VA, Wolfe JHN, Black SA, Liapis CD, Bergqvist D. Self-assessment of technical skill in surgery: the need for expert feedback. Ann R Coll Surg Engl 2008;90:286-290.

137. Sargeant J, Mann K, van der Vleuten C, Metsemakers J. “Directed” self-assessment: Practice and feedback within a social context. J Contin Educ Health Prof 2008;28(1):47-54.

138. Sargeant J, Mann K, Sinclair D, van der Vleuten C, Metsemakers J. Challenges in multisource feedback: intended and unintended outcomes. Med Educ 2007;41:583-591.

139. Sargeant J, Mann K, Sinclair D, van der Vleuten C, Metsemakers J. Understanding the influence of emotions and reflection upon multi-source feedback acceptance and use. Adv Health Sci Educ Theory Pract 2008;13:275-288.

140. KB, Vaishnavi SN, Lau SKM, Andriole DA, Jeffe DB. Cultural competency in medical education: demographic differences associated with medical student communication styles and clinical clerkship feedback. J Natl Med Assoc 2009;101:116-126.

141. Canavan C, Holtman MC, Richmond M, Katsufrakis PJ. The quality of written comments on professional behaviors in a developmental multisource feedback program. Acad Med 2010;85:S106-S109.

142. Fernando N, Cleland J, McKenzie H, Cassar K. Indentifying the factors that determine feedback given to undergraduate medical students following formative mini-CEX assessments.

Med Educ 2008;42:89-95.

143. Prins FJ, Sluijsmans DMA, Kirschner PA. Feedback for General Practitioners in Training: Quality, styles, and preferences. Adv Health Sci Educ Theory Pract 2006;11:289-303.

144. Sherbino J, Bandiera G. Improving communication skills: Feedback from faculty and residents. Acad Emerg Med 2006;13:467-470.

145. Blatt B, Confessore S, Kallenberg G, Greenberg L. Verbal Interaction Analysis: Viewing feedback trough a different lens. Teach Learn Med 2008;20(4):329-333.

146. Denchfield Schlecht K. Feedback. Int Anesthesiol Clin 2008;46(4):67-84.

147. Bing-You RG, Trowbridge RL. Why medical educators may be failing at feedback. JAMA 2009;302(12):1330-1331.

Chapter 1

40

148. Sepdham D, Julka M, Hofmann L, Dobbie A. Using the RIME model for learner assessment and feedback. Fam Med 2007;39(3):161-163.

149. McKimm J. Giving effective feedback. Br J Hosp Med 2009;70(3):158-161.

150. Donelly P, Kirk P. How to… give effective feedback. Educ Prim Care 2010;21:267-269.

151. Gigante J, Dell M, Sharkey A. Getting beyond “Good Job”: How to give effective feedback.

Pediatrics 2011;127:205-207.

152. Chur-Hansen A, McLean S. On being a supervisor: the importance of feedback and how to give it. Australas Psychiatry 2006;14(1):67-71.

153. Cantillon P, Sargeant J. Giving feedback in clinical settings. BMJ 2008;337:1292-1294.

154. Bhattarai MD. ABCDEFG IS – The principle of constructive feedback. JNMA Nep Med Assoc 2007;46(167):151-156.

155. DeLima Thomas J, Arnold RM. Giving feedback. J Palliat Med 2011;14(2):233-239.

156. Hamid Y, Mahmood S. Understanding constructive feedback: A commitment between teachers and students for academic and professional development. J Pak Med Assoc 2010;60(3):224-227.

157. Bienstock JL, Katz NT, Cox M, Hueppchen N, Erickson S, Puscheck EE. To the point: medical education reviews – providing feedback. Am J Obstet Gynecol 2007;196(6):508-513.

158. Milan FB, Parish SJ, Reichgott MJ. A model for educational feedback based in clinical communication skills strategies: Beyond the “Feedback Sandwich”. Teach Learn Med 2006;18:4247.

159. Carr S. The Foundation Programme assessment tools: An opportunity to enhance feedback to trainees? Postgrad Med J 2006;82:576-579.

160. Shute VJ. Focus on formative feedback. Rev Educ Res 2008;78:153-189.

161. Sepdham D, Morrow J, Dobbie A. Feedback on medical student’s performance valued. Fam Med 2008;40(7):463-464.

162. Roked F, Roked F, Oyebode F. A comparative study of methods of feedback in medical education. [letters to the editor]. Med Teach 2009;31:1038-1039.

163. Burr S, Brodier E. Integrating feedback into medical education. Brit J Hosp Med 2010;71(1):646649.

164. Baroffio A, Nendaz MR, Perrier A, Vu NV. Tutor training, evaluation criteria and teaching environment influence students’ ratings of tutor feedback in problem based learning. Adv Health Sci Educ Theory Pract 2007;12:427-439.

165. Kruidering-Hall M, O’Sullivan PS, Chou CL. Teaching feedback to first-year medical students: long-term skill retention and accuracy of student self-assessment. J Gen Intern Med 24(6):721-726.

166. Menarchy EP, Knight AM, Kolodner K, Wright SM. Physician characteristics associated with proficiency in feedback skills. J Gen Intern Med 2006;21:440-446.

167. Bullock AD, Hassell A, Markham WA, Wall DW, Whitehouse AB. How ratings vary by staff group in multi-source feedback assessment of junior doctors. Med Educ 2009;43:516-520.

168. Archer J, McGraw M, Davies H. Assuring validity of multisource feedback in a national programme. Arch Dis Child 2010;95(5):330-335.

169. Violato C, Worsfold L, Polgar JM. Multisource feedback systems for quality improvement in the health professions: assessing occupational therapist in practice. J Contin Educ Health Prof 2009;29(2):111-118.

170. Berk RA. Using the 360° multisource feedback model to evaluate teaching and professionalism. Med Teach 2009;31:1073-1060.

171. Kagan I, Kigli-Shemesh R, Tabak N. ‘Let me tell you what I relay think about you’- evaluating nursing managers anonymous staff feedback. J Nurs Manag 2006;14:356-365.

172. Paukert JL, Richards ML, Olney C. An encounter card system for increasing feedback to students. Am J Surg 2002;183(3):300-304.

173. Kligman BE, Haddock NT, Garfein ES, Levine JP. Microsurgery trainer with quantitative feedback: a novel training tool for microvascular anastomis and suggested training exercise.

Plast Reconstr Surg 2010;126(6):328e-330e.

174. Dattner L, Lopreiato JO. Introduction of a direct observation program into a pediatric resident continuity clinic: feasibility, acceptability, and effect on resident feedback. Teach Learn Med 2010;22(4):280-286.

41 General introduction

175. Van der Meijden OAJ, Schijven MP. The value of haptic feedback in conventional and robotassisted minimal invasive surgery and virtual reality training: a current review. Surg Endosc 2009;23:1180-1190.

176. Chmarra MK, Dankelman J, Van den Dobbelsteen JJ, Jansen FW. Force feedback and basic laparoscopic skills. Surg Endosc 2008;22:2140-2148.

177. Ström P, Hedman L, Särnä L, Kjellin A, Wredmark T, Felländer-Tsai L. Early exposure to haptic feedback enhances performance in surgical simulator training: a prospective randomized crossover study in surgical residents. Surg Edosc 2006;20:1383-1388.

178. Panait L, Akkary E, Bell RL, Roberts KE, Dudrick SJ, Duffy AJ. The role of haptic feedback in Laparoscopic simulation training. J Surg Res 2009;156:312-316.

179. Banerjee PP, Luciano CJ, Lemole GM, Charbel FT, Oh YM. Accuracy of ventriculostomy catheter placement using a head- and hand-tracked high resolution virtual reality simulator with haptic feedback. J Neurosurg 2007;107:515-521.

180. Rafiq A, Tamariz F, Boanca C, Lavrentyev V, Merrell RC. Objective assessment of training surgical skills using simulated tissue interface with real-time feedback. J Surg Educ 2008;65(4):270-274.

181. Judkins TN, Oleynikov D, Stergiou N. Enhancing robotic surgical training using augmented visual feedback. Surg Innov 2008;15(1):59-68.

182. Oleynikov D, Salomon B, Hallbeck S. Effect of visual feedback on surgical performance using the da Vinci® Surgical System. J Laparoendosc Adv A 2006;16(5):503-508.

183. Dine CJ, Gersh RE, Leary M, Riegel BJ, Bellini LM, Abella BS. Improving cardiopulmonary resuscitation quality and resuscitation training by combining audiovisual feedback and debriefing. Crit Care Med 2008;36(10):2817-2822.

184. Spooner BB, Fallaha JF, Kocierz L, Smith CM, Smith SCL, Perkins GD. An evaluation of objective feedback in basic life support (BLS) training. Resuscitation 2007;73:417-424.

185. Jacklin, R, Sevdalis N, Darzi A, Vincent CA. Efficacy of cognitive feedback in improving operative risk estimation. Am J Surg 2009;197:67-81.

186. Srinivasan M, Hauer KE, Der-Martirosian C, Wilkes M, Gesundheit N. Does feedback matter? Practice-based learning of medical students after a multi-insitutional clinical performance examination. Med Educ 2007;41:857-865.

187. Lazarski MP, Susarla SM, Bennett NL, Seldin EB. How do feedback and instructions affect the performance of a simulated surgical task? J Oral Maxillofac Surg 2007;65:1155-1161.

188. Roels P, van Roosmalen G, van Soom C. Adaptive feedback and student behaviour in computer-assisted instruction. Med Educ 2010;44:1185-1193.

189. Kopp V, Stark R, Fischer MR. Fostering diagnostic knowledge through computer-supported, case-based worked examples: effects of erroneous examples and feedback. Med Educ 2008;42:823-829.

190. Birdwhistell RL. Background to kinesics. ETC 1955;13:10-18.

191. Brinkman WB, Geraghty SR, Lanphear BP, Khoury JC, Gonzalez del Rey JA, DeWitt TG, Britto MT. Effect of multisource feedback on resident communication skills and professionalism. Arch Pediatr Adolesc Med 2007;161:44-49.

192. Stark R, Korenstein D, Karani R. Impact of a 360-degree professionalism assessment on faculty comfort and skills in feedback delivery. J Gen Intern Med 2008;23(7):969-972.

193. Malling B, Bonderup T, Mortensen L, Ringsted C, Scherpbier A. Effects of multisource feedback on developmental plans for leaders of postgraduate medical education. Med Educ 2009;43:159-167.

194. Stalmeijer RE, Dolmans DHJM, Wolfhagen IHAP, Peters WG, van Coppenolle L, Scherpbier AJJA. Combined student ratings and self-assessment provide useful feedback for clinical teachers. Adv Health Sci Educ Theory Pract 2010;15:315-328.

195. Grantcharov TP, Schulze S, Kristiansen VB. The impact of objective assessment and constructive feedback on improvement of laparoscopic performance in the operating room.

Surg Endosc 2007;212240-2243.

196. Harewood GC, Murray F, Winder S, Patchett S. Evaluation of formal feedback on endoscopic competence among trainees: the EFFECT trial. Ir J Med Sci 2008;177 253-256.

Chapter 1

42

197. Kruglikova I, Grantcharov TP, Drewes AM, Funch-Jensen P. The impact of constructive feedback on training in gastrointestinal endoscopy using high-fidelity virtual-reality simulation: a randomized controlled trial. Gut 2010;59:181-185.

198. White CB, Ross PT, Gruppen LD. Remediating students’ failed OSCE performances at one school: the effects of self-assessment, reflection and feedback. Acad Med 2009;84(5):651654.

199. Brar MK, Laube DW, Bett GCL. Effect of quantitative feedback on student performance on the National Board Medical Examination in an obstetrics and gynecology clerkship. Am J Obstet Gynecol 2007;197:530.e1-530.e5.

200. Chacko TV, Seetharaman N, Murthy SN. Improving competencies through peer feedback in primary care settings. Med Educ 2006;40:459-489.

201. Moineau G, Power B, Pion A-MJ, Wood TJ, Humphrey-Murto S. Comparison of student examiner to faculty examiner scoring and feedback in an OSCE. Med Educ 2011;45:183-191.

202. Porte MC, Xeroulis G, Reznick RK, Bubrowski A. Verbal feedback from an expert is more effective than self-accessed feedback about motion efficiency in learning new surgical skills.

Am J Surg 2007;193:105-110.

203. Stefanidis D, Korndorffer JR, Heniford BT, Scott DJ. Limited feedback and video tutorials optimize learning and resource utilization during laparoscopic simulator training. Surgery 2007;142:202-206.

204. Xeroulis GJ, Park J, Moulton C-A, Reznick RK, LeBlanc V, Dubrowski A. Teaching suturing and knot-tying skills to medical students: A randomized controlled study comparing computer-based video instruction and (concurrent and summary) expert feedback. Surgery 2007;141:442-449.

205. Boehler ML, Rogers DA, Schwind CJ, Mayforth R, Quin J, Williams RG, Dunnington G. An investigation of medical student reactions to feedback: a randomized controlled trial. Med Educ 2006;40:746-749.

206. Camp CL, Gregory JK, Lachman N, Chen LP, Juskewitch JE, Pawlina W. Comparative efficacy of group and individual feedback in gross anatomy for promoting medical student professionalism. Anat Sci Educ 2010;3:64-72.

207. Bywood, P, Lunnay B, Roche AM. Strategies for facilitating change in alcohol and other drugs (OAD) professional practice: a systematic review of the effectiveness of reminders and feedback. Drug Alcohol Rev 2008;27:548-558.

208. Gilbert H, Nazareth I, Sutton S. Assessing the feasibility of proactive recruitment of smokers to an intervention in general practice for smoking cessation using computer-tailored feedback reports. Fam Pract 2007;24:395-400.

209. Wilson SR, Farber HJ, Knowles SB, Lavon PW. A randomized trial of parental behavioral counseling and cotinine feedback for lowering environmental tobacco smoke exposure of children with asthma: results of the LET’S Manage Asthma trial. Chest 2010;139(3):581-590.

210. Lipkus IM, Prokhorov AV. The effects of providing lung age and respiratory symptoms feedback on community college smokers’ perceived smoking-related health risks, worries and desire to quit. Addict Behav 2007;32:516-532.

211. O’Connor PJ, Rush WA, Sperl-Hillen JA, Johnson PE, Rush WA, Crain AL. Customized feedback to patients and providers failed to improve safety or quality of diabetes care. Diabetes Care 2009;32(7):1158-1163.

212. Blanson Henkemans OA, van der Boog PJM, Lindenberg J, van der Mast CAPG, Neerincx MA, Zwetsloot-Schonk BJHM. An online lifestyle diary with a persuasive computer assistant providing feedback on self-management. Technol Health Care 2009;17:253-267.

213. Doron SI, Kifuji K, Hynes BT, Dunlop D, Lemon T, Hansjosten K, et al. A Multifaceted approach to education, observation, and feedback in a successful hand hygiene campaign.

Joint Comm J Qual Patient Saf 2011;37(1):3-10.

214. Cunningham TR, Austin J. Using goal setting, task clarification and feedback to increase the use of the hands-free technique by hospital operating room staff. J Appl Behav Anal 2007;40:673-677.

215. Abdella AA, Petersen BT, Ott, BJ, Fredericksen, Schleck CD, Zinsmeister AR, Grunewald KMJ, Zais T, Romero Y. Impact of feedback an didactic sessions on the reporting behavior of upper endoscopic findings by physicians and nurses. Clin Gastroenterol Hepatol 2007;5:326330.

43 General introduction

216. Skelly KS, Bergus GR. Does structured audit and feedback improve the accuracy of residents’ CPT E&M coding of clinical visits? Fam Med 2010;42(9):648-652.

217. Keuken DG, Haafkens JA, Mohrs J, Klazinga NS, Bindels PJE. Evaluating the effectiveness of an educational and feedback intervention aimed at improving consideration of sex differences in guideline development. Qual Saf Health Care 2010;19:e18. doi:10.1136/ qshc.2007.025643.

218. Dulko D, Hertz E, Julien J, Beck S, Mooney K. Implementation of cancer guidelines by acute nurse practitioners using an audit and feedback strategy. J Am Acad Nurse Pract 2010;22:4555.

219. Pieterse AH, van Dulmen AM, Beemer FA, Ausems MGEM, Bensing JM. Tailoring communication in cancer genetic counseling through individual video-supported feedback: a controlled pretest-posttest design. Patient Educ Couns 2006;60:326-335.

220. Kisuule F, Necochea A, Howe EE, Wright S. Utilizing audit and feedback to improve hospitalists’ performance in tobacco dependence counseling. Nicotine Tobacco Research 2010;12(8):797-800.

221. Malling B, Mortensen L, Bonderup T, Scherpbier A, Ringsted C. Combining a leadership course and multi-source feedback had no effect on leadership skills of leaders in postgraduate medical education. An intervention study with a control group. BMC Med Educ 2009;9:72 doi:10.1186/1472-6920-9-72.

222. Baker K. Clinical Teaching improves with resident evaluation and feedback. Anesthesiology 2010;113:693-703.

223. Johnson G, Barrett J, Jones M, Parry D, Wade W. Feedback from educational supervisors and trainees on the implementation of curricula and the assessment system for core medical training. Clin Med 2008;8(5):484-489.

224. Holla SJ, Ramachandran K, Isaac B, Koshy S. Anatomy education in a changing medical curriculum in India: Medical student feedback on duration and emphasis of gross anatomy teaching. Anat Sci Educ 2009;2:179-183.

225. Dewi SP, Achmad TH. Optimising feedback using the Mini-CEX during the final semester programme. Med Educ 2010;44:509.

226. Walsh AE, Arnson H, Wakefield JG, Leadbetter W, Roder S. Using a small-group approach to enhance feedback skills for community-based teachers. Teach Learn Med 2009;21(1):45-51.

227. Bydder S, Bloomfield L, Dally M, Harris P, Dorset L, Semmens J. Preparing to sit the Royal Australia and New Zealand College of Radiologist Faculty of Radiation Oncology Fellowship part 2 examination: the value of a workshop including practice and feedback. Australas Radiol 2007;51:465-471.

228. Rogers CR, Gutowski KA, Munoz-Del Rio A, Larson DA, Edwards M, Hansen JE, Lawrence WT, Stevenson TR, Bentz ML. Integrated plastic surgery residency applicant survey: characteristics of successful applicants and feedback about the interview process. Plas Reconstr Surg 2009;123:1607-1617.

229. Chaturvedi SN. Skill building program in Population-based research for medical undergraduates: Learners’ feedback. Indian J Public Health 2008;52(4):185-188.

230. Karanth KVL, Kumar MV. Implementation and evaluation by formal assessments and term end student feedback of a new methodology of clinical teaching in surgery in small group sessions. Ann Acad Med Singapore 2008;37:1008-1011.

231. Walsh K, Donnelly A. Constructing a multimedia resource for managing Clostridium difficile: feedback and effectiveness. Med Educ 2008;42:1119-1120.

232. Boulos MNK. Map of dermatology: ‘first impression’ user feedback and agenda for further development. Health Info Libr J 2006;23:203-213.

233. Wheeler DW, Whittlestone KD, Johnston AJ, Smith HL. Problems encountered with a pilot online attendance record and feedback scheme of medical students. Educ Health 2006;19(3):369-374.

234. Hrisos S, Illing JC, Burford BC. Portfolio for foundation doctors: early feedback on its use in the clinical workplace. Med Educ 2008;42:214-223.

235. Chur-Hansen A, Koopowitz L, Jureidini J, Abhary S, McLean S. An interdisciplinary course of trainee psychiatrists: feedback and implications. Australas Psychiatry 2006;24(2):186191.

Chapter 1

44

236. Khursheed I, Usman Y, Usman J. Student’s feedback of Objectively Structured Clinical Examination: a private medical college experience. J Pak Med Ass 2007;57(3):148-150.

237. Stappers JG. Publicistiek en communicatiemodellen.[Publicistic and communication models] [PhD thesis] Nijmegen: Drukkerij Busser & Co; 1966.

238. Suppes P, Pavel M, Falmagne J-Cl. Representations and models in psychology. Annu Rev Psychol 1994;45:517-544.

239. Deutsch KW. On communication models in the social sciences. Publ Opin Q 1952;16(3):356380.

240. Lachman R. The model in theory construction. Psychol Rev 1960;67(2):113-129.

241. “Lasswell, Harold”. In: Craig Calhoun, editor. Dictionary of the Social Sciences. New York:Oxford University Press; 2002. [updated 2012; cited 2013 December 28] Available from: http://www.oxfordreference.com.proxy.library.uu.nl/view/10.1093/ acref/9780195123715.001.0001/acref-9780195123715-e-930?rskey=avEqus&result=930 242. Johnson W. People in quandaries. The semantics of personal adjustment. New York: Harper & Row Publishers 1946.

243. Johnson W. Speech and personality. In: Bryson L, editor. The communication of ideas. A serie of addresses. New York: Cooper Square Publishers; 1964.

244. Johnson N. Wendell A.L. Johnson (1906-1956) Memorial Home Page [Internet]. 1997 [updated 2009 June 6; cited 2013 December 28] Available from: http: //www.uiowa.edu/~cyberlaw/ oldinav/wjhome.html.

245. Dunegan KJ. Fines, frames, and images: examining formulation effects on punishment decisions. Organ Behav Hum Decis Process 1996;68:58-67.

246. Levin IP, Schneider SL, Gaeth GJ. All frames are not created equal: a typology and critical analysis of framing effects. Organ Behav Hum Decis Process 1998;76:149-88.

247. Bing-You RG, Paterson J, Levine MA. Feedback falling on deaf ears: resident’s receptivity to feedback tempered by sender credibility. Med Teach 1997;19:40-44.

248. Watling C, Driessen E, van der Vleuten CPM, Lingard L. Learning from clinical work: the roles of learning cues and credibility judgements Med Educ 2012a; 46:192–200.

249. Watling C, Driessen E, van der Vleuten CPM, Vanstone M, Lingard L. Understanding responses to feedback: the potential and limitations of regulatory focus theory. Med Educ 2012b; 46:593–603.

250. Kirkpatrick DL, Kirkpatrick JD. The four levels: an overview. In: Kirkpatrick DL, Kirkpatrick JD. Evaluating training programs. The four levels. San Fransisco: Berett-Koehler Publishers; 2006.

251. Cook DA, Bordage G, Schmidt HG. Description, justification and clarification: a framework for classifying the purposes of research in medical education. Med Educ 2008;42:128-133.

252. Moher, D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. Ann Intern Med 2009;151:264-269.

253. Berkely Library. Finding Historical Primary Sources [Internet]. 2013 [updated 2013 Jan 28; cited 2013 December 28] Available from: http://www.lib.berkeley.edu/alacarte/subjectguide/163-Finding-Historical-Primary-Sources].

254. Glass GV. Primary secondary and meta-analysis of research. Educ Res 1976;5(10):3-8.

255. Greenalgh P. How to read a paper: getting your bearings (deciding what the paper is about).

BMJ 1997;315:243.

256. Gambling T, Brown P, Hogg P. Research in our practice- a requirement not an option: discussion paper. Radiography 2003;9:71-76.

257. Ferguson MH, Bargh JA. How social perception can automatically influence behavior. Trends Cogn Sci 2004;8(1):33-39.

258. Bargh JA, Chartrand T. The unbearable automaticity of being. Am Psychol 1999;54:462–479.

259. Bargh JA, Ferguson MJ. Beyond behaviorism: on the automaticity of higher mental processes. Psychol Bull 2000;126: 925–945.

260. Comrey AL. Factor analytic methods of scale development in personality and clinical psychology. J Consult Clin Psychol 1988;56:754-761.

261. Gorsuch RL. Exploratory factor analysis: its role in item analysis. J Pers Assmnt 1997;68:532560.

45 General introduction

262. Henson RK, Roberts JK. Use of exploratory factor analysis in published research: common errors and some comment on improved practice. Educ Psychol Meas 2006;66:393-416.

263. Moher D, Schulz KF,AltmanD. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. JAMA 2001;285:1987-1991.

What is feedback in clinical education?

Published as: Van de Ridder JMM, Stokking KM, McGaghie WC, Ten Cate OThJ. What is Feedback in Clinical Education? Med Educ 2008;42:189-197.

Chapter 2

48

Abstract Purpose Feedback is important in clinical education. However, the medical education literature provides no consensual definition of feedback. The aim of this study is to propose a consensual, research-based, operational definition of feedback in clinical education. An operational definition is needed for educational practice, teacher training, and for research on the effectiveness of different types of feedback.

Method A literature search about definitions of feedback was performed in general sources, meta-analyses, and literature reviews in the social sciences and other fields. Feedback definitions given from 1995-2006 in the medical education literature are also reviewed.

Results Three underlying concepts were found, feedback as: (a) information; (b) reaction, including information; and (c) a cycle, including both information and reaction. In most medical education and social science literature feedback is usually conceptualized as information only.

Comparison of feedback definitions in medical education reveals at least nine different features.

The following operational definition is proposed. Feedback is: Specific information about the comparison between a trainee’s observed performance and a standard given with the intent to improve the trainee’s performance.

Conclusions Different conceptual representations and the use of different key features might be a cause for inconsistent feedback definitions. Characteristics, strengths, and weaknesses of this research-based operational definition are discussed.

2.1 Introduction 49 What is feedback in clinical education?

eedback is crucial in clinical learning situations judging from the number of publications about feedback and related topics in medical education.1-3 Medical educators frequently believe they give feedback to medical trainees while trainees report that feedback is rare.4-8 To illustrate, Sender-Liberman et al. found that, while 90% of attending surgeons reported they gave feedback successfully, only 17% of their residents agreed with this assertion.4 This illustrates that agreement about the meaning of feedback is not evident. Clinical education is weakened when teachers, supervisors, students, and trainees do not agree about the definition and use of feedback as an educational tool.

In addition, research on effective feedback cannot be done without agreement about what it means. A clear, operational definition of feedback is needed.

The idea of feedback has a long history. Feedback as a feature of medical teaching is discussed in the writings of Hippocrates and other prominent ancient Greek physicians.9 The concept of feedback is now used in many scientific fields including mathematics, engineering, social science, logic, biology, and econometrics.10 The contemporary use of “feedback” dates from the beginning of the 20th Century. It was introduced in electronics in 1920 defined as, “The return of a fraction of the output signal from one stage of a circuit . . . to the input of the same or a preceding stage . . . tending to increase or decrease the amplification.”11 Electronic feedback was later described in 1936 as, “The effect whereby sound from a loudspeaker reaches a microphone feeding the speaker thereby distorting the sound and typically generating a screeching or humming noise.”11 A social science definition of feedback was proposed in 1943 stating, “... feed-back [signifies] that the behavior of an object is controlled by the margin of error at which the object stands at a given time with reference to a relatively specific goal.”12 In this definition feedback is viewed as a cycle that connects input and output. The cycle concept expanded over time and feedback in the social sciences also became “information” and “reaction.”13 The term feedback is now used and interpreted in many different ways.

There seems to be little consensus about the definition of feedback.10-11, 13-17 The aim of this report is to propose a research-based operational definition of feedback for learning situations in clinical education. To achieve this, we addressed four questions.

1. What is the most general conceptual formulation of feedback?

2. Which approaches to feedback are most commonly used in social sciences and medical education?

3. Which characteristics of feedback are commonly used in definitions about the learning process in medical education?

4. Which of these concepts and characteristics contribute most to an unambiguous description of feedback for clinical education?

2.2 Methods

Chapter 2

50

Questions 1 and 2 were investigated by three literature searches, performed in general literature sources, in the social science literature, and specifically in the area of medical education (supplementary material).

General literature To determine general conceptual formulations of the term feedback, definitions were collected from dictionaries (n=18), encyclopedias (n=11), lexicons (n=2) and handbooks (n=5) of different scientific fields available in the library collection of Utrecht University.

Social science literature Literature searches in the ERIC and PsycINFO databases were performed focusing on meta-analyses and literature-reviews.

Medical education literature Further searches in the ERIC, PsycINFO and MEDLINE databases focused on the term feedback in a supervisor-learner situation in medical education.

Feedback should be a defining theme in journal articles, Medical Subject Headings (MeSH), thesaurus terms, and titles of articles—“the shortest possible abstract”18— to be used as criteria to perform a literature search.

Table 1 in the supplementary material (online) describes the second and third search strategies. Feedback was the only term used in these searches to insure focus on its definition and meaning. Other terms including knowledge of results, reinforcement, Table 2.1. Characteristics and examples of concepts underlying feedback definitions Concept

Characteristic

Example of feedback definitions

Information

Focus is message content

“Feedback is information on progress of teaching and learning provided through various methods of assessment.”19 “Feedback is information provided to the learner concerning the correctness, appropriateness or accuracy. In short feedback is information about a learner’s performance.”20

Reaction

P) feedback provider R) feedback recipient

Focus is interaction with information Cycle

“The interchange of information on the part of human beings in a communication or problem-solving situation.”21 “A direct response by an individual or group to another person’s behavior, such as the reactions of an audience to a speaker’s remarks.”22

“In every instance, part of the output is fed back as new input to modify and improve the subsequent output of a system.”23 “Error correcting information returned to the control center of a servomechanism (or to the nervous Focus is receiving information, responding system and brain of a living organism) enabling it to to data, and improving response quality offset deviations in its course toward a particular goal.”22

2.3 Results General conceptual formulation of feedback A comparison of feedback definitions from 36 dictionaries, encyclopedias, lexicons and other general sources in medicine, biology, music, linguistics, communications, and programs in social sciences, leads to multiple feedback definitions. Three concepts dominate: (a) feedback as information, as a (b) reaction where information is included, or as a (c) cycle, involving information and reaction. Table 2.1 gives examples of the three approaches.

Feedback as information has message content as its focus. Central to feedback as a reaction is interaction, a process of information delivery and reception. Feedback as a cycle includes both information and reaction features but also includes a consequence or outcome of the message, e.g., response improvement. In addition, feedback as information is discrete while both the reaction and cycle formulations are processes.10 Feedback definitions in social science The social science literature search produced 133 reviews or meta-analyses. Articles on feedback within a “learning” situation were found in the context of education (n=82), clinical psychology or therapy (n=24), and the workplace (n=10). Most reviews describe written, oral, graphic or video forms of feedback to convey information about a performance. In addition, sensory feedback, bio[logical] feedback, and auditory feedback were described in 2, 12, and 3 reports, respectively.

Authors used four different strategies to address defining feedback. First, most do not define the term.24 Instead, they describe one or more characteristics of feedback such as its purpose, target, content, dimensions, type, and source. Second, other writers do not specifically define feedback but refer to definitions of feedback from the literature15, 17 and discuss their limitations.13, 25-26 Third, some define a derivative of feedback, for example feedback interventions27 and 360-degree feedback. A fourth group defines feedback explicitly. An overview of definitions is provided in table 2.2 of the supplementary materials. Feedback is mostly represented by the information concept28-38 (n=8), followed by reaction39-43 (n=5), and cycle44-48 (n=3). One definition combines these three concepts.49 Feedback definitions in medical education The focus in medical education literature is on feedback in a learning context, i.e., a situation in which two parties—a supervisor and trainee—aim to improve trainee knowledge and skills. We excluded articles that did not meet the criterion of a learning context based on titles and abstracts. Included were feedback interventions, guidelines about providing and receiving feedback, and perceptions of feedback.

51 What is feedback in clinical education?

reward, formative assessment, and appraisal were dropped. Clear inclusion and exclusion criteria were listed and described to decide on feedback definitions.

The content of different definitions in the medical education literature was compared to feedback characteristics to answer the third question. An answer to the fourth question required synthesis and evaluation of the different concepts and elements to formulate an operational definition of feedback.

Chapter 2

52

Concepts Feedback is mostly described as information (n= 7) and reaction (n= 4). Often (n=8) authors cite or base their definition on an existing definition4, 50-56 Ende is cited twice, but differently.57 His article describes important elements in the feedback process but he does not explicitly define the term.

Characteristics of feedback definitions Nine characteristics are evident when the definitions in table 2.2 are compared. All definitions state which concept is used—information or reaction. In addition, information is included about some of the following characteristics: 1. Content of information that should be conveyed: cognitive; evaluative; or about a standard, results, effects, behavior, or the feedback recipient; 2. Aim of the feedback: motivational, for improvement, or to promote reflection; 3. Feedback recipient: the person to whom the information is sent; Table 2.2. Conceptual categorization of feedback definitions in medical education literature Feedback definitions in the medical education learning context

Concept and characteristics

“Any information that is provided to the performer of any action about that performance”(p.509)51, 60

Information [1,3]

“Feedback is information about the result of a performance and this is often about a consultation and/or skill that has been performed by the learner and observed by the teacher.” Information (p. 691)52 [1,3,5,6] Based on: Shorter Oxford English Dictionary “[…] information from instructor to learners about their past performance on the wards which serve to enhance or modify future actions of learners[…]” (p.332)53, 61

Information [1,2,7]

“[…] information that gives learners knowledge of the results of their study and clinical work.” (p. 632)54, 62

Information [1,2]

Formative feedback: “information about how successfully something has been or is being done”63 and is provided to help individuals improve their performance.”(p.66)55

Information [1,2]

“Feedback is defined as specific information presented to a learner to promote reflection on performance. It focuses on both what was done and what the consequences of the action might be. The ultimate goal is to help learners in establishing their own goals and critiquing their own performance.” (p.470 )4, 64

Information [1,2,3,4]

“[…] information describing students’ or house officers’ performance in a given activity that is intended to guide their future performance in that same or in a related activity. It is a key step in the acquisition of clinical skills.” (p.777. )56-57

Information [1,2]

“[…] an informed, nonevaluative, and objective appraisal of performance that is aimed at improving clinical skills rather than estimating the student’s personal worth.” (p.1)50, 57

Reaction [1,4]

“Giving trainees feedback means letting them know, in a timely and ongoing way, how they are performing.” (p.267)58

Reaction [1,3,8]

“Audit and feedback involves collecting information on performance measures for individual physicians and then providing this information to the physicians with comparisons to colleagues or other standards.” (p.738)59

Reaction [1,5,6,9]

Characteristics: 1 = content, 2 = aim, 3 = recipient, 4 = form, 5 = preparation, 6 = source, 7 = provider, 8 = communication conditions, 9 = contextual factors

Definition of feedback in clinical education Based in the review above, we propose to define feedback in clinical education as, Specific information about the comparison between a trainee’s observed performance and a standard given with the intent to improve the trainee’s performance.

How is this definition of feedback novel, compared to earlier statements? Feedback in clinical education is seen as a form of communication. Dissection of the definition into ten key elements provides clarity. The elements are: (a) clinical education, (b) performance and task, (c) trainee, (d) feedback provider, (e) comparison between observed performance and a standard, (f) observation, (g) standard, (h) specific information, (i) intention, and (j) improvement.

Clinical education This refers to the “on-the-job” context in general at a hospital or clinic such as on the ward, in an outpatient clinic, operating room, GP consultation room, or any other place where the trainee is involved in patient care.

Performance and task In the clinical setting, many tasks are suitable for providing feedback: history taking, clinical examinations, reporting during patient handover, and working with colleagues on a team. The task must be observable. Even clinical reasoning, done aloud, is subject to feedback.

Trainee In clinical education the feedback recipient is the trainee. The trainee can be anyone in a clinical learning situation, be this student, clerk, resident or other health care trainee.

The trainee receives feedback to acquire the knowledge, skills, and attitudes necessary to become a superb practitioner.

Feedback provider The feedback provider is a clinical teacher conceived broadly. This may be a clinical staff member who is formally responsible for clinical teaching. The feedback provider may also be a resident acting as a clinical teacher for students, or an attending physician for residents. Essentially a feedback provider is someone who can envision a standard against which to compare the trainee’s performance. Key is this expertise of the feedback provider. This broad concept of ‘clinical teacher’ may therefore extend to anyone in this position.

53 What is feedback in clinical education?

4. Form of the information should be communicated: oral, written, specific, nonevaluative; 5. Preparation before the information can be conveyed: collecting results or observing the subject; 6. Source of the information: from the person him/herself (internal feedback), task results, or from another source (external feedback); 7. Feedback provider: the person who gives the information; 8. Communication conditions: timeliness, directness; 9. Contextual factors such as the place where feedback is given.

These different characteristics can be described even when definitions have the same conceptual meaning. Most definitions include information about the type and content of the information and describe its aim.

Chapter 2

54

Comparison between performance and a standard The difference between performance and a standard determines the content of the feedback. This gap may be large or small, and positive or negative. In contrast with cybernetic feedback definition of a negative feedback loop, feedback in clinical education for trainees who outperform a standard or expected level does not aim at decreasing the difference but may stimulate further development.

Observation One cannot address feedback in clinical education when the trainee’s performance has not been observed. The way in which the feedback provider observes depends on the nature of the task. Direct observation occurs when the observer and the feedback provider are the same person. Observation can involve skill performance but also reading written products. In both situations the feedback provider receives first hand information about a task. Indirect observation occurs when the observer and feedback provider are different. An example is when a supervisor bases feedback on comments from other observers in a multisource situation.

Standard The feedback provider needs to know the standard of comparison to describe the difference between a performance and its outcome. Examples of standards include a protocol where a performance is described, the performance of colleagues, a trainee’s previous performance, and clinical teachers’ opinions about the performance standard.

Standards vary from objective to subjective and from absolute to relative.

Specific information Feedback must contain a minimum amount of specification to serve its purpose. Utterances that cannot be understood by the feedback recipient in behavioral terms, i.e. in terms of what has been done well or what could be improved, should not be called feedback.

Intention An intention to give feedback for trainee performance improvement characterizes the learning situation. This might be seen in the amount of time, the tone and accuracy in which the information is conveyed, or readiness to observe the learner again.

Improvement The aim of feedback is trainee performance improvement. As explained earlier, improvement is not limited to a fixed endpoint. Continuous development of expertise makes feedback valuable in nearly all situations.

Table 2.3. Weak versus strong modalities of feedback, based on its definition Weak modality

Strong modality

Competencies that are not observable

Well observable tasks and competencies

Uninformed or non-expert observer

Expert observer and feedback provider

Global information

Highly specific information

Implicit standard

Explicit standard

Second hand information

Personal observation

No aim of performance improvement

Explicit aim of performance improvement

No intention to re-observe

Plan to re-observe

Weak versus strong feedback

55

Our definition is meant to improve communication about the feedback concept and to serve research purposes. Given the ten constituting elements, we rather speak of strong and weak feedback, than totally exclude forms of feedback that do not meet our definition. Some of those elements can be seen as a variable that can vary from weak to strong. Our definition could include and exclude some commonly used communication forms as “feedback in clinical education”. This is not the purpose of our contribution. In many cases the distinction will not be clear cut. However listing these elements allows us to distinguish “weaker” and “stronger” types of feedback. In table 2.3 we show examples of these extremes.

What is feedback in clinical education?

2.4 Discussion An operational definition should increase conceptual understanding. An operational definition provides insight about a concept’s characteristics and explains concept specificity, precision, and generalizability;65 reveals tacit assumptions or presumptions; discloses premises;13, 17 and makes concept features plain.66 An operational definition also specifies procedures to identify or to produce the defined concept reliably.67 Does our definition meet these criteria?

Operational definition Four procedures to produce or identify feedback are described in the definition. The procedures are: 1. Information gathering: the definition states that a trainee’s performance should be observed.

2. Content: the content of the information is about the comparison between trainee’s observed performance and a standard.

3. Direction of the provided information: the clinical teacher provides information toward the trainee.

4. Intention of providing information: clinical teacher’s intent is to improve trainee performance. This is not always observable directly, but can be discerned by asking about the feedback provider’s motives when feedback is given or from indirect observation.

An operational definition should be reliable. It should be clear, measurable, and reproducible.67 Feedback data are collected by observation, its content is about a gap between trainee performance and a standard, and the direction of the feedback is from a clinical teacher toward a trainee. This is measured by asking participants or observing the feedback process. However, an intention is not directly observable. This must be probed or observed indirectly.

Conceptual representation

Chapter 2

56

A conceptual definition refers to a general idea behind a definition. Feedback in medical education refers chiefly to information. Defining feedback as a cycle also has advantages. It is a rich definition because it includes information and reaction. The cycle analogy gives emphasis to outcomes compared with feedback as information or reaction. From a linguistic perspective it better represents the meaning of the word feedback. The word feedback in itself suggest movement (process of feeding) and cycle (back in itself refers to a beginning). So from this point of view feedback as a cycle is most sufficient. Finally, it is the original definition.10-11, 16 Why do we not consistently use the original definition and define feedback as a cycle? Defined as a cycle, feedback includes the trainee’s behavior adjusted after observation and information exchange by the feedback provider.

Historically, the exact meaning of feedback in social science has evolved and expanded in scope over the past 30 years.13 Methodologically, a clear operational definition improves scholarly understanding and exchange. Table 2.2 reminds us that a concept may have several operational definitions.

Our definition has been specified along a weak-strong dimension. This signifies a linguistic variability of the concept. We do not intend to equalize this with “worse” or “better” feedback. Weak and strong are only related to our definition. Another question is: which feedback is effective and which is not effective. This actually is a research question. Many variables have been mentioned in the literature as possibly influencing the effectiveness of feedback. To describe and validate these variables is a task for future research.

We believe that our definition of feedback in clinical education provides a clear point of departure for teacher training and research on the effectiveness of variables in feedback processes. The definition discriminates authentic feedback statements from those that do not qualify and contrasts weak and strong feedback.

References 1.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.

23.

24.

57 What is feedback in clinical education?

2.

Veloski J, Boex JR, Grasberger MJ, Evans A, Wolfson DB. Systematic review of the literature on assessment, feedback and physicians’ clinical performance: BEME Guide No. 7. Med Teach 2006;28(2):117-128.

Rolfe IE, Sanson-Fisher RW. Translating learning principles into practice: a new strategy for learning clinical skills. Med Educ 2002;36:345-352.

Irby DM. Teaching and learning in ambulatory care settings: a thematic review of the literature. Acad Med 1995;70:898-931.

Sender-Liberman A, Liberman M, Steinert Y, McLeod P, Meterissian S. Surgery residents and attending surgeons have different perceptions of feedback. Med Teach 2005;27(5):470472.

McIlwrick J, Nair B, Montgomery G. “How Am I doing?” Many problems but few solutions related to feedback delivery in undergraduate psychiatry education. Acad Psychiatry 2006;30(2):130-135.

Gil D, Heins M, Jones PB. Perceptions of medical school faculty members and students on clinical clerkship feedback. J Med Educ 1984;59:856-864.

Sheenan TJ. Feedback: giving and receiving. J Med Educ 1984;59:913.

Isaacson JH, Posk LK, Litaker DG, Halperin AK. Resident perception of the evaluation process. J Gen Intern Med 1995;10(4, Suppl):S89.

Puschmann T. A History of Medical Education. New York: Hafner Publishing Co., 1966.

[Originally published 1891] Richardson GP. Feedback Thought in Social Science and Systems Theory. Philadelphia: University of Pennsylvania Press 1991.

Oxford English Dictionary Online. Oxford University Press, 2006. Available at http://dictionary.oed.com/cgi/entry/50083243?query_type=word&quote Accessed 13 November 2006.

Rosenblueth A, Wiener N, Bigelow J. Behavior, purpose, and teleology. Phil Science 1943;10(1):8-24.

Clement DA. Frandsen KD. On conceptual and empirical treatments of feedback in human communication. Commun Monogr 1976;43:11-28.

Bailey KD. Reviews. Contemp Sociol 1992;21:546-547.

Alvero AM, Bucklin BR, Austin J. An objective review of the effectiveness and essential characteristics of performance feedback in organizational settings. J Organ Beh Manage 2001;21(1):3-29.

Ramaprasad A. On the definition of feedback. Beh Science 1983;21:4-13.

Frandsen KD, Millis MA. On conceptual, theoretical and empirical treatments of feedback in human communication: fifteen years later. Commun Reports 1993;6:79-91.

Bordage G, McGaghie WC. Title, author and abstract. Acad Med 2001;76(9):943-947.

Terry Page G, Thomas JB. International Dictionary of Education. London: Kogan, 1979; p.134.

Meyer RE. Feedback. In: Anderson LW (ed), International Encyclopedia of Teaching and Teacher Education, 2nd ed. Oxford: Pergamon, 1995; p. 249-251.

Good CV, Merkel WR, (eds) Dictionary of Education, 3rd ed. New York: McGraw-Hill 1973; p.

227-228.

Corsini RJ. The Dictionary of Psychology. Philadelphia: Brunner/Mazel Taylor& Francis 1999; p.370-371.

The New Encyclopaedia Britannica, 15th ed. Chicago: Encyclopaedia Britannica 1992; p. 715.

Smither JW, London M, Reilly RR. Does performance improve following multisource feedback? A theoretical model, meta-analysis, and review of empirical findings. Personnel Psychol 2005;58(1):33-66.

Chapter 2

58

25. Nadler DA. The effects of feedback on task group behavior: a review of experimental research. Organ Beh Hum Perf 1979;23:309-338.

26. Madzar S. Feedback seeking behavior: a review of the literature and implications for HRD practitioners. Hum Res Devel Quart 1995;6(4):337-349.

27. Kluger AN, DeNisi A. The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychol Bull 1996;119(2):254-284.

28. Dana RH, Graham ED. Feedback of client-relevant information and clinical practice. J Person Assess 1976;40(5):464-469.

29. Clairborn CD, Goodyear RK, Horner PA. Feedback. Psychotherapy 2001;38(4):401-405.

30. Blanchard EB, Young LD. Clinical applications of biofeedback training: a review of evidence.

Arch Gen Psych 1974;30(5):573-589.

31. Azvedo R, Bernard RM. A meta-analysis of the effects of feedback in computer-based instruction. J Educ Comput Res 1995;13(2):111-127.

32. Cohen VB, A reexamination of feedback in computer-based instruction: implications for instructional Design. Educ Technol 1985;25(1):33-37.

33. Tait K, Hughes IE. Some experiences in using computer-based learning system as an aid to self-teaching and self-assessment. Computers Educ 1984;8(3):271-278.

34. Taylor DB. Computer based testing in an introductory marketing course. Program Learn Educ Technol 1983;20:76-89.

35. Mead DE. Reciprocity counseling: practice and research. J Marital Fam Ther 1981;7(2):189200.

36. Jacobson NS, Margolin G. Marital therapy: strategies based on social learning and behavior exchange principles. New York: Bruner/Mazel, 1979, p.190.

37. Mechling L. The effect of instructor-created video programs to teach students with disabilities: a literature review. J Spec Educ Tech 2005;20(2):25-26.

38. Raymond DD, Dowrick PW. Affective response to seeing oneself for the first time on unedited videotape. Counsel Psychol Quart 1996;6(3):193-200.

39. Lundgren DC. Social feedback and self-appraisals: current status of the Mead-Cooley Hypthesis. Symbol Inter 2004;27(2):267-286.

40. Shake MC. Teacher interruptions during oral reading instruction: self-monitoring as impetus for change in corrective feedback. RASE: Remed Special Educ 1986;7(5):18-24.

41. Hoffman JV. On providing feedback to reading miscues. Reading World 1979;18:342-350.

42. Getsie RL, Langer P, Glass GV. Meta-analysis of the effects of type and combination of feedback on childrens’s discrimination learning. Rev Educ Res 1985;55(1):9-22.

43. Skiba RJ. Nonaversive procedures in the treatment of classroom behavior problems. J Spec Educ 1986;19(4):459-481.

44. Spink A, Losee RM. Feedback in information retrieval. Ann Rev Info Sci Tech (ARIST) 1996;31:33-78.

45. Weiner N. The Use of Human Beings: Cybernetics and Society Boston: Houghton Mifflin, 1950, p. 71.

46. Cobb DE, Evans JR. The use of biofeedback techniques with school-aged children exhibiting behavioral and/or learning problems. J Abnor Child Psych 1981;9(2):251-281.

47. Schwartz GE, Beatty J, eds. Biofeedback: Theory and Research. New York: Academic Press, 1977.

48. Balzer WK, Doherty ME, O’Connor R. Effects of cognitive feedback on performance. Psych Bull 1989;106(3):410-433.

49. Borden GJ. An interpretation of research on feedback interruption in speech. Brain Language 1979;7(3):307-319.

50. Richardson BK. Feedback. Acad Emerg Med 2004;11(12):1-5.

51. Rushton A. Formative assessment: a key to deep learning? Med Teach 2005;27(6):509-513.

59 What is feedback in clinical education?

52. Moorhead R, Maguire P, Thoo SL. Giving feedback to learners in the practice. Aust Fam Physician 2004;33(9):691-695.

53. Paul S, Dawson KP, Lamphaer JH, Cheema MY. Video recording feedback: a feasible and effective approach to teaching history-taking and physical examination skills in undergraduate paediatric medicine. Med Educ 1998;32:332-336.

54. Parikh A, McReelis K, Hodges B. Student feedback in problem based learning: a survey of 103 final year students across five Ontario medical schools. Med Educ 2001;35(7):632-636.

55. Chur-Hansen A, Koopowitz LF. Formative feedback in teaching undergraduate psychiatry.

Acad Psychiatry 2005;29(1):66-68.

56. Milan FB, Parish SJ, Reichgott MJ. A model for educational feedback based on clinical communication skills strategies: beyond the “feedback sandwich.” Teach Learn Med 2006;18(1):4247.

57. Ende J. Feedback in clinical medical education. JAMA 1983;250:777–781.

58. Vickery AW, Lake FR. Teaching on the run tips 10: giving feedback. Med J Aust 2005;183(5):267268.

59. Paxton ES, Hamilton BH, Boyd VR, Hall BL. Impact of isolated clinical performance feedback on clinical productivity of an academic surgery faculty. J Am Coll Surg 2006;202(5):737745.

60. Black P, William D. Assessment and classroom learning. Assess Educ: Prin, Policy & Prac 1998;5(1):7-68.

61. Nadler DA. Feedback and Organizational Development: Using Data-Based Methods. Menlo Park, CA: Addison-Wesley, 1977, p. 67.

62. McGaghie WC, Frey JJ, eds. Handbook for the Academic Physician. New York: Springer-Verlag, 1986.

63. Sadler DR. Formative assessment and the design of instructional systems. Inst Sci 1989;18:119–144.

64. Westberg J, Jason H. Providing constructive feedback. In: Jason H, Westberg J, eds. A CIS Guidebook for Health Professions Teachers. Boulder, CO: Center for Instructional Support, University of Colorado, 1991.

65. Fodor JA, Garrett MF, Walker ECT, Parkes CH. Against definitions. Cognition 1980;8:263267.

66. Medin DL, Lynch EB, Solomon KO. Are there kinds of concepts? Ann Rev Psychol 2000;51:121147.

67. Dodd SC. Operational definitions operationally defined. Am J Sociol 1943;48(4):482-491.

Variables that affect the process and outcome of feedback, relevant for medical training: a meta-review

Submitted as: Van de Ridder JMM, McGaghie WC, Stokking KM, Ten Cate OthJ. Variables that affect the process and outcome of feedback, relevant for medical training: a meta-review. Med Educ 2015; (in press)

Chapter 3

62

Abstract Introduction Feedback is considered important in medical education. The literature is not clear about the mechanisms that contribute to its effects which are often small to moderate and at times contradictory. A variety of variables seem to influence its impact on learning. The aim of this study is to determine which variables influence the process and outcomes of feedback in settings significant for medical education.

Method A myriad of feedback studies have been carried out. To determine the most researched variables we limited our review to meta-analyses and literature reviews published in the period 1986 to February 2012.

According to our protocol, we first identified features of the feedback process that influence its effects and subsequently variables that influence these features. We used a chronological model of the feedback process to categorize all variables found.

Results A systematic search of ERIC, PsycINFO and Medline yielded 1101 publications, which we reduced to 203, rejecting papers on six exclusion criteria. From these, 46 studies met the inclusion criteria.

In our four-phase model, we identified 33 variables linked to task performance (e.g. task complexity, task nature) and feedback reception by trainees (e.g. self-esteem, goal setting behaviour), and observation (e.g. focus, intensity) and feedback provision by supervisors (e.g. form, content) that influence the subsequent effects of the feedback process. Variables from all phases influence the feedback process and effects, but variables that influence the quality of the observation and rating of the performance dominate the literature. There is a paucity of studies that address other, seemingly relevant variables.

Conclusions The larger picture of variables that influence the process and outcome of feedback, relevant for medical education, shows many open spaces. We suggest that targeted studies be carried out to expand our knowledge of this important field of feedback in medical education.

3.1 Introduction

Figure 3.1. Representation of the feedback process -phases A1, B, C and D- and the feedback effect which can be found when two performances are compared (Δ A2-A1)

63 Variables that affect the process and outcome of feedback, relevant for medical training: a meta-review

eedback is meant to improve employees’ and learners’ performance and to implement procedures. Feedback in clinical education may be defined as: “Specific information about the comparison between a trainee’s observed performance and a standard given with the intent to improve trainee’s performance”.1 The literature on feedback is abundant.2-6 Several studies show impact of feedback on motivation and performance.4-8 In health care education, medical students receive feedback in the clinical context, a dynamic learning environment with lack of continuity, in which the relationship between patient, clinician and student is key.9-10 Feedback is often provided by different clinicians who are usually not trained to do this, and allocating time to provide feedback is not common practice.10-11 Feedback is a teaching and learning tool, with high educational impact.10, 12-15 As feedback is considered a cornerstone in clinical education16 it is important that clinicians understand the feedback process and learn which variables relate to assessment and feedback procedures and its effects. This knowledge can also support the skills of clerks and residents as feedback recipients. This together should raise the quality of feedback procedures. Then we can use it optimally and it will result in better collaboration, more competent physicians and better patient care.

The effects of feedback are equivocal and confusing,4 because feedback can both increase and decrease motivation and performance.8 While Hattie & Timperley report large overall effect sizes,5 Kluger & DeNisi and Ivers et al. report that the impact of feedback is small to moderate,8, 17 One explanation for these inconsistent findings may be the existence of variables that influence the process and effect of feedback.

The feedback process can be considered a communication process with a loop nature, consisting of five phases6, 18-22 depicted in figure 3.1.

Chapter 3

64

In phase A1 the Feedback recipient (FR) receives instructions and performs a task according to certain standards. Phase B relates to feedback provider (FP)’s observation and interpretation of FR’s task performance while comparing it to an explicit or implicit standard. During phase C feedback is communicated towards the FR. In phase D the FR receives the feedback and interprets it.

When the same task is performed again, the cycle is closed (phase A2). The feedback effect (Δ A2-A1) can be found when performances A1 and A2 are compared. Effects of feedback may be found in FR’s cognition, attitude, and/or performance change as a consequence of received feedback.

Figure 3.1 visualises the chronology of the feedback process in a schematic model. In reality, phase A and B will mostly take place simultaneously, as will C and D. According to this model, phase B, the assessment phase, in which the observation and rating takes place, directly influences phase C, in which the feedback is communicated. For example, when two clinicians independently observe the same performance but one has observed the clerk many times, and the other only once, they might observe the same performance but due to the different frames of reference they have, their observations may be different. This may turn out in two different feedback messages based on the same observation. When the interrater reliability appears low between two FP’s observing a task and this observation is used for feedback, the FR will receive two different messages, which makes corrective action difficult. For a more elaborate explanation we refer to the Supplementary Material, section I. Each phase can influence all subsequent phases. The phases A, B, C, and D together influence the effect of feedback.

Feedback and models of feedback stem from a technical background21 and over time feedback is applied in various disciplines, such as communication, teaching and learning.1 This feedback process model is based on Johnson’s and Lasswell’s communication models.23-27 It has similarities with other feedback models used in the communication, teaching and learning context.2-3, 28-31 The aim of our study is to provide an overview of variables that influence the process of feedback, the effects of the feedback, or both, and describe how this influence relates to the different phases of the feedback process and the feedback effect. Cross-disciplinary literature from education, psychology, labour, and management studies, which take both the process and the effects of feedback into account - were brought together.

From this overview directions for further feedback research can be derived, both inside and outside healthcare education.

We aimed to answer the following research questions: Which variables in the feedback process influence either the one or more subsequent phases in this process or a second next performance on the same task? What is the direction of the influence of each variable on the subsequent phases of the feedback process or on the final effect?

3.2 Method This article was prepared using most reporting conventions described in the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) Statement for meta-analyses and systematic reviews.32 Study eligibility and identification To determine the most researched variables we limited our review to literature reviews and meta-analyses. The advantage of reviews and meta-analyses over individual stud-

Exclusion and inclusion criteria Six criteria were used to exclude articles based on their titles and abstracts: 1. Non-relevant population. Studies based on animal data or on children, handicapped persons or patients; 2. Non-behavioral topics. Studies focusing on descriptions, reviews, comparisons, and evaluations of ‘non-behavioral’ topics: such as (student) feedback on teaching/learning programs, materials, guidelines, curricula, et cetera.

3. Methodological focus. Studies focusing on methodology of meta-analyses and reviews in particular or on methodology problems in general.

4. Educational procedures focus. Studies focusing on differences in methods (e.g., traditional-authentic), models, measurement, tools and approaches of assessments and evaluation of performances and competencies; 5. Non-relevant miscellaneous topics. Studies with other topics than feedback as their focus, such as job commitment and turnover; personnel selection; grading and scoring; self-assessment; influences or predictions on job/study performance; physical processes such as biofeedback; video-feedback and rewards.

6. Non-review nature. Studies that provide comments or critique on reviews, or studies in which reviewing the literature was not the main purpose of the study.

Subsequently, from all articles retained this way, we included all studies meeting at all of the following criteria: 1. Studies which use meta-analyses or systematic literature reviews of empirical research as their methodology, and have a good quality, by reporting search terms,

65 Variables that affect the process and outcome of feedback, relevant for medical training: a meta-review

ies is twofold: they summarize common themes and conclusions, and focus on a usually well-defined domain. This reduces the likelihood of reporting atypical or misleading results.33 We conducted searches of the ERIC, PsycINFO and Medline databases on meta-analyses and literature reviews in English language, peer-reviewed journals, published in the period 1986 to February 2012 (figure 1.2). To yield a complete identification of relevant articles, the search strategy was developed and discussed with two experienced reference librarians (LO’D and RK).34 Dependent on the database, we used the following thesaurus terms and medical subject headings: “Reinforcement (psychology)”, “Reinforcement, Verbal”, “Reinforcement, Social” “Formative-Evaluation”, “Personnel-Evaluation”, “Student-Evaluation”, “Job-Performance”, “Informal-Assessment”, “Performance-Based-Assessment”, “Employee Performance Appraisal”, “Error-Correction”, “Feedback (psychological)”, “Performance-Tests”, “Knowledge-of-Results (psychology)”, “Delayed-Feedback”, combined with “State-of-the-Art-Reviews”, “Literature-Reviews”, “Meta-Analysis”. To retrieve results pertaining to an educational and psychological context we added in Medline also terms related to medical education (“Schools, Medical”, “Students, Medical”, “Clinical Competence”, “Internship and Residency”, ”Clinical Clerkship”, “Teaching”, “Education”, “education”[Subheading], “Hospitals, Teaching”, ”Competency-Based Education”, “Education, Continuing”, “Education, Medical, Continuing”, “Competency-Based Education”, “Education, Medical, Undergraduate” OR “Education, Medical, Graduate).

One rater (JMMvdR) excluded articles matching one or more of the exclusion criteria (below) based on the information of abstracts and titles. Three raters (JMMvdR, WCMcG and KMS) independently rated the remaining articles on the three inclusion criteria. Agreement on inclusion was reached by consensus.

Chapter 3

66

time span and sources. The use of predetermined inclusion criteria and especially exclusion criteria was also considered in itself a criterion for quality;35 2. Studies which population is healthy adults; 3. Studies with the topic of (a) the feedback process in general, (b) the process of providing information, (c) the process of receiving information or (d) the effect of the information on outcomes.

Process of selection of variables and synthesis of results We used an eight-step protocol to select variables and synthesize findings. The descriptive model (figure 3.1) was used as a tool to relate the variables to the different phases in the feedback process. The Supplementary Material Section II provides more details.

1) Each study was categorized in one or more phases of the feedback process or the feedback effect. We first determined the dependent variables or the outcome variables of the included studies [table S1 (available online) column H].

2) Dependent, independent, moderator, mediator and confounding variables were determined (table S1 column I). Moderator, mediator and confounding variables elucidate the causal process by which the independent variable influences the dependent variable.

Initial search ERIC; PsychINFO;Medline (1101)

-------

Application of exclusion criteria: Non relevant population (172) Non-behavioral topics (137) Methodological focus (20) Educational procedures focus (97) Non-relevant miscellaneous topics (320) Non-review nature (152) After application of exclusion criteria (203)

No full text article available (7)

Available for application of inclusion criteria (196) Application of inclusion criteria: -- relevant meta-analysis or review type -- population of healthy adults -- focus on general feedback process, information provision, information reception, effect of information on outcome

Figure 3.2. Search strategy and search results

Resulting studies examined (46)

3.3 Results Studies found Figure 3.2 summarizes the results of the search strategy. An initial search yield of 1101 publications was reduced to 46 relevant studies to be examined: 22 meta-analyses and 24 literature reviews. The dependent variable in one study were related to phase A, of thirteen studies to phase B, in four studies to phase C, in two studies to phase D and in 24 studies to the feedback effect, and the independent variables of six studies pertained to a combination of the phases and the effect (table S1 column E). Ten of the 46 studies were performed in a health care setting (table S1 column G).14, 16, 36, 43 Section IV in the Supplementary Materials document provides a detailed description of the evidence found in the literature reviews and meta-analysis about how each of the 33 variables influence the feedback process, the feedback effect, or both.

Variables found Based on the 46 reviews and meta-analyses, we found 33 variables related to task, standards and task performance (phase A1), task observation and interpretation (phase B), feedback communication (phase C), and feedback reception and interpretation (phase D), influencing the feedback process, the feedback effect, or both. A description of each variable can be found in table 3.1.

Phase A general context, task, standard and first task performance by FR The nine variables that relate to Phase A1 are mostly about the task: number of different tasks that need to be performed,44 complexity,8, 36, 45-50 nature,8, 50-53 subject matter,44-54 and the perception of the task.45 One variable is about the organization’s culture, the context in which the task will be performed.47, 49, 55 Three variables relate to the learner

67 Variables that affect the process and outcome of feedback, relevant for medical training: a meta-review

3) We assessed the unit of measurement for each study. In meta-analyses the effect sizes are the usual units of measurement. Effect size d (ESd) indicates the extent to which means differ. Effect size r (ESr) indicates the size of correlation effects between variables. In the literature reviews, units of measurements could be the number of studies that report a change, or percentages of studies in which effects were found. Narrative reviews just describe differences. We only reported variables which were clearly indicated as effective.

4) We identified and selected those variables which have a reported impact on the feedback process and feedback effect. In meta-analyses we only considered variables with a significant effect. In the quantitative and narrative reviews only those variables that were clearly reported as influencing the direction of the effect are included.

5) Variables that did not influence the feedback process and the feedback effect or both are reported in table S1 column J. 6) To detect which phase of the feedback process and feedback effect was influenced, we considered (i) which dependent variable was influenced and, (ii) in which phase of the feedback process or the feedback effect this variable could be classified.

7) To create a comprehensive overview, we clustered variables with similar meanings or similar content (table 3.1).

8) From the variables that reportedly influenced the phases of the feedback process or the feedback effect or both, we determined the direction of the effect. Table S1 describes the characteristics of the included studies.

who performs the task and will receive feedback: FR’s cultural background,55-57 age,54 and initial skills.36-45

Chapter 3

68

Phase B observation, interpretation and rating by FP Twelve independent variables are embedded in the phase of observation, interpretation and rating, as carried out by the feedback provider before the communication of the feedback. A first group of variables is about the use of observation and rating methods and instruments: training content and method,58-60 instrument,37, 46, 60-65 utility of Table 3.1. A description of the independent variables that influence the feedback process, the feedback effect, or both1 Variable Task variation Task complexity Task nature Task perception Task subject matter

Description The differences in level, subject matter, complexity of tasks performed during observation and rating.

The extent to which a task is easy or difficult to analyze, to understand or to perform.

The basic or inherent features, or characteristics of a tasks.

Examples: communication, memory, algorithm.

The interpretation or impression based on one’s understanding of the performed task.

The topic dealt with, or the subject represented in the task.

Example: mathematics, language, biology.

Culture and context

The circumstances that form the setting for the feedback process, and in terms of which it can be fully understood. Example: habits and customs of a country, or the working environment.

FR’s age

The length of time that the learner has lived.

FR’s cultural background

FP’s previously acquired understanding or knowledge that allows utterances, beliefs, and actions to have explicit meaning, especially related to cultural interpretation.

FR’s skills

The ability of the FP to perform the task: expertise.

Training content and method

The topics that are contained in the training. Example: rating errors, rating accuracy, rating instruments, etc. The procedure for approaching the goals of the training: through discussion, lectures, practice, observation, etc.

Instrument Assessment method (utility)

The tool or implement to observe and rate FP’s performance: Mini-CEX, checklists, etc.

The usefulness of methods for measuring the impact of a course or lesson on learners’ levels of attainment: observation, OSCE’s, knowledge tests, etc.

Rubrics

The written instructions provided for candidates as part of a test.

Purpose of observation

The reason for which the observation is done.

Example: research or administration.

Focus of observation

The center of interest during the observation.

Intensity of observation

The degree or strength of the observation: in real life, based on video tapes, focused on specific performances, one to one, in a group.

Type of standard

The category of the required or agreed level of quality or attainment used for FR’s observation or rating: result oriented measures of performances, amount of time spend on a task, subjective measures of performances with space for personal interpretation.

The effort put in to connected with somebody else.

FP’s position

The particular way in which the FP is placed, especially as it influences FP’s power to act/ Examples: subordinates, supervisor.

FP’s task familiarity

The extent to which the FP a task has encountered or experienced.

FP’s cultural background

FR’s previously acquired understanding or knowledge that allows utterances, beliefs, and actions to have explicit meaning, especially related to cultural interpretation.

Source of FB

A person or thing from which the feedback message originates or can be obtained. Examples: computer, peers, colleagues.

Medium of FB

The means by which the feedback message is communicated or expressed.

Form of FB

Style, design, and arrangement of the feedback message as distinct from its content. Examples of form: dialogue, as part of education, multi-facetted interventions, organized feedback meeting.

Content of FB Complexity of FB Timing of FB

The material, information dealt with in the feedback message, as distinct from its form or style.

The extent to which the feedback is easy or difficult to analyze, to understand.

Examples: difficult formulation, lengthy messages.

A particular point or period of time when feedback is given. Examples: immediate, delayed, after an item, after a test.

Frequency of FB

The rate at which the feedback occurs over a particular period of time.

Intensity of FB

The degree or strength of the feedback communication. Examples of aspects of intensity: over a long time period, by credible feedback providers, with rich content, with regular checks whether or not the message is understood, length of the feedback messages.

Time: feedback-performance2

Duration of the interval between receiving feedback and the next performance.

Activities: feedbackperformance2

Task performed in the interval between receiving feedback and the next performance.

FRs’ self-esteem

FR’s confidence in one’s own worth or abilities.

FRs’ goal setting behavior

FR’s ability to strive for an aim or desired result.

assessment method,14, 38-39, 53 and rubrics.66 A second group pertains entirely to the mode of observation: the purpose (why),46, 61-62 the focus (what),56-57, 63 the intensity (how),37, 45-46, 60, 62 and the standard which is used during the observation.61, 63-64 The variable about the time the FP has to build a relation with the FR61, 67-68 links with the variable FP’s position.

Three variables describe characteristics of the feedback provider: FP’s positions to1

The descriptions of the variables are based on the Oxford Dictionary of English89

Variables that affect the process and outcome of feedback, relevant for medical training: a meta-review

Time to build relation

69

wards the FR50-51, 60, 62, 65, 68 FP´s acquaintance with the observed task,50-51, 65, 68 and FP’s cultural background.55

Chapter 3

70

Phase C communication of the feedback message by the FP to the FR The eight variables we found in this category are about the person providing the feedback (source),16, 47 about the means (medium),8, 69 about the message form,16, 36, 40-41, 45, 52, 54 the message content,4, 8, 49, 52, 70-73 and the message complexity,4 about organizational aspects (timing,4, 52, 74-75 frequency8, 42, 49) and about the intensity.16, 36, 47, 54 Phase D reception, perception and interpretation of the feedback message by the FR Two variables pertain to the episode after receiving feedback and before the next performance (time interval: feedback-performance254, 74 and disturbing activities in the period between the feedback and performance274). The other two variables relate to the learner who receives the feedback: FR’s self-esteem and FR’s goal setting behavior.8 Direction of variables To answer the second research question we first consider the phases of the process and the feedback effect that are influenced by the 33 variables, then we explain the influence of these variables (table 3.2).

The main effect of feedback providing is that feedback is effective, and it improves performance,8, 16, 36, 40, 70, 76-77 for example safety related performances,76 work productivity,49 judgment abilities,70 learner’s goal setting abilities,4 and clinicians’ and physicians’ performance.16, 40 The impact of feedback is often small to moderate.8, 36, 43, 47, 52 Three variables influence the general context, task, standards and first task performance (table 3.2 column Phase A). Sixteen variables influence the observation, rating and interpretation by the FP (column Phase B). The columns of Phase C and Phase D in table 3.2 are nearly empty. We found only two variables (training content and method, and the use of rubrics) that influenced the communication of the feedback message (column Phase C). The variables “culture and context”, “feedback medium”, and “feedback content” influence how the feedback is received, perceived and interpreted (column Phase D). Twenty variables were found influencing the feedback effect (column Effect). It is striking that only two variables are related to observation, interpretation and rating.

Examples from each phase show variables that are mentioned in meta-analyses or literature reviews how they influence the feedback process (see table 3.2).

The effects of twelve variables are unequivocal, and showed a clear direction. Six variables influence outcome variables related to phase B observation, interpretation and rating, and six variables influence the feedback effect. Table 3.3 summarizes our main findings.

Variables influencing the outcome of phase B: observation, interpretation and rating. a) Rating high complexity task results in low interrater agreement.

Task complexity influences how observers rate observed performances in the workplace.46, 50 Feedback providers have less agreement on ratings on highly complex tasks compared to low complex tasks. Possible explanations are the number of subtasks included in complex tasks, a lack of clear standards (for each subtask), an FP not being skilled enough to perform the task him- or herself, and the difficulty to observe complex task behaviour.

Table 3.2. Literature sources that report the existence of effects of various variables within phases A through D on one or more subsequent phases in the feedback process Phase A:

Phase B:

Phase C:

Phase D: Effect:

Task variation Task complexity

9 4 and 17

Task nature Task subject matter Task perception Culture and context Feedback recipient’s cultural background Feedback recipient’s age Feedback recipient’s skills Phase B: observation and interpretation Training content and method

6 ,12, 15, 19, 25 and 31† 2,11, 12 and 16

4 and 22 9 26 5, 13 and 26

26

16 6† and31

9

Assessment method - utility Rubrics Purpose of observation

40, 45 and 46 40, 45 and 46 3, 7†, 8, 18, 27 and 45 28 and 36 33 33 17, 18, and 32

Focus of observation Intensity of observation

3, 5 and 13 17, 32 and 45

Type of standard Time to build relation, trust and safety

3, 8, and 18 18, 38 and 43

Feedback provider’s position

4, 7, 22, 32, 38 and 45 4, 7 22 and 38

Nature of the instrument

Feedback provider’s task familiarity Feedback provider’s cultural background Phase C: the communication of the feedback message Feedback source Feedback medium Feedback form

11 and 35

6 and 27

26

Feedback content Feedback complexity Feedback timing Feedback frequency Feedback intensity Phase D: reception and interpretation of the feedback message Time between feedback-performance2 Activities between feedback-performance2 Feedback recipient’s self-esteem Feedback recipient’s goal setting behavior

15 and 44 12 2, 6, 16, 31, 37, 42 and 44 20 and 39 2, 10, 12, 19, 23, 30 and 39 39† 2, 14, 21, 30 and 39 12 and 41 15, 16, 31 and 44 29

) Results on these variables seem to be influential but also contra-indications are found The numbers in the cells refer to the numbers of the reviews or meta-analyses in Table 1 (Appenidx A) †

6 15 19 and 26

16 and 21† 21 12 12

71 Variables that affect the process and outcome of feedback, relevant for medical training: a meta-review

Independent variables connected with: Phase A: tasks, standards and first task performance

Chapter 3

72

b) High task familiarity of the FP leads to more agreement in ratings.

FP’s who are familiar with the observed task show high interrater agreement. The interrater agreement of ratings of management tasks, diagnosing problems, job knowledge, and judgment ability is higher among supervisors than among peers or subordinates.

Supervisors have an overview of the entire task, due to their competence or their experience with performing similar tasks themselves. However, peers show higher interrater agreement on ratings of observed interpersonal behavior. Working more often with colleagues gives them a better insight in these aspects than subordinates or supervisors.50-51, 65 c) Training FPs in using observation instruments reduces ratings errors.

Rating errors influence the quality of the observation and in a later phase the content of the feedback message. Examples of such errors are: halo effect, leniency effect, rating accuracy, and observational accuracy.58-60 Training on how to work with observation instruments reduces rating errors and increases accuracy.60 Training in behavior observation had a high positive impact on the quality of the task observation (d=0.59).58 Trainings in which discussion about the meanings of rating categories, practice and feedback are central components are very effective: they reduce rating errors with between 50% and 85%.59 d) The use of rubrics increases the reliability of scoring and facilitates feedback communication.

A rubric is ‘a scoring tool for qualitative ratings of authentic or complex student work.

It includes criteria for rating important dimensions of performance as well as standards of attainment for those criteria.’.66 In rubrics criteria and standards are made explicit, so the meaning and focus of dimensions of the task is clear, which results in high reliability of scoring.

Because of its concrete nature, FPs perceive rubrics also as facilitating the feedback communication. Jonsson and colleagues did not use empirical data to illustrate their findings.66 e) Similar cultural background of FRs and FPs result in higher performance ratings.

Learners’ and supervisors’ perceptions about acting as a professional are determined by (country) cultural background and by the organizational culture. A FR working in his own country and being supervised by a FP with the same cultural background has higher task performance ratings.55 Because of the similarity of cultural background they have comparable perceptions about professionalism and they value certain job dimensions similar.55 f) More time available for the FP to build a relationship with the FR leads to higher correlations of ratings with objective performance measures.

Objective outcome measures -such as: time spend on a task, a person’s production pace- are less prone to observation bias than ratings of human feedback providers.

Norton found that in a peer feedback situation, FP’s time investment in the relationship with FR influenced the correlation between FP’s performance rating and objective measures of performance (r=0.69). When the FP’s time investment in the relationship was high, the correlations between the FP’s ratings and objective ratings of performance were high.61 The results suggest that getting to know the FR reduces rating errors of bias from the FP. And especially for students with lower self-esteem or high test anxiety, it can be a factor of trust to know the supervisor.68

Feedback effect

Observation, interpretation and rating

Table 3.3. Overview of variables that have a clear direction and a unequivocal effect on the observation, interpretation and rating, and the feedback effect Influencing variable FP’s rating high complexity tasks

Effect → Decrease of

Outcome measure interrater agreement

FP’s having high task familiarity

→ Increase of

agreement in ratings

FP’s trained in using observation instruments

→ Decrease of

rating errors

FP’s using rubrics

→ Increase of

reliability of scoring

FP’s and FR’s having similar cultural background

→ Higher

performance ratings

FP’s having time to build relationship with FR

→ Higher

correlations between subjective and objective performance measures.

FR’s having low initial task performance

→ High

feedback effect

Feedback message is threat to FR’s self-esteem

→ Low

feedback effect

FR’s having goal setting behavior

→ Increase of

feedback effect

Feedback is part of a multi-facetted intervention

→ Increase of

feedback effect

Feedback content: encouraging, specific, and elaborate Feedback message frequent ly given

→ Increase of

feedback effect

→ Increase of

feedback effect

73 Variables that affect the process and outcome of feedback, relevant for medical training: a meta-review

Variables influencing the feedback effect a) Low initial skill of the FR results in high feedback effects.

From research in the clinical setting it becomes clear that feedback on clinical practice was more effective with FR’s who did not meet the required baseline of recommended practice, than with FR’s who met this requirement.36 A FR with low skills has more opportunities for improvement, and effects might be visible sooner. b) The FR’s self-esteem influences the feedback effect.

Self-esteem was not often mentioned in the reviews and meta-analyses. An explanation is that most variables studied relate to impersonal aspects instead of personal aspects.

When feedback is perceived as threat to a person’s self-esteem the effect of the feedback is lower.8 c) Having FRs show goal setting behavior increases the feedback effect.

When a FR has a goal in mind and knows what he or she wants to accomplish, the received feedback might be used to reach that goal. Feedback is more effective when a FR sets a goal, than when not.8 d) Feedback as part of a multi-facetted intervention (see table 3.1: form of FB) increases the feedback effect.

The instructional content in which feedback is embedded, especially in a test situation, increases the feedback effect.52 The combination of a feedback intervention with other educational interventions (so-called multi-facetted interventions), such as audits, supervision, educational outreach visits or education, increases the feedback effect.16, 36, 40-41, 52 e) Encouraging, specific, elaborate feedback content increases the feedback effect.

Content of the feedback message that is tailored to the desired feedback effect increases its effectiveness.49 Feedback consisting of specific, relevant and encouraging information, such as: correct answers (the standard), task information, additional explana-

Chapter 3

74

tion, information about performance in relation to previous trials, is more effective than shallow feedback, such as general information, ‘right’ ‘wrong’ information, or compliments.4, 8, 52, 70, 72-73 f) Frequent feedback increases the feedback effect.

With simply trainable short visual and psychomotor tasks, frequently provided feedback increases the feedback effect more than infrequent feedback.8 Soumerai et al. found a positive change in drug prescribing practice when ongoing feedback reports about medical doctor’s drug prescribing behavior were provided.42 Even being used to receiving frequent feedback does not appear to decrease the feedback effect.49 Twenty one of the 33 variables show equivocal results: they influence the process and effect of feedback in one study, but not in other studies, or influences of other variables made interpretations about the direction impossible, or extra information is needed to determine the direction (see Supplementary Materials section III). For example, research on interrater reliability between FPs with different positions reveals that it is higher among supervisors than between supervisors and subordinates.50-51 Based on these results we assume that the position of raters influence the ratings, and that ratings of supervisors do correspond more with ratings from other supervisors than with ratings from subordinates. However, this does not give us information who is a more reliable rater.

Several studies show that the length of the feedback (see table 3.1: intensity of FB) influences the feedback effect: messages of different length have different effects.16, 47 But the results do not make clear whether long or short messages are better. Guidelines for the optimum feedback time cannot be derived from this, and are probably task dependent.

3.4 Discussion We met the objectives of our study: cross-disciplinary feedback research is brought together, and this results in an overview of 33 variables influencing, moderating, or interacting with either the feedback process, the feedback effect, or both.

Following our summary table 3.3, we conclude that the reliability of feedback information and feedback scores is high when it pertains to low-complexity or familiar tasks and when observers are trained and use rubrics. Feedback is more favorable when the feedback provider and recipient share a cultural background and feedback messages correlate with objective performance findings when provider and recipient know each other well.

Effects of feedback are strong when the feedback recipient has a low initial task performance, when the message does not contain any threat to the recipient’s self-esteem, when the recipient shows goal-setting behavior, and when the message is encouraging, specific, elaborate and frequently given.

Regarding the framework, our study shows that a) variables from each phase of the feedback process influence the next phase in the feedback process, and b) all four phases in the feedback process influence the feedback effect. The consequence for clinical practice is that we should not only focus on how well the FP communicates the feedback (phase C), but also about the performed task (phase A), and the observation and rating (phase B), and FR’s reception and the perception (phase D).

We can illustrate this with an example. When a senior clerk receives feedback, its effect

Limitations The present study has limitations that need to be taken into account when considering the study and its contributions. This meta-review does not pretend to give an exhaustive overview of all the literature about feedback. Several studies and variables are missing for a variety of reasons. a) ‘New’ or ‘rare’ topics, like feedback propensities,78 feedback framing,79 characteristics of FPs80 require more publications for being included in qualitative reviews or meta-analyses. Sometimes these topics will turn up in narrative reviews.60 b) Reviews or meta-analyses did not always turn-up because of the search criteria we used, for example the reviews of Sadler (1989), and Black & Wiliam (1998), and or due to the methodological criteria,81-82 for example studies on the Barnum effect,83 feedback seeking,84 and Hattie & Timperley’s review which did not mention search criteria.5 c) Studies did not fall into the time span 1986-February 2012, like a review from 1984 on the effect of providing feedback on the FP.85 d) Variables might be researched in single qualitative studies or experiments, but meta-analyses or reviews on the topic are lacking.

A consequence of focusing on meta-analyses and literature reviews is that the reported variables are not new; they have been researched in primary studies. Meta-analyses and reviews often suffer from threats to validity: the mixing of dissimilar studies, publication bias and inclusion of poor quality studies.86-88 This leads to misrepresentations of

75 Variables that affect the process and outcome of feedback, relevant for medical training: a meta-review

might be low if the clerk had difficulty taking a history with an elderly couple who disagree (high task complexity). The resident supervising the clerk just started training (lacking expertise) (A → B). Consequently, the resident was not able to describe observed behavior (unspecific message) (B → C), which led the clerk to conclude that the job was performed poorly (influencing self-esteem) (C → D). This example shows these variables affect subsequent phases in the feedback process. The influence of these processes on the effect of feedback of learning and behavior cannot be predicted. They may increase the effect because the clerk understands that specific information is needed to improve the history taking (A). The feedback effect may decrease due to lack of concentration and expertise by the supervising resident (B), because the message is not specific (C), or because the message is interpreted as a threat to self-esteem (D).

We can learn from this that the feedback process is crucial to the feedback effect. Focusing only on one phase has large consequences for ecological validity of feedback studies. Thus if a model is proposed to predict the feedback effect the influences of the four phases should be represented: Variables influencing Phase A + Variables influencing Phase B + Variables influencing phase C + Variables influencing phase D determine the feedback effect. We assume that the influences in the difference phases will interact.

We also assume that the impact of some phases will be stronger than other phases.

However, we need more evidence to predict feedback effects work in combination.

Conclusions about variables with a clear direction are that FP’s can increase the quality of an assessment from task familiarity, training in use of observation instruments, using rubrics, and building a relationship with the FR. The variables from Phase B are important because their outcomes inform the feedback in phase C. In phase C FPs can also increase the effectiveness of feedback if it is given frequently, when feedback, has encouraging, specific and elaborate content, and when feedback is not a threat to FR’s self-esteem. Feedback is also more effective when it is part of a multifaceted intervention Feedback effectiveness is improved when FRs are able to set goals.

Chapter 3

76

outcomes and we cannot exclude the possibility that it also influences this meta-review.

We took the quality of the included studies into account, and only reported variables from narrative reviews that support findings from other included studies, but this might still have influenced our results. Further, not all included variables should get the same weight. Variables in meta-analyses and in systematic quantitative reviews deserve a higher weight than in narrative reviews, because they are based on more included studies.

The reverse side of bringing cross-disciplinary research together is the variety of study contexts, tasks and participants. Results found in one context may not simply be generalized to the other contexts: both task aspects and context influences findings.8, 47, 52, 55 Research agenda As can be seen from table 3.2, we were hardly able to identify any relationships between variables related to observation, interpretation and rating (phase B) and the feedback effect (column Effect). Students often address the importance of observation before receiving feedback. Questions for further research could be: “What is the relationship between being observed and the feedback effect?” or “How does the reliability and validity of observation instruments influence the feedback process and the feedback effect?”

Most variables influence Phase A, Phase B or the feedback effect. The large number of empty cells in the columns phase C and D indicates a lack of systematic research of variables influencing feedback communication (phase C) and the reception, perception and interpretation of feedback (phase D). From the 46 included studies respectively three38, 60, 68 and six4, 38, 55, 60, 69, 71 studies presented outcome variables in phase C and D (table S1 column E). None of these studies were explicitly pertained to feedback communication (phase C). From the six studies in phase D only one study focused explicitly on the reception, perception and interpretation of the feedback. This illustrates the complexity of researching these topics. For further research in phase C and D we have three recommendations. First, to determine whether the variables mentioned in table 3.2 influence these phases, in which direction and how much. Second, to study more aspects of verbal behavior: framing of feedback messages, the ‘quality’ of explanations and examples in the feedback, the order of positive and negative aspects in the feedback message. Last, to study aspects of non-verbal behavior such as tone of voice, facial expressions, and search for features in FP’s and FR’s non-verbal behavior that might contribute to better communication and reception of the feedback.

Five variables show in some phases mixed results (table 3.2): they influence the feedback process and the effect but also contra-indications are found. This indicates that more systematic research in this topic is needed.

Although FP and FR are crucial actors in the feedback process, the numbers of independent variables do not reflect this. Good quality systematic reviews on personal characteristics of FRs and FPs are lacking.8 Personal characteristics are often not incorporated because of their complexity.20, 85 Further research on this topic is also suggested.

Practical implications To make feedback effective, systematic research on the relationships between variables represented by empty cells can contribute to our further knowledge of effective feedback; evidence-based guidelines on influential variables in different phases of the feedback process are needed to bring this knowledge into practice. Clearly, only focusing on

77 Variables that affect the process and outcome of feedback, relevant for medical training: a meta-review

the phase of feedback communication is no guarantee for effective feedback. A focus on all phases of the feedback process is likely to yield a more predictable effect.

Techniques that appear influential in other disciplines are working with rubrics,66 and systematic approaches such as the Productivity Measurement and Enhancement System (ProMES), where regular outcome-based feedback is provided on different indicators.49 Training on assessment and feedback can focus on variables that clearly affect the quality of observation, interpretation, rating, and the effectiveness of feedback. In addition to rater training, education about developing multi-faceted assessments in clinical environments where feedback is embedded should receive attention. Courses on receiving feedback and goal setting can also increase the utility of feedback in clinical settings for clerks and residents.

References

Chapter 3

78

* References marked with an asterisk indicate studies included in the meta-review.

1. Van de Ridder JMM, Stokking KM, McGaghie WC, Ten Cate OThJ. What is Feedback in Clinical Education? Med Educ 2008;42:189-197.

2. Ilgen DR, Barnes Farrell JL, McKellin DB. Performance appraisal process research in the 1980s: What has it contributed to appraisals in use? Organ Behav Hum Decis Process 1993;54(3):321-368.

3. Smither JW, London M, Reilly RR. Does performance improve following multisource feedback? A theoretical model, meta-analysis, and review of empirical findings. Pers Psychol 2005;58:33-66.

4. *Shute VJ. Focus on Formative feedback. Rev Educ Res 2008;78(1):153-189.

5. Hattie J, Timperley H. The power of feedback. Rev Educ Res 2007;77(1):81-112.

6. Cusella LP. Feedback, motivation and performance. In: Porter F, Jablin F, Putnam L, Roberts K, editors. Handbook of Organizational communication: an interdisciplinary perspective.

Thousand Oaks. Sage publications; 1987.

7. Vollmeyer R, Rheinberg F. Motivational effects on self-regulated learning with different tasks. Educ Psychol Rev 2006;18:239-253.

8. *Kluger AN, DeNisi A. Effects of feedback intervention on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychol Bull 1996;119(2):254284.

9. Dornan T. Experienced based learning. Learning clinical medicine in workplaces [PhD thesis]. Manchester: Tim Dornan; 2006.

10. Irby DM. Teaching and learning in ambulatory care settings: A thematic review of the literature. Acad Med 1995;70(10):898-931.

11. Bowen JL, Irby DM. Assessing quality and costs of education in the ambulatory setting: a review of the literature. Acad Med 2002;77(7):621-680.

12. Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No.31.

Med Teach 2007;29(9):855-871.

13. McGee SR, Irby DM. Teaching in the outpatient clinic. Practical tips. J Gen Intern Med 1997;12 Suppl 2:S34-S40.

14. *Hamilton KEStc, Coates V, Kelly B, Boore JRP, Cundell JH, Mcfetridge B, McGonigle M, Sinclair M. Performance assessment in health care providers: a critical review of evidence and current practice. J Nurs Manag 2007;15:773-91.

15. Irby DM, Wilkerson L. Teaching rounds. Teaching when time is limited. BMJ 2008;336:384387.

16. *Veloski J, Boex JR, Grasberger MJ, Evans A, Wolfson DB. Systematic review of the literature on assessment, feedback and physicians’ clinical performance: BEME Guide No. 7. Med Teach 2006;28(2):117-128.

17. Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, O’Brien MA, Johansen M, Grimshaw J, Oxman AD. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2012;6:CD000259. doi: 10.1002/14651858.

CD000259.pub3.

18. Clement DA, Frandsen KD. Om conceptual and empirical treatments of feedback in human communication. Commun Monogr 1976;43:11-28.

19. Frandsen KD, Millis MA. On Conceptual, Theoretical and Empirical Treatments of Feedback in Human Communication: Fifteen Years Later. Comm Rep 1993;6(2):79-91.

20. Ilgen DR, Fisher CD, Taylor SM. Consequences of individual feedback on behavior in organizations. J Appl Psychol 1979;64(4):349-371.

21. Richardson GP. Feedback thought in social science and systems theory. Waltham: Pegasus communications inc.; 1999.

79 Variables that affect the process and outcome of feedback, relevant for medical training: a meta-review

22. Watzlawick P, Helmick Beavin J, Jackson DD. Pragmatics of Human communication. A study of interactional patterns, pathologies and paradoxes. New York: WW Norton & Company inc.; 1967.

23. Johnson W. People in quandaries. The semantics of personal adjustment. New York: Harper & Row Publishers; 1946.

24. Johnson W. Speech and personality. In: Bryson L, editor. The communication of ideas. A serie of addresses. New York: Cooper Square Publishers; 1964.

25. Lasswell HD. The structure and function of communication in society. In: Bryson L, editor.

The communication of ideas. A serie of addresses. New York: Cooper Square Publishers; 1964.

26. Lasswell HD. Describing the contents of communications. In: Smith BL, Lasswell HD, Casey RD, editors. Propaganda, communication, and public opinion. A comprehensive reference guide. Princeton: Princeton University Press; 1946.

27. Lasswell HD. Describing the effects of communications. In: Smith BL, Lasswell HD, Casey RD, editors. Propaganda, communication, and public opinion. A comprehensive reference guide. Princeton: Princeton University Press; 1946.

28. Mory EH. Feedback research revisited. In: Jonassen DH, editor. Handbook of Research on Educational Communications and Technology. Mahwah: Lawrence Erlbaum Associates; 2004.

29. Gallagher JJ. Teaching and learning: new models. Ann Rev Psychol 1994;45:171-195.

30. Fenner DE. Avoiding pitfalls: lessons in surgical teaching. Obstet Gynecol Clin N Am 2006;33:333-342.

31. Spark JI, Rowe S. Clinical governance: its effect on surgery and the surgeon. ANZ J Surg 2004;74:167-170.

32. Moher, D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. Ann Intern Med 2009;151:264-269.

33. Harden A, Thomas J. Methodological issues in combining diverse study types in systematic reviews. Int J Soc Res Meth 2005;8(3):257-271.

34. Albanese M, Norcini J. Systematic reviews: what are they and why should we care? Adv Health Sci Educ Theory Pract 2002;7:147-151.

35. Moja LP, Telaro E, D’Amico R, Moschetti I, Coe L, Liberati A. Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study. BMJ 2005;330(7499):1053. doi:10.1136/bmj.38414.515938.8F 36. *Jamtvedt G, Young JM, Kristoffersen DT, O’Brien MA, Oxman AD. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2006:19;(2)(2):CD000259.

37. *Craig, S. Direct observation of clinical practice in emergency medicine education. Acad Emerg Med 2011;18(1):60-67.

38. *Rout A, Roberts P. Peer review in nursing and midwifery: a literature review. J Clinical Nurs 2008;17:427-442.

39. *Miller A, Archer J. Impact of workplace based assessment on doctor’s education and performance: a systematic review. BMJ 2010; Sep 24;341:c5064. doi: 10.1136/bmj.c5064.

40. *Steinman MA, Ranji SR, Shojania KG, Gonzales R. Improving antibiotic selection: a systematic review and quantitative analysis of quality improvement strategies. Med Care 2006;44(7):617-628.

41. *Rowe AK, de Savigny D, Lanata CF, Victora CG. How can we achieve and maintain highquality performance of health workers in low-resource settings? Lancet 2005; 366(9490):10261035.

42. *Soumerai SB, McLaughlin TJ, Avorn J. Improving drug prescribing in primary care: a critical analysis of the experimental literature. Milbank Q 1989;67(2):268-317.

43. *Cheraghi-Sohi S, Bower P. Can the feedback of patient assessments, brief training, or their

44.

80 Chapter 3

45.

46.

47.

48.

49.

50.

51.

52.

53.

54.

55.

56.

57.

58.

59.

60.

61.

62.

63.

64.

65.

66.

combination, improve the interpersonal skills of primary care physicians? A systematic review. BMC Health Serv Res 2008;8:179. *Huang C. Magnitude of task-sampling variability in performance assessment: a meta-analysis. Educ Psychol Meas 2009;69(6):887-912. *Fukkink NG, Trienekens N, Kramer LJC. Video feedback in education and training: Putting learning in the picture. Educ Psychol Rev 2011;23:45–63. *Murphy KR, Herr BM, Lockhart MC, Maguire E. Evaluating the performance of paper people. J Appl Psychol 1986;71(4):654-61. *Li S. The effectiveness of corrective feedback in SLA: A meta-analysis. Lang Learn 2010:60(2):309-65. *Clariana RB, Koul R. Multiple-Try Feedback and Higher-Order Learning Outcomes. Int J Instr Media 2005;32(3):239-45. *Pritchard RD, Harrell MM, DiazGranados D, Guzman MJ. The productivity measurement and enhancement system: a meta-analysis. J Appl Psychol 2008;93(3):540-567. *Conway JM, Huffcutt AI. Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. Hum Perform 1997;10(4):331360. *Viswesvaran C, Ones DS, Schmidt FL. Comparative analysis of the reliability of job performance ratings. J Appl Psychol 1996;81(5):557-574. *Bangert Drowns RL, Kulik CLC, Kulik JA, Morgan M. The instructional effect of feedback in test-like events. Rev Educ Res 1991;61(2):213-238. *Kingston N, Nash B. Formative assessment: a meta-analysis and a call for research Educ Meas 2011;30(4):28-37. *Lyster R, Saito K. Oral feedback in Classroom SLA. Stud Sec Lang Acquis 2010;32:265-302. *Claus L, Briscoe D. Employee performance management across borders: a review of relevant academic literature. Int J Manag Rev 2009;11(2):175-196. *Kraiger K, Ford JK. The relation of job knowledge, job performance, and supervisory ratings as a function of ratee race. Hum Perform 1990;3(4):269-279. *Ford JK, Kraiger K, Schechtman SL. Study of race effects in objective indices and subjective evaluations of performance: A meta-analysis of performance criteria. Psychol Bull 1986;99(3):330-337. *Woehr DJ, Huffcutt AI. Rater training for performance appraisal: A quantitative review. J Occup Organ Psychol 1994;67(3):189-205. *Smith DE. Training programs for performance appraisal: A review. Acad Manage Rev 1986;11(1):22-40. *Wanguri DM. A Review, an Integration, and a Critique of Cross-Disciplinary Research on Performance Appraisals, Evaluations, and Feedback, 1980-1990. J Bus Commun 1995;32(3):267-293. *Norton SM. Peer assessments of performance and ability: An exploratory meta-analysis of statistical artifacts and contextual moderators. J Bus Psychol 1992;6(3):387-399. *Jawahar IM, Williams CR. Where all the children are above average: The performance appraisal purpose effect. Person Psychol 1997;50(4):905-925. *Bommer WH, Johnson J, Rich GA, Podsakoff PM. On the interchangeability of objective and subjective measures of employee performance: A meta-analysis. Person Psychol 1995;48(3):587-605. *Heneman RL. The relationship between supervisory ratings and results-oriented measures of performance: A meta-analysis. Person Psychol 1986;39(4):811-826. *Harris MM, Schaubroeck J. A Meta-Analysis of Self-Supervisor, Self-Peer, and Peer-Supervisor Ratings. Person Psychol 1988;41(1):43-62. *Jonsson A, Svingby G. The use of scoring rubrics: Reliability, validity and educational consequences. Educ Res Rev 2007;2:130-144.

81 Variables that affect the process and outcome of feedback, relevant for medical training: a meta-review

67. *Van Gennip NAE, Segers MSR, Tillema HH. Peer assessment for learning form a social perspective: The influence of interpersonal variables and structural features. Educ Res Rev 2009;4:41-54.

68. *Sasanguie D, Elen J, Clarebout G, Van den Noortgate W, Vandenabeele J, De Fraine B. Disentangling instructional roles: the case of teaching and summative assessment. J High Educ 2011;36(8):897-910.

69. *Hepplestone S, Holden G, Irwin B, Parkin HJ, Thorpe L. Using technology to encourage student engagement with feedback: a literature review. Res Learn Tech 2011;19(2):117-127.

70. *Balzer WK, Doherty ME, O’Connor R. Effects of cognitive feedback on performance. Psychol Bull 1989;106(3):410-33.

71. *Tang SH, Hall VC. The overjustification effect: A meta-analysis. Appl Cogn Psychol 1995;9(5):365-404.

72. *Jaehnig W, Miller ML. Feedback types in programmed instruction: a systematic review. The Psychol Rec 2007;57:219-232.

73. *Karelaia N, Hogarth RM. Determinants of linear judgment: A meta-analysis of lens model studies. Psychol Bull 2008;134(3):404-426.

74. *Travlos AK, Pratt J. Temporal locus of knowledge of results: A meta-analytic review. Percept Mot Skills 1995;80(1):3-14.

75. *Kulik JA, Kulik ClC. Timing of feedback and verbal learning. Rev Educ Res 1988;58(1):79-97.

76. *McAfee RB, Winn AR. The use of incentives/feedback to enhance work place safety: A critique of the literature. J Safety Res 1989;20(1):7-19.

77. *Azevedo R, Bernard RM. A meta-analysis of the effects of feedback in computer based instruction. J Educ Comput Res 1995;13(2):111-127.

78. Herold DM, Fedor DB. Individual differences in feedback propensities and training performance. Hum Resour Manage R 2003;13(4):675-689.

79. Waung M, Jones DR. The effect of feedback packaging on ratee reactions. J Appl Soc Psychol 2005;35:1630-52.

80. Menarchy EP, Knight AM, Kolodner K, Wright SM. Physician characteristics associated with proficiency in feedback skills. J Gen Intern Med 2006;21:440-446.

81. Sadler DR. Formative assessment and the design of instructional systems. Instr Sci 1989;18:119-144.

82. Black P, Wiliam D. Assessment and classroom learning. Assess Educ Princ Pol Pract 1998;5:775.

83. Furnham A, Schofield S. Accepting personality test feedback: A review of the Barnum effect.

Curr Psychol Res and Rev 1987;6(2):162-178.

84. Madzar S. Feedback seeking behavior: A review of the literature and implications for HRD practitioners. Hum Resource Dev Q 1995;6(4):337-349.

85. Larson JR. The performance feedback process: A preliminary model. Organ Behav Hum Dec 1984;33:42-76.

86. Delgado-Rodriguez M. Systematic reviews of meta-analyses: applications and limitations. J Epidemiol Cummun H 2006;60:90-92.

87. Sharpe D. Of apples and oranges, file drawers and garbage: why validity issues in metaanalysis will not go away. Clin Psychol Rev 1997;17(8):881-901.

88. Chambers, EA. An introduction to meta-analysis with articles from the Journal of Educational Research (1992-2002). J Educ Res 2004;98(1):35-44.

89. Oxford Reference. English Dictionaries, Oxford University Press 2013 [approached 2011 January 7]. Available from: http://www.oxfordreference.com.proxy.library.uu.nl/browse?type_ 0=englishdictionaries 90. Burke, D. Strategies for using feedback students bring to higher education. Assess Eval High Educ 2009; 34(1):41-50.

91. Whitelock, D. Editorial: e-assessment: Developing new dialogues for the digital age. Br J Educ Tech 2009;40(2):199–202.

Chapter 3

82

Measuring trainee Perception of the value of Feedback in Clinical Settings (P-FiCS)

Unpublished study: Van de Ridder JMM, McGaghie WC, Stokking KM, Ten Cate O.

Chapter 4

84

Abstract Introduction Knowledge of feedback perceptions is relevant to understand the extent to which feedback contributes to meaningful learning. This study describes the development of an instrument to measure trainee perceptions about feedback in clinical settings.

Method An eight step approach was used to develop an instrument to measure perceptions towards feedback in clinical settings. To establish generalizability, with a 90-item questionnaire, data from Dutch (n=382) and USA( n=292) medical students and residents were used.

Results Using factor analysis (PCA, varimax rotation) resulted in nine scales containing 46 items.

Three scales relate to the feedback provider’s teaching behavior: Purposeful and Trustworthy, Involved, and Detached. Two scales relate to the feedback recipient’s self-evaluations of task performance: Low and High. One scale consists of items regarding Message clarity. Three scales are associated with feedback context: Privacy, Formative Nature of Feedback, and Critical Nature of the Task. The internal consistency of the scales varied from α = 0.61 to α = 0.76. Group differences in scale scores are found for gender, seniority, nationality, and medical specialty.

Conclusion This developmental study successfully produced an instrument to measure feedback perception in clinical settings. Further validation of scale data is recommended to increase its theoretical and practical utility.

4.1 Introduction

Figure 4.1. The feedback process and the feedback effect

85 Measuring trainee Perception of the value of Feedback in Clinical Settings (P-FiCS)

roviding feedback is effective and often improves performance1-7 including safety related performances,5 work productivity,8 judgment abilities,2 learner’s goal setting abilities,9 and physicians’ performance.6-7 The impact of feedback is often small to moderate,3-4, 7, 10 and the statistical effect sizes vary from small to large.4, 11 Medical trainees do not always perceive feedback as useful and their satisfaction with the feedback process is often low.12 The literature about feedback perceptions is scattered and diffuse. It shows that medical faculty, residents, and students have different perceptions about feedback providers, feedback message quality, and the usefulness of feedback.13-16 In the past years, causes for suboptimal feedback effects in the clinical workplace have been also been attributed to student behavior.17 Feedback seeking behavior by medical trainees appears to have its own dynamics.18-19 Students may avoid seeking feedback because it can negatively affect their feeling of self-efficacy or intrinsic motivation.20-21 Feedback processes have predominantly been viewed from the teacher’s perspective.22-23 Taking the students’ views in feedback processes has been recommended.24 Our study focuses on the perception of feedback among medical trainees in different (international) cultures. We designed an instrument to measure feedback perception grounded in theory and empirical research.

Residents’ perception of a feedback provider’s credibility is influenced by the trust, respect, level of knowledge, experience, and status of the provider.25-27 Student’s perception of the educational value of feedback seems less influenced by the status of the feedback provider e.g.: resident or specialist.28 The quality of the feedback message medical students receive from residents is perceived as equivalent to feedback from faculty.29 Gil and colleagues found that feedback

Chapter 4

86

sufficiency, specificity, timeliness, regularity, relevance, encouraging tone, giving recommendations for improvement, and reciprocity are perceived as important by both students and faculty.13 There is general agreement that initiative-taking by students, e.g., asking for feedback, improves its perceived value.28 Sargeant et al., showed that use of multisource feedback outcomes by faculty is influenced by their perceptions of the feedback provider’s credibility and the accuracy of the feedback message.16 Previous research on feedback perceptions explored several features of the feedback process as shown in figure 4.1. This leaves the following question: what are feedback recipients’ perceptions of its educational value under varying conditions of the feedback provider, its form and content, the context, and recipient in the medical clinical setting?

Eva et al. and Watling and Lingard stress the importance of exploring the recipients’ perceptions about the feedback process.22, 30 Knowledge of feedback perceptions is relevant to understand the extent to which feedback contributes to meaningful learning.

We believe that study of perceptions can reveal why feedback effects are sometimes small and also can inform investigators about factors that produce variability in effects.

Research may also show whether feedback is perceived similarly in different cultures such as countries, hospitals, or medical departments. In addition we may identify variables useful for experimental studies about the influence of feedback perceptions on its effects.

The goal of this study was to develop an instrument, grounded in theory and research to measure feedback recipients’ perceptions about the educational value of feedback, in clinical settings. Feedback in this study is defined as “specific information about the comparison between trainee’s performance and a standard that is given with the intent to improve trainee performance.”31 This contribution describes the development of a measure of Perceptions about the value of Feedback in Clinical Settings (P-FiCS) that yields reliable data, the evaluation of the psychometric properties of measurement data, and analyses of demographic group differences.

4.2 Method We used an eight step scale development process, following systematic guidelines presented by Comrey,32 Gorsuch,33 and Wetzel.34 We used other studies of scale development as examples.34-36 The results are reported according to eight guidelines given by Henson and Roberts.37 Determining what is to be measured The measurement development is based on a descriptive modeldisplayed in figure 4. 1 and mapped in table 4.1.38-40 In figure 4.1 the Phases A1, B, C and D represent the feedback process: Phase A1 includes the task, standard (T&S) and task performance (P1). Phase B relates to feedback provider’s observation and interpretation of the task performance (O&I). In Phase C the feedback provider communicates the feedback to the recipient (C). In phase D the feedback recipient receives and interprets (R&I) the feedback from the provider. When the same task is performed again, the cycle is closed (Phase A2). The feedback effect can be found when two performances are compared (Δ P1-P2).

Table 4.1. Blue print: a thirty-cell descriptive model with in cells 1, 3, 4, 7, 8, 10, 16, 22, 25, and 28 the number of questions in the Initial Dutch-English (I) and the Final (F) questionnaire Chronological Phase and Actor Questionnaire

Phase A1

Phase B Phase C Phase D Phase A2

T&S P1 O&I No actors Recipient Provider I

F

Component I

Cell 1

Feedback provider

7

I

F

Cell 2

5

Component II

Cell 7

Feedback Recipient

6

Component III

Cell 13

10

F

Cell 3 11

Cell 8

1

I

7

Cell 9

7

Cell 14

C RI P2 Provider Recipient Recipient I

Cell 4 16

Message: content Cell 19

Cell 20

Cell 21

Message: form Component IV

Cell 25

Feedback context

4

3

Total

17

9

Cell 26 10

7

Cell 27 11

7

Cell 6

Cell 11

I

F

34

19

18

10

19

8

Cell 12

Cell 17

Cell 18

Cell 23

Cell 24

6

Cell 22 6

Cell 5

2

Cell 16 13

Total items

7

Cell 10 2

Cell 15

F

2

Cell 28

Cell 29

Cell 30

15

6

19

9

52

23

90

46

Note. The five phases (A1, B, C, D en A2) correspond with the five phases of the feedback process in Figure 1.

87 Measuring trainee Perception of the value of Feedback in Clinical Settings (P-FiCS)

The horizontal axis in table 4.1 represents a chronological description of the six phase feedback process beginning with a task and standard (T & S) and continuing with task performance (P1), observation and interpretation of task performance (O & I), feedback communication (C), feedback reception and interpretation (R & I), through next task performance (P2). The vertical axis has four, or, dividing message (Component III) into content and form, five components. These are feedback provider, recipient, content, form, and context. The matrix formed by the two axes yields a 30-cell blueprint. Each cell represents a union of a feedback phase and a feedback component. The focus of our measurement covers 10 of the 30 cells from the blueprint: (a) feedback provider and his or her relationship with the task and standard [cell 1], (b) feedback provider’s observation of performance [cell 3], (c) feedback provider’s communication of the feedback [cell 4], (d) relationship of the recipient towards the task and standard [cell 7], (e) feedback recipient’s perception of own task performance [cell 8], (f) recipient’s perception of the feedback communication [cell 10], (g) the content [cell 16] and (h) form [cell 22] of the feedback message, (i) the context in which the task had to be performed [cell 25] and (j) the context in which the feedback was provided [cell 28].

We expect these ten cells to reflect the most important categories of feedback recipients’ perceptions of the feedback process. Blank cells represent less relevant content.

Phases D and A2 are omitted because we wanted to measure perceptions of the potential value of feedback, and not perceptions about recipient’s receipt, interpretation and application of feedback. Component III is about the feedback message but in the phases A1 and B (cells 13-15 and 19-21) feedback has not been given yet, so these cells are also excluded.

Chapter 4

88

Generating an item pool We initially wrote 2 to 16 items for each of the ten relevant cells. The items were derived from a literature study and a survey among medical students (unpublished data). Table 4.1 shows the blueprint and the number of items written for each cell. This resulted in 34 items addressing the feedback provider, 18 items about the feedback recipient, 19 items on feedback content and form, and 19 items covering feedback context.

Determining the format for measurement We chose a seven-point Likert-scale format: 1 = totally disagree, 2 = mainly disagree, 3 = somewhat disagree, 4 = neutral, 5 = somewhat agree, 6 = mainly agree, and 7 = totally agree. A seven point scale permits generation of a continuous response distribution and facilitates statistical treatment.32 All items had the following stem: “I find feedback most instructive when‌â€? We used the word instructive because we assumed that feedback perceived as potentially instructive would be more acceptable and more readily applied than feedback perceived as not instructive.41 Expert reviewing of the initial question pool Both the format of measurement and the item pool were subjected to critical review by three panels: (a) social science researchers and methodologists (n=7), (b) professionals including physicians and medical education researchers (n=10), and (c) junior clerks who reviewed several versions of the questionnaire on language clarity (n=15). Based on the comments from these panels the items were edited, and the layout and accompanying text were adjusted.

The Dutch questionnaire was translated into English and from English back into Dutch to check translation and to guarantee cross-cultural accuracy. The English version was discussed with a native American English speaker (WCM), and American clerks and residents. Based on their comments the items were semantically adjusted to the American clinical situation.

Considering inclusion of validation items In an early version of the questionnaire we had noticed that questions with opposite formulations were not answered in opposite directions as we had expected. Thus we included negatively stated items in the initial questionnaire. This keeps the participants sharp and reduces response sets. Ten randomly chosen items were worded negatively to discourage negligent responses.

Administering the items to a development sample The questionnaire was administered in 2006 and 2007 to a Dutch (n = 382) and a USA (n = 292) sample (total N = 674). Data on the 90 items was gathered in the Netherlands from (a) third year medical students (junior clerks) (n = 92), (b) sixth year medical students (senior clerks) (n = 155), and (c) residents in a variety of medical specialties (n = 135). In the USA the questionnaire was distributed among third year medical students (junior clerks) (n = 50), fourth year medical students (senior clerks) (n = 108), and residents (n = 134). Ethical approval was received from the Northwestern University Institutional Review Board (IRB). At the time of the study in The Netherlands ethical approval of questionnaire research among medical students or residents was not necessary. Participants received a letter in which they were invited to participate, and they received information about the aim of the study, the use of the data, confidentiality, and presentation of the results.

Optimizing scale length Cronbach’s alpha was used to evaluate the internal consistency of the resulting scales.

Based on the previous steps the optimal solution for Component I Feedback Provider consists of three scales (19 items), for Component II Feedback Recipient of two scales (8 items), for Component III Feedback Message of one scale (8 items) and for Component IV Feedback Context of three scales (11 items).

Personal and professional data In addition to questionnaire and site location data (UMCU, Northwestern) we also collected data on subjects’ gender (F, M) and age. For clerks we collected their year of study (Dutch 3, 6; USA 3, 4), and residents’ specialty (Internal Medicine, Surgery, Pediatrics, Ob/Gyn, Emergency Medicine, Psychiatry, Neurology, Anaesthesiology, Family Medicine, Other).

Data analyses Bivariate correlations were calculated for P-FiCS scale scores and trainee age. Analysis of Variance (ANOVA) was used to assess P-FiCS group differences due to gender, nationality, seniority, and residents’ medical specialty. For the analysis of between group differences in residents’ medical specialty only specialties with at least 20 residents were included. Specialties with n < 20 were clustered into the category “others.” Bonferonni correction for multiple statistical tests was used, resulting in an adjusted alpha of p < 0.006 (0.05/9≈0.006).

89 Measuring trainee Perception of the value of Feedback in Clinical Settings (P-FiCS)

Evaluating the items After removing trainees whose responses included missing data (n = 30), complete data from the 90 items for the Dutch and USA administrations (N = 644) was factor analyzed for each of four Feedback components (I Feedback Provider, II Feedback Recipient, III Feedback Message (combining content and form), and IV Feedback Context) using principal components analysis (PCA) as a method of extraction and varimax rotation to determine the number of relevant factors.

Through an exploratory process, two researchers (MvdR and KS) independently determined the number of factors per component based on scree-plots, Eigenvalue ≥ 1, and factor stability when groups were created based on nationality and seniority. Consensus was reached through discussion. The Kaiser-Meyer-Olkin (KMO) index was used to evaluate the statistical adequacy of the obtained solution and sample size sufficiency.42 All items having factor loadings lower than 0.40 were removed. When an item loaded on two factors and the differences between the two loadings was smaller than 0.10 it was also removed. Factors consisting of only two items, or having low interpretability were removed. Interpretability of factors was checked by two researchers (WCM and ThJtC) who independently labeled the factors. Consensus was reached through discussion. A repetition of the factor analysis on the final set of items was performed to check for factorial stability.

4.3 Results Factor solutions

Chapter 4

90

The exploratory analysis and the application of the criteria Eigenvalue > 1, factor loadings higher than 0.40, simple structure, more than two items, good interpretability, and reasonable internal consistency of scales, resulted in nine scales.

We determined three scales in Component I, two in Component II, one in Component III, and three in component IV. In component I the first scale contains items about feedback provider’s teaching behavior: is it purposeful and trustworthy (P-P&T). In the second scale the items reflects how involved the feedback provider is in observation and teaching (P-I). The third scale consists of items which illustrated detached teaching behavior (P-D). In Component II the items in the first scale indicate high self-evaluations on task performance by feedback recipients (R-HSe). In the second scale the items refer to low self-evaluations on task performance (R-LSe). The items in the scale of Component III mention aspects of the clarity of the feedback message (M-C). In Component IV, the items in the first scale refer to the privacy conditions in which feedback is given (C-P). The items of the second scale, the formative nature of the feedback (C-Fn), refer to different circumstances, such as timing and frequency, of the feedback. The third scale consists of items giving information about the critical nature of the task: tasks with risks of complications, that can be performed in different ways, or a task which can have negative consequences for the patient, or health care (C-Tn).

A repetition of the factor analysis for each component on the final items produced the same results. Therefore, these items were used for further analysis and as scales in the final questionnaire.

The KMO index for statistical adequacy of the factor analytic solution for each component is: 0.75, 0.72, 0.82 and 0.71. The KMO values all approximate the “meritorious” value of 0.80.43 Cronbach’s alphas for the resulting scales were varied from 0.61 to 0.76.

Relationship with the blueprint The initial item pool consisted of questions related to ten of the cells from the blue print. The 46 items from the final questionnaire still represent these ten cells (boldface numbers table 4.1). This confirms our initial assumption that these aspects are key in recipients’ perceptions of feedback.

Subscale and inter-correlations Within each component the inter-correlations between the scales are small, suggesting that the scales within components are largely independent from each other. The largest amount of variance two subscales had in common was 11%, for the Provider- Purposeful and Trustworthy (P-P&T) scale and the Provider- Involved (P-I) scale in Component I. The correlations between all the nine scales vary. The Provider- Detached (P-D) scale, Recipient- High Self-evaluation on task (R-HSe) scale, Recipient Low Self-evaluation on task (R-LSe) scale, Context- Privacy (C-P) scale, and the Context- Critical nature of the task (C-Tn) scale have small (r ≤ 0.29) correlations with the other scales. This suggests they are independent from the other scales.

The Message Clarity (M-C) scale had medium (r = 0.30 to 0.49) correlations with the P-P&T scale, P-I scale, and the Context- Formative nature of the task (C-Fn) scale. This suggests that a person who perceives message clarity as contributing to feedback value,

Table 4.2. Measurement of Perceptions toward Feedback in Clinical Settings (P-FiCS): scale alphas, item means, standard deviations, factor loadings, and communalities

Scale 1 α = 0.76 Purposeful and Trustworthy teaching behavior (P-P&T) 1.Assigns the tasks for me to perform (1).a 2.Observes me directly (3).

3.Explains the task for me before I start (1).

4.Compares my current performance with my past performance(3).

5.First demonstrates the task before I start (1).

6.Tells what I have to change during the observation (3).

7.Is usually the same person (4).

8. Knows me well (4).

M

SD

Factor

h2

1

2

3

91

4.78 5.05 5.36

1.40 1.82 1.39

0.54 0.75 0.73

0.08 -0.07 0.08

0.09 0.03 0.01

0.30 0.57 0.54

5.76

1.10

0.51

0.24

0.00

0.32

5.12 4.61 4.56 5.14

1.31 1.70 1.39 1.37

0.62 0.54 0.43 0.58

0.18 0.04 0.17 0.11

0.05 0.00 0.10 -0.15

0.42 0.29 0.23 0.37

Scale 2 α = 0.64 Involved teaching behavior (P-I) 9.Is an expert on the task I have to perform (1).

10.Observes my performance in person (3).

11.Observes me discreetly (3).

12.Observes me several times (3).

13.Gives feedback directly, not through a third person (4).

14.Is someone who enjoys giving feedback (4).

6.13 6.16 4.98 5.91 6.43 5.52

0.90 0.91 1.35 0.97 0.84 1.15

0.17 0.16 -0.07 0.36 0.06 0.19

0.50 0.73 0.53 0.53 0.72 0.49

-0.06 -0.02 0.35 -0.15 -0.19 -0.09

0.28 0.55 0.40 0.43 0.56 0.29

Scale 3 α = 0.67 Detached teaching behavior (P-D) 15.Observes me just once (3).

16.Is not an expert on the task I have to perform (1).

17.Does not know me very well (4).

18.Is someone who is reluctant to give feedback (4).

19.Gives feedback through a third person (4).

2.71 2.54 2.90 2.48 2.07

1.18 1.19 1.30 1.21 1.10

0.02 0.06 -0.14 0.10 0.20

-0.03 -0.06 0.11 -0.26 -0.45

0.60 0.64 0.70 0.64 0.55

0.37 0.42 0.52 0.49 0.55

M

SD

Component II Feedback Recipient in relation to the task and standard: Stem: Feedback is most instructive when (it concerns a task)…

Factor 1

2

h2

Total n = 639; Two factor solution; KMO = 0.72; Total variance explained 53.8% Scale 1 α = 0.70 High self-evaluation of ability (R-HSe) 1. I can complete the task with ease (8).

2. It is unsolicited by me (10).

3. My performance is better than most of my peers (8).

4. I am satisfied with my performance (8).

4.07 4.62 4.24 4.64

1.34 1.31 1.25 1.27

0.61 0.61 0.82 0.83

0.20 0.02 0.08 0.12

0.41 0.37 0.68 0.71

Scale 2 α = 0.70 Low self-evaluation of ability (R-LSe) 5. That I find difficult (7).

6. I ask for it (10).

7. My performance is worse than most of my peers (8).

8. I am convinced that I could have done better (8).

5.54 5.16 4.91 5.16

1.19 1.55 1.44 1.37

0.02 0.22 0.11 0.08

0.66 0.60 0.83 0.78

0.43 0.40 0.69 0.61

Measuring trainee Perception of the value of Feedback in Clinical Settings (P-FiCS)

Component I Feedback Provider: Stem: I perceive feedback as most instructive when the feedback provider…

Total n = 623; Three factor solution; KMO = 0.75; Total variance explained 41.6%

Component III Feedback Message: Stem: Feedback is most instructive when it…

Chapter 4

92

M

SD

Factor

h2

1

Total n = 639; One factor solution; KMO = 0.82; Total variance explained 38,0% Scale 1: α = 0.76 Message Clarity (M-C) 1. Is tailored to the goals of my training (16).

2. Focuses on main points (16).

3. Addresses both strong points and points for improvement (16).

4. Contains a description of my performance (22).

5. Presents facts and examples to illustrate the main points (16).

6. Goes into details (16).

7. Allows me to ask questions and make comments (22).

8. Addresses weaknesses in a constructive way (16).

0.61 0.52 0.63 0.53 0.69 0.63 0.65 0.65

5.94 5.35 6.22 5.26 6.06 5.76 6.14 6.00

1.03 1.28 0.98 1.14 0.88 1.11 0.87 1.09

M

SD

Scale 1 α = 0.71 Privacy (C-P) 1. It is given in front of patients (28).

2. It is given in front of residents (28).

3. It is given in front of attending physicians (28).

4. It is given in front of medical students (28).

2.52 3.48 3.77 2.95

1.37 1.41 1.42 1.36

0.68 0.78 0.69 0.74

0.09 0.11 -0.06 0.19 0.02 -0.01 -0.13 0.12

0.48 0.64 0.48 0.58

Scale 2 α = 0.64 Formative nature feedback (C-Fn) 5. It has no consequences for my grades (8).

6. It is given during my clinical activities (28).

7. It is given directly after completing a task (28).

8. It is given at regular intervals (28).

4.64 3.93 5.57 5.00

1.45 1.80 1.18 1.52

0.03 0.03 -0.07 -0.04

0.58 0.80 0.64 0.73

0.14 0.04 0.04 0.08

0.36 0.65 0.42 0.54

0.16

0.67

0.49

0.05

0.85

0.73

0.08

0.67

0.50

Component IV Feedback Context: Stem: Feedback is most instructive when…

1

0.65 0.64 0.41 0.29 0.59 0.54 0.51 0.46 Factor 2

3

h2

Total n = 632; Three factor solution; KMO = 0.71; Total variance explained 53.1 %

Scale 3 α = 0.61 Critical nature of the task (C-Tn) 9. It concerns a task that can be performed in several ways (25). 4.77 1.28 0.11 10. It concerns a task which does not have to be done 4.58 1.23 0.03 perfectly (8).

11. It concerns a task that has no negative consequences for the 3.85 1.37 0.19 patient (25). a The number in parenthesis after each item refer to the corresponding blueprint cells.

also perceives feedback as valuable when it is formative in nature and is given by a feedback provider perceived as involved, purposeful and trustworthy.

The P-P&T scale also has a medium correlation with the P-I scale and a high (r ≥ 0.50) correlation with the C-Fn scale. People who highly value feedback from a provider whose teaching behavior is purposeful and trustworthy (P-P&T scale) also perceive involved (P-I scale) teaching behavior and feedback having a formative nature as contributing to the feedback instructiveness.

Age had significant, but small negative correlations with the R-LSe scale r = -0.13 (p < 0.01) and the C-P scale r = -0.13 (p < 0.01).

Demographic group differences Using ANOVA, we found differences in perceived feedback value between groups based

4.4 Discussion We aimed to develop an instrument to measure student perception of the quality and usefulness of feedback. The resulting 46-item Perceptions towards Feedback in Clinical Settings (P-FiCS) instrument has nine scales relating to four components of a cyclic feedback process: (a) the feedback provider’s behavior, (b) the recipient’s own evaluation of task performance, (c) the feedback message, and (d) the feedback context. This study yielded scales with reasonable internal consistency, independent scales within components, scales with medium to high interpretable correlations with scales from other components, scales distinguishing participants according to gender, seniority, nationality, and medical specialty.

Utility of P-FiCS The P-FiCS is useful to stimulate reflection by clerks and residents on clerkships or residency, or to guide discussions about how to handle feedback provided in the clinical setting.44 It takes participants less than ten minutes to complete the questionnaire.

Discussing the outcomes creates awareness in both learners and supervisors of why feedback does not always meet learner expectations, and why not all feedback is perceived as useful or applicable.

Feedback is just one of several topics in many learning climate questionnaires. The P-FiCS provides an opportunity to evaluate the dynamics of feedback in much more detail as individual features (scales) or in combination. The P-FiCS also measures feedback elements that learners judge important: including supervisor, message, task and learning context.

Further, in hospitals where supervisors and learners have a variety of cultural backgrounds, this instrument can reveal cultural differences and, based on the outcomes, provide both learners and supervisors with directions for practice.

Limitations The scales on Involved and Detached teaching behavior and the scales on Low and High Self-evaluations are worded oppositely. We assumed that those questions would be perceived as opposites and thus answered differently. However, in pilot versions of the questionnaire we discovered from observing the answering patterns that opposite worded questions were not necessarily perceived as opposites. We therefore included several items twice, in oppositely worded versions. This leads to seemingly opposite

93 Measuring trainee Perception of the value of Feedback in Clinical Settings (P-FiCS)

on gender, nationality, seniority, and residents’ medical specialty, however, most effect sizes are small.43 Exceptions are the differences related to nationality and residents’ medical specialty. Nationality explains in the P-P&T scale 26%, in the R-LSe scale 8%, in the M-C scale 8% and in C-Fn scale 42% of the variance. Residents’ medical specialty explains 18% of the variance in P-P&T scale, 11% in the C-P scale, and 20% in the C-Fn scale. Trainee’s from the USA value feedback higher (M = 5.56, SD = 0.66) when it is given purposeful and trustworthy than Dutch trainee’s ( M = 4.66, SD =0.81). When feedback has no negative consequences for the trainees it is perceived as more valuable by the USA trainees (M = 5.55, SD = 0.86) compared to the Dutch trainees (M = 4.18, SD = 0.75). On the C-Fn scale we also see difference between medical specialties. Family medicine values feedback with a formative nature lower compared to the other specialties involved in this study (Table 4.4a and 4.4b).

6 5

P-D: Detached teaching behavior

4 3

C-Fn: Formative nature

C-Tn: Critical nature of task

N

P-P&T P-I

-0.17** (0.67)

P-D

R-HSe R-LSe

0.27** 0.03

0.29** (0.70)

4.39 0.97 642 0.16** 0.07

0.16** 0.29** 0.14**

0.16** 0.10*

-0.07 0.28** 0.20** 0.15**

4.78 1.05 639 0.51** 0.18** 0.05

3.18 1.01 639 -0.01

5.84 0.64 639 0.43** 0.47** -0.15** 0.23** 0.20**

5.84 0.64 641 0.03

4.39 0.94 642 0.24** 0.22** 0.11** (0.70)

2.55 0.79 633 0.04

5.86 0.62 640 0.33** (0.64)

5.05 0.87 637 (0.76)

SD

* Correlation is significant at the 0.05 level (2-tailed). ** Correlation is significant at the 0.01 level (2-tailed)

4

C-P: Privacy

Component IV Feedback Context

M-C: Message clarity

8

4

R-LSe: Low self-evaluation of ability

Component III Feedback Message

4

R-HSe: High self-evaluation of ability

Component II Feedback Recipient in relation to task and standard

8

P-I: Involved teaching behavior

Items M

P-P&T: Purposeful and trustworthy teaching behavior

Component I Feedback Provider

Scale

Table 4.3. P-FiCS scale characteristics and inter-correlations. On the diagonal the alpha reliability is given C-P

0.20**

0.45**

(0.64)

C-Fn

0.28** 0.22**

-0.03

-0.14** (0.71)

(0.76)

M-C

Chapter 4

94

(0.61)

C-Tn

5.76 (0.64)*

5.91 (0.60) 2.48 (0.75)

P-D: Detached teaching behavior

4.36 (0.89)

4.74 (1.03) 4.42 (1.01)

* Significant difference (p< 0.006).

C-Tn: Critical nature of task

4.84 (1.08)

3.07 (1.04)

3.37 (0.95)*

5.78 (0.70)

C-Fn: Formative nature

5.87 (0.60)

4.25 (0.91)* 5.00 (1.06)*

4.35 (1.01)

4.18 (0.75)

3.30 (0.93)

5.68 (0.57)

5.51 (0.91)

4.43 (0.95)

2.48 (0.68)

5.87 (0.57)

4.66 (0.81)

Dutch (n = 358-361) M (SD)

4.44 (0.92)

5.55 (0.86)*

3.02 (1.10)*

6.05 (0.67)*

4.78 (0.98)*

4.34 (0.93)

2.63 (0.90)

5.83 (0.68)

5.56 (0.66)*

USA (n = 275-281) M (SD)

Nationality

4.40 (1.00)

4.69 (1.09)

3.42 (0.92)

5.72 (0.62)

5.24 (0.88)

4.42 (0.88)

2.49 (0.77)

5.79 (0.60)

5.12 (0.76)

Junior clerks (n = 137-141) M (SD)

4.41 (1.04)

4.77 (1.10)

3.18 (0.95)

5.93 (0.64)

5.12 (1.11)

4.46 (0.98)

2.43 (0.71)

4.37 (0.89)

4.83 (0.98)

3.04 (1.01)*

5.83 (0.65)

5.24 (0.97)

4.31 (0.94)

2.69 (0.84)*

5.82 (0.66)

5.03 (0.83)

5.04 (0.98) 5.94 (0.58)

Resident (n = 255-259) M (SD)

Senior clerks (n = 240-244) M (SD)

Seniority

Measuring trainee Perception of the value of Feedback in Clinical Settings (P-FiCS)

C-P: Privacy

Component IV Feedback Context

M-C: Message clarity

Component III Feedback Message

4.47 (0.95) 5.30 (0.96)

R-HSe: High self-evaluation of ability

R-LSe: Low self-evaluation of ability

Component II Feedback Recipient in relation to the task and standard

2.67 (0.83)*

5.12 (0.86)

5.00 (0.88)

Male (n = 230-234) M (SD)

Component I Feedback Provider P-P&T: Purposeful and trustworthy teaching behavior P-I: Involved teaching behavior

Scale

Female (n = 401-407) M (SD)

Gender

Table 4.4a. P-FiCS scale means broken down by Gender, Nationality and Seniority

95

5.92 (0.57) 2.86 (0.84)

P-I: Involved teaching behavior

P-D: Detached teaching behavior

5.20 (1.04) 4.32 (0.98)

C-Fn: Formative nature

C-Tn: Critical nature of task

* Significant difference (p< 0.006).

3.00 (1.20)

C-P: Privacy

Component IV Feedback Context

M-C: Message clarity

5.95 (0.70)

4.89 (0.84)

R-LSe: Low self-evaluation of ability

Component III Feedback Message

4.09 (0.96)

R-HSe: High self-evaluation of ability

Component II Feedback Recipient in relation to the task and standard

5.45 (0.74)

Mean (SD)

P-P&T: Purposeful and trustworthy teaching behavior

Component I Feedback Provider

Scale

Internal medicine (N = 57-61)

4.22 (0.70)

5.39 (0.88)

2.18 (0.94)

5.96 (0.62)

4.75 (0.93)

4.08 (0.77)

2.59 (0.71)

5.84 (0.51)

5.23 (0.80)

Mean (SD)

Pediatrics (N = 30-31)

4.38 (0.76)

5.45 (0.86)

3.31 (1.15)

6.09 (0.67)

5.61 (0.68)

4.70 (0.98)

3.21 (0.93)

5.72 (0.78)

5.41 (0.74)

Mean (SD)

Emergency Medicine (N = 21)

4.39 (1.00)

4.28 (0.77)

3.04 (0.92)

5.74 (0.55)

5.51 (0.99)

4.40 (0.90)

2.52 (0.76)

5.73 (0.79)

4.64 (0.83)

Mean (SD)

Family Medicine (N = 64-67)

Medical specialty

4.42 (0.86)

4.68 (0.86)*

3.33 (1.04)*

5.70 (0.64)

5.39 (0.95)*

4.42 (1.01)

2.62 (0.88)

5.86 (0.61)

4.86 (0.72)*

Mean (SD)

Other (N = 65-73)

Table 4.4b. Overview of group differences between residents in different Medical Specialties in means and standard deviations for each scale

Chapter 4

96

Recommendations The P-FiCS offers opportunities for further research. Replication of this study will provide opportunities for reliability estimation using new datasets. Administering the questionnaire on two occasions among a new participant group and determining the test-retest reliability would add information about the stability of the scale data.

Relating the P-FiCS to other instruments - the feedback environment scale,47 the desire for feedback scale,48 and the feedback propensity scale49 - can give additional information about the validity of the P-FiCS constructs. To determine whether scale scores differ among tasks, questionnaire data can be related to specific activities in clinical settings, e.g., patient presentations and physical examination tasks. Correlations between scales and variables such as feedback recipients’ clerkship grades, or results in medical school could also be informative.

When medical specialists are included in a sample we will be able to determine whether the questionnaire is applicable to different subgroups, whether medical specialists’ perceived value of feedback on their daily routines differs from medical students and residents, and whether or not differences are seen among medical specialties, cultures, gender, and seniority.

The current scale scores give directions for practice. In general, medical trainees do not like to receive feedback in public, but our findings show that residents value this as even less instructive than students. These results agree with studies that stress the importance of a ‘safe’ feedback environment. Further, the nature of the task contributes to how feedback recipients value feedback. When the task gives opportunities for variation in performance and the task has no negative consequences for patients, then the task performance is less stressful and the feedback is felt as more informative. Research on the relation between task and feedback perception has hardly received attention in medical education.

Feedback is also perceived as instructive when it has no consequences for grades or formal evaluations. When feedback recipients perceive it as only beneficial for their own

97 Measuring trainee Perception of the value of Feedback in Clinical Settings (P-FiCS)

scales in the P-FiCS. The low correlations between these scales are not surprising. Regardless of whether or not recipients judge their own task performance as high or low, their perception can be that receiving feedback is potentially instructive. However, we found that clerks and residents see feedback as more instructive when they evaluate their own ability as low (M = 5.84, SD = 0.64) compared to high (M = 4.39, SD = 0.94).

Similarly, both detached and involved teaching behavior are perceived as contributing to feedback value, although learners perceive feedback as considerably more instructive if teachers are involved (M = 5.86, SD = 0.62) compared to detached (M = 2.55, SD = 0.79).

Three intended scales could not be included in the questionnaire. One six-item scale, intended to measure “perceived similarity between feedback recipient and feedback provider”, could not be interpreted because some questions appeared ambiguous. Two scales, meant to measure “recipient’s task acquaintance” and “feedback environment” were left out because they contained only two items. However, several studies have indicated that these aspects of feedback are important. In research on social interaction “perceived similarity”45 is an important aspect, and studies on workplace-based education emphasize the importance of the learning environment.46 Future research should include items about “perceived similarity” to explore additional scales of perceived feedback.

Chapter 4

98

learning process, they value feedback as instructive. Medical educators may be led by this knowledge when giving feedback in the clinical setting by explaining the purpose of the feedback.

Although the scales reflect the recipientsâ&#x20AC;&#x2122; perceptions of valuable feedback, and does not give information about its effectiveness, its perceived instructive value can be seen as a condition for its use, acceptance, and impact.22, 41

References 1.

2.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.

99 Measuring trainee Perception of the value of Feedback in Clinical Settings (P-FiCS)

3.

Azevedo R, Bernard RM. A meta-analysis of the effects of feedback in computer based instruction. J Educ Compu Res 1995;13:111-127.

Balzer WK, Doherty ME, O’Connor R. Effects of cognitive feedback on performance. Psychol Bull 1989;106:410-433.

Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, O’Brien MA, Johansen M, Grimshaw J, Oxman AD. 2012. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev, 6, CD000259 DOI: 10.1002/14651858.

Kluger AN, DeNisi A. Effects of feedback intervention on performance: a historical review, meta-analysis, and a preliminary feedback intervention theory. Psychol Bull 1996;119:254284.

McAfee RB, Winn AR. The use of incentives/feedback to enhance work place safety: A critique of the literature. J Safety Res 1989;20:7-19.

Steinman MA, Ranji SR, Shojania KG, Gonzales R. Improving antibiotic selection: a systematic review and quantitative analysis of quality improvement strategies. Med Care 2006;44:617-628.

Veloski J, Boex JR, Grasberger MJ, Evans A, Wolfson DB.. Systematic review of the literature on assessment, feedback and physicians’ clinical performance: BEME Guide No. 7.

Med Teach 2006;28:117-128.

Pritchard RD, Harrell MM, DiazGranados D, Guzman MJ. The productivity measurement and enhancement system: a meta-analysis. J Appl Psychol 2008;93:540-567.

Shute VJ. Focus on Formative feedback. Rev Educ Res 2008;78:153-189.

Cheraghi-Sohi S, Bower P. Can the feedback of patient assessments, brief training, or their combination, improve the interpersonal skills of primary care physicians? A systematic review. BMC Health Serv Res 2008;8:179. DOI:10.1186/1472-6963-8-179 Hattie J, Timperley H. The power of feedback. Rev Educ Res 2007;77:81-112.

Isaacson JH, Posk LK, Litaker DG, Halperin AK. Resident perception of the evaluation process. J Gen Intern Med 1995;10(S4):89.

Gil DH, Heins M, Jones PB. Perceptions of medical school faculty members and students on clinical clerkship feedback. J Med Educ 1984;54:856-864.

Liberman AS, Liberman M, Steinert Y, McLeod P, Meterissian S. Surgery residents and attending surgeons have different perceptions of feedback. Med Teach 2005;27:470-472.

Perera J, Lee N, Win K, Perera J, Wijesuriya L. Formative feedback to students: the mismatch between faculty perceptions and students expectations. Med Teach 2008;30:395-399.

Sargeant J, Mann K, Ferrier S. Exploring family physicians’ reactions to multisource feedback: perceptions of credibility and usefulness. Med Educ 2005;39:497-504.

Teunissen PW, Stapel DA, van der Vleuten C, Scherpbier A, Boor K, Scheele F. Who wants feedback? An investigation of the variables influencing residents’ feedback-seeking behavior in relation to night shifts. Acad Med 2009;84(7):910–917.

Bok HGJ, Teunissen PW, Spruijt A, Fokkema JP, van Beukelen P, Jaarsma DA, van der Vleuten CP.Clarifying students’ feedback-seeking behaviour in clinical clerkships. Med Educ 2013;47(3):282-291.

Crommelinck M, Anseel F. Understanding and encouraging feedback-seeking behaviour: a literature review. Med Educ 2013;47(3):232-241.

Galbraith RM. Got feedback? Med Educ 2013;47(3):224-225.

Ten Cate OTJ. Why receiving feedback collides with self determination. Adv Health Sci Educ Theory Pract 2013; 18:845-849.

Watling CJ, Lingard L. Toward meaningful evaluation of medical trainees: the influence of participants’ perceptions of the process. Adv Health Sci Educ Theory Pract 2012;17:184-193.

Chapter 4

100

23. Molloy E, Boud D. Seeking a different angle on feedback in clinical education: the learner as seeker, judge and user of performance information. Med Educ 2013;47:227-229.

24. Pelgrim EA, Kramer AW. How can medical education benefit from the evidence on learners seeking and using feedback? Med Educ 2013;47:225-227.

25. Archer JC. State of Science in health professional education: Effective feedback. Med Educ 2010;44:101-108.

26. Bing-You RG, Paterson J, Levine MA. Feedback falling on deaf ears: resident’s receptivity to feedback tempered by sender credibility. Med Teach 1997;19:40-44.

27. Bing-You RG, Trowbridge RL. Why medical educators may be failing at feedback, JAMA 2009;302:1330-1331.

28. van Hell EA, Kuks JBM, Raat AN, van Lohuizen MT, Cohen-Schotanus J. Instructiveness of feedback during clerkships: influence of supervisor, observation and student initiative.

Med Teach 2009;31:44-50.

29. Rubin DJ, Rarey KY.. Resident and faculty feedback: the student’s perspective. Journal of Thomas Jefferson University Hospital Department of Internal Medicine 2004;6:25-26.

30. Eva KW, Armson H, Holmboe E, Lockyer J, Loney E, Mann K, Sargeant J. Factors influencing responsiveness to feedback: on the interplay between fear, confidence, and reasoning processes. Adv Health Sci Educ Theory Pract 2012;17:15–26.

31. van de Ridder JMM, Stokking KM, McGaghie WC, ten Cate ThJ. What is feedback in clinical education? Med Educ 2008;42:189-197.

32. Comrey AL. Factor analytic methods of scale development in personality and clinical psychology. J Consult Clin Psychol 1988;56:754-761.

33. Gorsuch RL. Exploratory factor analysis: its role in item analysis. J Pers Assmnt 1997;68:532-560.

34. Wetzel AP. Factor analysis methods and validity evidence: A review of instrument development across the medical education continuum. Acad Med 2012;87:1060-1069.

35. McGaghie WC, Richards BF, Petrusa ER, Camp M, Harward DH, Smith AS, Willis SE. Development of a measure of medical attitudes toward clinical evaluation of students. Acad Med 1995;70:47-51.

36. McGaghie WC, Van Horn L, Fitzgibbon M, Telser A, Thompson JA, Kushner RF, Prystowsky JB. Development of a measure of attitude toward nutrition in patient care. Am J Prev Med 2001;20:15-20.

37. Henson RK, Roberts JK. Use of exploratory factor analysis in published research: common errors and some comment on improved practice. Educ Psychol Meas 2006;66:393-416.

38. Lasswell, Harold (1902-1978). 2012. In: C. Calhoun (Ed.), Dictionary of the Social Sciences.

Oxford Reference Online. Oxford University Press. Retrieved March 8, 2013. http://www. oxfordreference.com.proxy.library.uu.nl/view/10.1093/acref/9780195123715.001.0001/acref9780195123715-e-930?rskey=AbyVOr&result=851&q 39. Johnson W. People in quandaries. The semantics of personal adjustment. Harper & Row Publishers, New York 1946.

40. Johnson W. Speech and personality. In: Bryson L. editor. The communication of ideas. A serie of addresses. New York: Cooper Square Publishers; 1964.

41. Ilgen DR, Fisher CD, Taylor SM. Consequences of individual feedback on behavior in organizations. J Appl Psychol 1979;64:349-371.

42. Kaiser HF. An index of factorial simplicity. Psychometrika 1974;39:31-36.

43. Cohen J. Statistical Power Analysis for the Social Sciences. Mahwah NJ, Lawrence Erlbaum Publishers 1988.

44. Hulse-Killacky D, Orr JJ, Paradice LV. The corrective feedback instrument - revised. J Specialists Group Work 2006;3:263-281.

45. Liden RC, Wayne SJ, Stilwel D. A longitudinal study on the early development of leadermember exchanges. J Appl Psychol 1993;78:662-674.

101 Measuring trainee Perception of the value of Feedback in Clinical Settings (P-FiCS)

46. Dornan T. Experienced based learning. Learning clinical medicine in workplaces [PhD thesis]. Manchester: Tim Dornan; 2006.

47. Steelman LA, Levy PL, Snell AF. The feedback environment scale: construct definition, measurement, and validation. Educ Psych Meas 2004;64:165-184.

48. Snyder CR, Ingram RE, Handelsman MM, Wells DS. Desire for personal feedback: who wants it and what does it mean for psychotherapy? J Personality 1982;50:316-330.

49. Herold DM, Parsons CK. Assessing the feedback environment in work organizations: development of the job feedback survey. J Appl Psych 1985;70:290-305.

Framing of feedback impacts student’s satisfaction, self-efficacy and performance

Published as: Van de Ridder JMM, Peters CMM, Stokking KM, McGaghie WC, Ten Cate OThJ. Framing of Feedback Impacts Student’s Satisfaction, Self-efficacy and Adv Health Sci Educ Theory Pract 2014; Early online pre-publication

Chapter 5

104

Abstract Purpose Feedback is considered important to acquire clinical skills. Research evidence shows that feedback does not always improve learning and its effects may be small. In many studies, a variety of variables involved in feedback provision may mask either one of their effects. E.g., there is reason to believe that the way oral feedback is framed may affect its effect if other variables are held constant. In a randomised controlled trial we investigated the effect of positively and negatively framed feedback messages on satisfaction, self-efficacy, and performance.

Method A single blind randomised controlled between-subject design was used, with framing of the feedback message (positively-negatively) as independent variable and examination of hearing abilities as the task. First year medical students’ (n = 59) satisfaction, self-efficacy, and performance were the dependent variables and were measured both directly after the intervention and after a two weeks delay.

Results Students in the positively framed feedback condition were significantly more satisfied and showed significantly higher self-efficacy measured directly after the performance. Effect sizes found were large, i.e. partial η2 = 0.43 and η2= 0.32 respectively. They showed a better performance throughout the whole study. Significant performance differences were found both at the initial performance and when measured two weeks after the intervention: effects were of medium size, respectively r = -0.31 and r = -0.32. Over time in both conditions performance and self-efficacy decreased.

Conclusions Framing the feedback message in either a positive or negative manner affects students’ satisfaction and self-efficacy directly after the intervention be it that these effects seem to fade out over time. Performance may be enhanced by positive framing, but additional studies need to confirm this. We recommend using a positive frame when giving feedback on clinical skill.

5.1 Introduction

Satisfaction Evaluating students’ perception of provided feedback is important. Positive reactions of students ( e.g. ‘I liked to receive this feedback’) can motivate, stimulate their learning

105 Framing of feedback impacts student’s satisfaction, self-efficacy and performance

eedback is considered to be important in clinical training.1-5 In clinical education feedback has been defined as ‘Specific information about the comparison between trainee’s performance and a standard given with the intent to improve trainee’s performance’.6 Students view feedback as supportive when participating in the clinical context,2 and they rate “giving constructive feedback” as the second most important behavioral characteristic of a preceptor.7 Feedback enhances learning and may lead to more responsibility and autonomy.2 Most importantly, effective feedback has the potential to reduce trial-and-error behavior by trainees in patient care.

However, from meta-analyses and reviews we know that feedback does not always improve performance but can also decrease performance,8 and that effect sizes of feedback interventions are small.9-10 Sometimes feedback is perceived as effective, but performance outcomes after the feedback is received do not reflect a change in behavior.11 A possible explanation for the diverse feedback effects are the different ways in which it is provided.12-14 Valence framing -framing the feedback message positively or negatively- is one variety.15-16 Differences in supervisors’ feeling of time constraint,17 feelings towards the role of teacher,1 differences in training in providing feedback,18 or awareness of the importance of feedback5 have been suggested to underlie the variety in such framing.

Valence framing has defined as casting ‘the same critical information in either a positive or a negative light’ (p.150).16 So, framing refers to the packaging of the message and it is independent of the message content, which can consist of positive -about good points- or negative feedback -about points for improvement-, or both. “You did this quite well, but there are some points for improvement…” would be considered a positively framed feedback message. Negatively framed feedback is: “You did not do this correctly. You should…” In both examples the content of the feedback message is negative.

Message framing is an important aspect of communication. In studies outside the medical field it has been shown to influence the outcomes of punishment,15 the process of decisions taking, especially in the presence of risk,16, 19-21 evaluations of objects,16, 22 persons’ attitude certainty,23 and persuasiveness in communication.16 We identified two studies specifically focusing on positive and negative framing of a feedback message, but the task and the context in these studies are non-medical.24-25 Since framing is an important aspect of communication, and it also is applied in giving feedback we would like to measure the impact on a clinical task performance in a medical setting. In this study we evaluated the effect of positively and negatively framed feedback on outcome measures such as students’ satisfaction, self-efficacy, and performance. These outcome measures relate to two levels of Kirkpatrick’s hierarchy of evaluation outcomes:26 reaction (satisfaction and self-efficacy) and learning (performance).

Chapter 5

106

and lead to loyalty,26-27 even if their perceptions are not accurate.28 Research shows that positive feedback on performance positively influences students’ satisfaction,29 receiving process feedback in a virtual team leads to an increase of satisfaction by team members30 and a comparison of students’ satisfaction after receiving praise or feedback shows higher satisfaction for receiving praise.11 We expect students’ satisfaction to be higher in a ‘positively framed feedback’ than in a ‘negatively framed feedback’ condition.

Self-efficacy Bandura defines self-efficacy as: ‘people’s judgments of their capabilities to organize and execute courses or action required to attain designated types of performances’ (p.391). It can be classified as self-perception.31 This self-perception is based on enactive attainment, vicarious experiences, verbal persuasion, and one’s physiological state.

Self-efficacy influences for example choice behavior, effort expenditure and persistence, thought patterns, and emotional reactions. Self-beliefs contribute in various ways to psychosocial functioning.31 Self-enhancement theory suggested that everyone wants to improve their own functioning and therefore only positive feedback -about good points- is effective.32 Results in line with this theory show that particularly negative feedback reduces self-efficacy and positive feedback increases self-efficacy.33 Negative feedback has more influence on subjects with low self-efficacy compared to subjects showing high self-efficacy.34 Self-verification theory states that when the feedback is in line with a person’s selfconcept they will endorse the feedback as valid, even when it is negative.32 Empirical research in line with self-verification theory shows that negative feedback is better accepted by people with low self-efficacy about a specific competency compared to subjects with high self-efficacy.35 Performance Based on research evidence it is hard to formulate how feedback framing will influence performance. Generally feedback is considered to have a small or moderate impact on performance9 effect sizes are often small,8 or show an enormous variation.36 When feedback is given systematically, by a credible source, combined with other interventions such as education or guidelines, it appears to be more effective.9-10 Most studies on the effect of feedback only report direct effects, not many collect data after a time delay. To detect the sustained influence of feedback over time we decided to repeat measures of performance and self-efficacy after two weeks.

The following research question was formulated: What is the effect of feedback framing on students’ satisfaction, self-efficacy, and performance?

5.2 Method Design A single-blind randomised controlled between-subject design was used, with feedback framing (positive-negative) as the independent variable. Dependent variables used were students’ satisfaction, self-efficacy, and performance outcomes.

Task The task given to students was to practice the Weber and Rinne tuning fork procedures to test hearing deficiencies on a standardized patient (SP) according to guidelines pro-

Framing In both conditions students received a feedback message with negative content, provided by a final year medical student with stage-play experience, acting as an experienced physician familiar with the W&R task. In the positive framing condition, feedback was voiced as: “You did this well; some tips are…” Feedback framed negatively started with: “This is not well done; you should change…”

To ensure that the interaction was natural, the feedback provider was allowed to engage in an ordinary dialogue with the student. To keep the feedback message focused, the feedback provider choose, after observing students’ performance, from a list of four pre-selected feedback points one item which feedback point was most suitable for this student. The four pre-selected points consisted of students’ most frequent mistakes in performing the W&R task as determined in a pilot study.40 Besides the pre-selected points, no other feedback was given to optimally keep the conditions under control.

The feedback provider was trained in using the experimental protocol, selecting the feedback points, framing the feedback in a similar manner for both conditions, and in engaging in a natural dialogue that would not negatively affect the conditions in the study.

Participants All 210 first year medical students from the University Medical Center Utrecht (UMCU) potentially available in the study period were approached orally and by email three weeks before the study and asked to participate. The medical student population in the Netherlands consists of a stable 65-70% woman since early 1990s.41 By quota sampling we established a group, representative for gender of the medical student population. It was explained that “volunteers were sought for research on clinical skills teaching” and that data would treated confidentially. Participants were asked to sign an informed consent form. Students were informed that they would receive a 7.50 Euros reward after completing the experiment and the follow-up.

Procedure The experiment was carried out in June 2007, in the skills lab of the UMCU. Other instructional events were unlikely to interfere with the study, as at this time of the year no classes take place and students are preparing for exams.

107 Framing of feedback impacts student’s satisfaction, self-efficacy and performance

vided by an otorhinolaryngologist on an instructional video. This test is used in the clinical setting in both the otolaryngology and neurology.37-38 The task is complex, in the sense that it consists of several elements: explaining the task to the patient, placing the tuning fork at patient’s forehead and mastoid bone and asking the right questions at the right time. The students need to combine the retrieved information and diagnose whether the patient has a hearing impairment and if so, if it is a sensorineural or a conductive impairment.38 The student concludes the task with informing the patient about the findings. For a detailed description we refer to a short video-instruction.39 This task is suitable for our experiment because the task is not time consuming to perform, it is observable from a video recording, the pathology is easy to simulate by a SP, and it has obvious relevance for medical training. The curricular approach for all first year medical students had been the same prior to our study. The first year students were not acquainted with the W&R task, as it is only taught in the second year of this medical curriculum.

Chapter 5

108

To collect the outcome measures for this study, each individual went through the route according to figure 5.1. By drawing lots, students were randomized by one investigator (CP) over the two conditions (at time T0). Males and females were almost equally divided over the two conditions. The proportion in the sub groups were representative for the medical student population in the Netherlands (figure 5.1). Each student was asked to rate his/her self-efficacy in performing the W&R task (SE1) at time T1. Next, the student watched an instructional video about the W&R task procedure. At time T2 each student carried out the W&R task (performance 1 (P1)) on a SP while being observed by the feedback provider. The SP was instructed to silently signal to the feedback provider when a procedure was not correctly done, eg., when a tuning fork was not firmly placed upon the mastoid bone or the forehead. Subsequently, the student would rate his or her self-efficacy again (SE2). Directly after this, the student would receive negatively or positively framed feedback from the supervisor. Then, the student was to perform the tests for the second time on the same SP (P2), and would rate their self-efficacy for the third time (SE3) and fill out the satisfaction (SAT) scale (T3). Two weeks later a followup took place (T4). The student would rate self-efficacy for the fourth time (SE4), and perform the W&R task for the third time on the same SP (P3) All performances were video taped.

Instruments Students’ satisfaction with the video instruction, training opportunities, and the received feedback is measured with a five-item scale.

A visual analogue scale (VAS) was used to measure self-efficacy regarding the detection of hearing loss by using the W&R task (0 mm = ‘I am extremely confident’, 100 mm = ‘I am extremely unconfident’). This procedure for measuring students’ self-efficacy towards medical skills has been used in a study by Turner.42 This study provides evidence for the validity of the instrument for a similar purpose.

Video recordings of the students’ performances for each element of the W&R task were scored by two independent raters (CP and MvdR) using a 13-item dichotomous observational checklist (0 = not performed; 1 = performed).

Data analysis Correlations were determined to get insight in the relationship between the variables satisfaction (SAT), performance (P1, P2, P3) and self-efficacy (SE1, SE2, SE3 and SE4).

Whenever assumptions of normality, linearity, homogeneity of variances, and/or homogeneity of regressions slopes were violated, a Mann-Whitney U test was performed, otherwise a mixed model analysis or T-test was used to compare framing effects on satisfaction (SAT), performance (P1, P2, P3) and self-efficacy (SE1, SE2, SE3 and SE4).

A balance between type I and type II error was sought by setting α = 0.10, because of the relatively small group. Applying a Bonferroni correction for multiple statistical testing on the six measurements of the dependent variables resulted in a significance level of (0.1/5 = 0.02).44 Effect sizes (ES) were reported using partial η2 and r. Small, medium, and large effect sizes are respectively 0.01, 0.06, and 0.14 for partial η2 (for those unfamiliar with partial η2, this is equal to d= 0.2, 0.5. and 0.8) and 0.1, 0.3, and 0.5 for r.45 All analyses were done with subjects having a complete data set (n = 59). Subjects with incomplete data sets (n = 15) were removed (figure 5.1).

109

5.3 Results Participants Seventy-four students participated in the study. Sixty-one (82%) students completed all W&R tasks of the study. In the positive and negative frame condition 4 (10%) and 9 (24%) students respectively dropped out at T4 (figure 5.1). We had complete data sets of 59 students (Figure 5.1). Their mean age was 19.5 (SD = 2.3) No significant differences in age and sex distribution between conditions were found.

A comparison of gender, age distribution and performance at baseline, pretest and posttest between the drop-outs and students with a complete data set, did not show significant differences.

Quality of the instruments In this study, the rather complex task to be carried out appeared to have never done perfectly by any of the first year students. In all cases, it appeared theoretically possible to provide positive as well as negative feedback.

The satisfaction scale showed an internal consistency of Cronbachâ&#x20AC;&#x2122;s Îą = 0.85.

The VAS for measuring self-efficacy is a one-item instrument and cannot yield reliability estimates.

Framing of feedback impacts studentâ&#x20AC;&#x2122;s satisfaction, self-efficacy and performance

Figure 5.1. CONSORT flow-chart43 of the chronological procedure of the trial and the amount of participants who participated in each stage and were included in the analyses

33

33

33

T2 Pre-test

T3 Post-test

T4 Two weeks

2.6 †

3.8†

M

0.7

0.7

SD

Satisfaction

2.4-2.8

3.6-4.0

90% CI

45.5

50.9 †

61.0

53.8

48.3

73.5†

63.3

53.3

M

23.1

21.8

17.5

22.0

19.2

11.5

15.4

17.2

SD

Self-efficacy*

37.7-53.2

43.6-58.2

55.1-66.9

46.4-61.1

42.6-54.0

70.1-76.9

58.8-67.8

48.3-58.4

90% CI

5.8

6.9

6.8

6.7

7.5

7.5

M

1.5

1.6

1.2

1.7

1.3

1.2

SD

6.0†

7.0

7.0†

7.0†

8.0

8.0†

Mdn

Performance

†) These values significantly differ in the positively and negatively framed feedback condition. *As most readers will associate a high VAS scores with a high self-efficacy, these results are reported after reversal of the values of the VAS as used in our study.

26

26

T3 Post-test

26

T2 Pre-test

T4 Two weeks

26

T1 Baseline

Negative frame

33

T1 Baseline

Positive frame

N

5.3-6.3

6.3-7.4

6.4-7.2

6.2-7.2

7.2-7.9

7.1-7.8

90% CI

Table 5.1. Means (M), Standard Deviations (SD), Medians (Mdn) and 90% Credibility Intervals (CI) of satisfaction, self-efficacy, performance measures by feedback framing condition and time

Chapter 5

110

The inter-rater agreement between the raters of the observational checklist for scoring the video recordings was taken as a reliability indicator for the performance measurement.46 Four items showed low inter-rater agreement (Cohen’s κ ≤ 0.45) and were removed. The nine remaining items of the checklist used as performance measurement had high average inter-rater agreement (Cohen’s κ = 0.78). The sum scores on the checklist were used as performance outcome (0 = minimum; 9 = maximum).

Figure 5.2 self-­‐ efficacy

80

Positive frame

75

Negative frame

70 65 60 55 50 45 40 T1

T2

T3

T4

Figure 5.3. performance

Figure 5.2. Self-efficacy (M) by feedback condition over time

9

Positive frame

8.5

Negative frame

8 7.5 7 6.5 6 5.5 5 T2

T3

T4

Figure 5.3. Performance score (Mdn) by feedback condition over time

111 Framing of feedback impacts student’s satisfaction, self-efficacy and performance

Assumptions SE3 showed a significant Levene’s test outcome (F(1,57) = 13.7, p < 0.01), indicating no homogeneity of regression slopes. The performance scores (P1, P2, and P3) were not

normally distributed and therefore Mann-Whitney U tests were performed.47

Chapter 5

112

Correlations Correlations were determined with Spearmanâ&#x20AC;&#x2122;s rank correlation coefficient. There was a positive relationship between SE1 and the first performance (P1) rs = .26, p < .05 and with SE2, rs = .34, p < .01. SE2 correlates high with SE3 (rs = .53, p < .01). SE3 has a positive relationship with satisfaction (rs = .56, p < .01).

The first performance (P1) positively correlated with P2 (rs = .34, p < .01), P3 (rs = .28, p < .05), satisfaction (rs = .26, p < .05) and SE4 (rs = .43, p < .01).

Effect of feedback framing on satisfaction An independent samples t-test revealed a significant difference in satisfaction between the positively (M = 3.8, SD = 0.7, CI = 3.6-4.0) and negatively (M = 2.6, SD = 0.7, CI = 2.4-2.8) framed feedback condition; t (57) = 6.7, p < 0.001, ES partial Ρ2 = 0.43 (equal to d = 1.73) (Table 5.1).

Effect of feedback framing on self-efficacy The self-efficacy scores increased after the first task performance (SE2). They further increased in the positively framed feedback condition and decrease in the negatively framed feedback condition at SE3, and at SE4 in both conditions the score was lower than the initial baseline score. A linear mixed model analysis was employed to evaluate the effect of framing on the improvement concerning self-efficacy. The interaction between time and the framing conditions was tested to examine changes of the effect over time. The differences in self-efficacy scores between the two conditions were not equal over time. The linear mixed model showed an interaction effect between self-efficacy and time (F (3,57) = 10.7, p < 0.01). Consequently, we tested the differences between the groups at the specific time points. Directly after the feedback intervention (T3) the group in the positive framing condition had a significantly higher self-efficacy score than the group in the negative framing condition t (57) = 5.11, p < 0.00, ES partial Ρ2 = 0.32. We did not find significant differences at T1, T2 and T4 (table 5.1, figure 5.2).

Effect of feedback framing on performance Between T2 and T3 performance scores stayed the same, and they decreased at T3. A Mann-Whitney U test revealed a significant difference between P1 and P3. During P1 students in the positively framed feedback condition (Mdn = 8.00) performed better than students in the negatively framed condition (Mdn = 7.00) (U = 277, Z = -2.40, p < 0.02, r = -.31). No significant difference was found during P2 (U = 308, Z = -1.91, p < 0.06. r = -.25). However after two weeks, the students in the positively framed feedback condition (Mdn = 7.00), performed better than the students in the negatively framed condition (Mdn = 6.00) U = 272, Z = -2.43, p < 0.02, r = -.32 (table 5.1, figure 5.3).

In contrast, analyses of covariance using P2 and P3 as dependent measures with P1 performance as a covariate indicated that there were no significant differences between positively and negatively framed conditions (P2: F (1, 56) = 1.06, p = 0.31; P3: F (1, 56) = 2.45, p = 0.12).

5.4 Discussion When medical students receive a feedback message with a negative content -about points for improvement- the framing of the message, either positively or negatively, affects their satisfaction with the feedback, their self-efficacy regarding task performance and their task performance. We conclude that the positively framed feedback

113 Framing of feedback impacts student’s satisfaction, self-efficacy and performance

group was more satisfied, had higher self-efficacy immediately after receiving feedback, and performed better two weeks after receiving feedback than the group in the negatively framed condition. In addition, we found that in the negatively framed feedback condition, students showed significantly lower performance scores at T2, i.e. even before feedback was received.

How do we explain our findings? The results on satisfaction show a pattern which is in line with other studies: positive feedback messages, such as compliments, praise11 or high performance feedback29 increase students’ satisfaction.

The changes over time in the SE scores can be explained by both the influence of the W&R performance and the feedback. Before T1 students had neither seen the instructional video nor performed the task. The explanation on the video about the W&R performance and its actual performance gave students an impression of the task difficulty in relation to their own capability. This might explain why SE scores for both conditions are higher at SE2. Between SE2 and SE3 students had either received positively or negatively framed feedback and performed the task again. It appears that the feedback affects their feeling of competence regarding the W&R task, which can explain further increase of self-efficacy in the positively framed feedback condition and decrease in the negatively framed feedback condition. How do we know self-efficacy was not affected by performance instead of by the feedback? No significant correlation between P2 and SE3 was found, which we interpret as SE3 being more influenced by the feedback than by P2. The time between SE3 and SE4 is two weeks, in which students have not practiced the W&R task. The lack of practice of this new, complex task might explain why SE4 dropped so much. At SE4 students were no longer ignorant and they knew what to expect regarding task difficulty. This may explain why self-efficacy at T4 was even lower than at T1.

The findings on performance are somewhat puzzling. At P1 students in the positively framed feedback condition performed significantly better than in the negatively framed condition. Is this coincidence or an experimental effect? A possible explanation is that students in both conditions unconsciously ‘perceived’ signals from feedback provider’s non-verbal behavior and tone of voice about the message type, and that this influenced their first performance, as the actor was aware of the condition and may have unconsciously disclosed this awareness while briefly instructing the candidate. We see a similar phenomenon in situations in which bad news is transmitted: feedback recipients often have a ‘feeling’ of what comes.48 It is very unlikely that the SP could have been instrumental in any unconscious revelation of the condition prior to T2 as she was not pre-informed about the experimental condition of the students.

We observed a non-significant P2 difference suggesting a benefit in the positively framed condition with a close to medium effect size (r = -0.25). A larger cohort of students might have generated a significant effect. The absence of a significant effect might also be a consequence of task type. Kluger and DeNisi address this possibility,8 suggesting that feedback on complex tasks is less effective than on easy tasks. A possible explanation is that feedback about a complex psychomotor task might need more time to be processed and to sink in, compared to feedback on single task, or feedback on cognitive task. The effect found at P3 seems to illustrate this.

Another explanation is that feedback given on one specific aspect of the task is applied in the next performance but at the expense of the good performance of other task aspects, and this does not lead to an increase of overall performance.

Chapter 5

114

The students were randomly placed in a feedback condition; however, the two conditions were significantly different on initial performance and differences in the learning processes are retained. Despite the random allocation of students to conditions, we cannot exclude the possibility that by chance the students in de positively framed condition were somewhat better from the outset.

We do not think that drop-out of subjects has affected our results, given the fact that no differences in outcomes were found at baseline (T1), pretest (T2) and posttest (T3) when the analyses were repeated with inclusion of the data from the drop-outs.

All students received feedback from the same person, and they all performed the task on the same SP, so this does not offer an alternative explanation of our findings.

A limitation of this study is that we were not able to control for all possible influences.

For reasons of ecological validity the feedback dialogue had to sound natural. Therefore, the supervisor had received only global guidelines on what had to be said during the encounter, but he was specifically trained in similar framing of the feedback in the two conditions. Thus, small variations in the feedback dialogues might have been present. No indications are present that the specific feedback content caused the effects we found. All students received feedback on task aspects they could improve. Difficulties with Weber and Rinne tuning fork test in medical students had previously been explored, revealing two topics on which feedback is most necessary: (a) doctor-patient communication and (b) how to use the tuning fork.40 This study was only performed in the content area of the Weber and Rinne tuning fork test, and only in one institution. Additional studies in other contexts and content areas are needed to know if our findings can be generalized. The fact that all students received feedback from one and the same person, and they all performed the task on the same SP eliminates the possibility of confounding of framing condition and SP, but it is a limitation from the perspective of the generalizability of the results.

Framing and communicating an oral feedback message in daily life cannot be disentangled from the supervisorâ&#x20AC;&#x2122;s tone of voice, facial expressions, and body posture. We conclude that the results of this study are caused by the positive versus negative framing of the feedback, plus that non-verbal cues as tone-of-voice might have strengthened the impact of framing on the studentâ&#x20AC;&#x2122;s self-efficacy and satisfaction.

Implications What is the meaning of these results for daily practice in the clinical learning environment? The positive formulation led to more satisfied students with higher self-efficacy immediate after the feedback. Although satisfaction and self-efficacy are measured on the perceptual level, we know perceptions affect behavior.28, 49 Further, we see that in the positively framed feedback condition the pre-existing differences still exist after two weeks, which adds to the recommendation to give feedback using a positive framing. We acknowledge however that further studies need to confirm this recommendation.

References 1.

2.

3.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

115 Framing of feedback impacts student’s satisfaction, self-efficacy and performance

4.

Bowen JL, Irby DM. Assessing quality and costs of education in the ambulatory setting: a review of the literature. Acad Med 2002;77:621-680.

Dornan T. Experienced based learning. Learning clinical medicine in workplaces [PhD thesis]. Manchester: Tim Dornan; 2006.

Irby DM. Teaching and Learning in Ambulatory care settings: A thematic review of the literature. Acad Med 1995;70(10):898-931.

Liberman A, Liberman M, Steinert Y, McLeod P, Meterissian S. Surgery residents and attending surgeons have different perceptions of feedback. Med Teach 2005;27:470-472.

McIlwrick J, Nair B, Montgomery G. “How Am I doing?” Many problems but few solutions related to feedback delivery in undergraduate psychiatry education. Acad Psychiatr 2006;30:130-135.

Van de Ridder JMM, Stokking KM, McGaghie WC, Ten Cate OThJ. What is Feedback in Clinical Education? Med Educ 2008;42:189-197.

Schultz KW, Kirby J, Delva D, Godwin M, Verma S, Birtwhistle R, Knapper C, Seguin R. Medical students’ and residents’ preferred site characteristics and preceptor behaviours for learning in the ambulatory setting: a cross-sectional survey. BMC Med Educ 2004;4:12 doi:10.1186/1472-6920-4-12.

Kluger AN, DeNisi A. Effects of feedback intervention on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychol Bull 1996;119(2):254284.

Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, O’Brien MA, Johansen M, Grimshaw J, Oxman AD. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2012;6:CD000259. doi: 10.1002/14651858.

CD000259.pub3.

Veloski J, Boex JR, Grasberger MJ, Evans A, Wolfson DB. Systematic review of the literature on assessment, feedback and physicians’ clinical performance: BEME Guide No. 7. Med Teach 2006;28(2):117-128.

Boehler ML, Rogers DA, Schwind CJ, Mayforth R, Quin J, Williams RG, Dunnington G. An investigation of medical student reactions to feedback: a randomized controlled trial. Med Educ 2006;40:746-749.

Branch WT, Paranjape A. Feedback and reflection: teaching methods for the clinical setting.

Acad Med 2002;77(12):1185–1188.

Daelmans HEM, Hoogenboom RJI, Scherpbier AJJA, Stehouwer CD, van der Vleuten CPM. Effects of an in-training assessment programme on supervision of and feedback on competences in an undergraduate Internal Medicine clerkship. Med Teach 2005:27;158-163. van de Ridder JMM, Stokking KM, ten Cate ThJ. (2006, September) Relevant variables in feedback processes in clinical education. Paper presented at the Association for Medical Education in Europe 2006 Conference; September, Genoa Italy.

Dunegan KJ. Fines, frames, and images: examining formulation effects on punishment decisions. Organ Behav Hum Decis Process 1996;68:58-67.

Levin IP, Schneider SL, Gaeth GJ. All frames are not created equal: A typology and critical analysis of framing effects. Organ Behav Hum Decis Process 1998;76:149-188.17 Kilminster SM, Jolly BC. Effective supervision in clinical practice settings: a literature review.

Med Educ 2000;34:827-840.18 Salerno, S.M., O’Malley, P.G., Pangaro, L.N., Wheeler, G.A., Moores, K.D., & Jackson, J.L. (2002). Faculty development seminars based on the one-minute preceptor improve feedback in the ambulatory setting. Journal of General Internal Medicine, 17, 779-787.

Dunegan KJ. Framing, cognitive modes and image theory: toward an understanding of a glass half full. J Appl Psychol;1993:78,491-503.

Chapter 5

116

19. Kahneman D, Tversky A. Choices, values and frames. Am Psychol 1984;39:341-350.

20. Tanner C, Medin DL. Protected values: no ommision bias and no framing effects. Psychonomic Bull Rev 2004;11:185-191.

21. Schul Y, Ganzach Y. The effects of accessibility of standards and decision framing on product evaluations. J Consum Psychol 1995;4:61-83.

22. Rucker DD, Petty RE, Briñol P. What’s in a frame anyway? A meta-cognitive analysis of the impact of one versus two sided messages on attitude certainty. J Consum Psychol 2008;18:137149.

23. Dunegan KJ. Image theory: testing the role of image compatibility in progress decisions.

Organ Behav Hum Decis Process 1995;62:79-86.

24. Waung M, Jones DR. The effect of feedback packaging on ratee reactions. J Appl Soc Psychol 2005;35:1630-1655.

25. Kirkpatrick DL, Kirkpatrick JD. The four levels: an overview. In: Kirkpatrick DL, Kirkpatrick JD. Evaluating training programs. The four levels. San Fransisco: Berett-Koehler Publishers; 2006.

26. Morgan NA, Rego LL. The value of different customers satisfaction and loyalty metrics in predicting business performance. Market Sci 2006;25:426-439.

27. Jussim L. Social perception and social reality: A reflection-construction model. Psychol Rev 1991;98(1):54-73.

28. Nesbit PL, Burton S. Students justice perceptions following assignment feedback. Assess Eval High Educ 2006;31:655-670.

29. Geister S, Konradt U, Hertel G. Effects of process on motivation, satisfaction, and performance in virtual teams. Small Group Research 2006;37:459-489.

30. Bandura A. Self-Efficacy. In: Bandura A. Social foundations of thought and action. A social cognitive theory. Englewood Cliffs NJ; Prentice Hall: 1986.(p.390-453) 31. Swann WB, Griffin JJ, Predmore, SC, Gaines B. The cognitive-affective crossfire: when selfconsistency confronts self-enhancement. J Pers SocPsychol 1987;52:881-889.

32. Reynolds D. To what extent does performance-related feedback affect managers’ self-efficacy? Int J Hospit Manag 2006;25:54-68.

33. Baker DF. The development of collective efficacy in small task groups. Small Group Research 2001;32:451-474.

34. Nease AA, Mudgett BO, Quiñones MA. Relationships among feedback sign, self-efficacy, and acceptance of performance feedback. J Appl Psychol 1999;84:806-814.

35. Hattie J, Timperley H. The power of feedback. Rev Educ Res 2007;77(1):81-112.

36. Boatman DF, Miglioretti DL, Eberwein C, Alidoost M, Reich SG. How accurate are bedside hearing tests? Neurology 2007;68:1311 -1314.

37. Bagai A, Thavendiranathan P, Detsky, A.S. Does this patient have hearing impairment?

JAMA 2006;295(4):416-428.

38. Ear, Nose & Throat Examination - Medi-Vision Films 12. (2010). Retrieved January 5, 2013, from http://www.youtube.com/watch?v=dsS7d6_k1F8.

39. Kruisheer EM, van de Ridder JMM, Meeuwsen A. Evaluatie van knelpunten bij het vaardigheidonderwijs van het Otologisch onderzoek. [Students’ perception of difficulties in learning tests for detecting hearing loss.] (unpublished report).Utrecht; University Medical Center Utrecht: 2006.

40. Ten Cate, O. Medical Education in the Netherlands. Med Teach 2007;28:752-757.

41. Turner NB, van de Leemput AJ, Draaisma JMT, Oosterveld P, ten Cate TJ. Validity of the visual analogue scale as an instrument to assess self-efficacy in resuscitation skills. Med Educ 2008;42:503-511.

42. Moher, D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. Ann Intern Med 2009;151:264-269..

117 Framing of feedback impacts student’s satisfaction, self-efficacy and performance

43. Stevens J. Applied multivariate statistics for the social sciences. Mahwah NJ; Lawrence Erlbaum Associates Publishers: 1996.

44. Cohen J. Statistical Power Analysis for the Social Sciences. Mahwah NJ; Lawrence Erlbaum Publishers: 1988.

45. Downing SM. Reliability: on the reproducibility of assessment data. Med Educ 2004;38:10061012.

46. Field A. Discovering statistics. Using SPSS. London: Sage Publications; 2005.

47. Maynard DW. On “realization” in everyday life: the forecasting of bad news as a social relation. Am Socio Rev 1996;61:109-131.

48. Sitzman T, Brown KG, Casper WJ, Ely K, Zimmerman RD. A review and meta-analysis of the nomological network of trainee reactions. J Appl Psychol 2008;93:280-295.

Feedback providers’ credibility impacts students’ satisfaction with feedback and delayed performance

Published as: Van de Ridder JMM, Berk FCJ, Stokking KM, Ten Cate OthJ. Feedback providers’ credibility impacts students’ satisfaction with feedback and delayed performance. Med Teach 2014: Early online pre-publication

Chapter 6

120

Abstract Purpose Medical students receive feedback during clerkships from many different sources: attendings, residents, paramedics, other clerks and even patients. Not all feedback providers have similar impact on learning. One characteristic that is believed to have impact is their credibility to the recipient. This study investigates the effects of feedback provider credibility on medical student satisfaction, self-efficacy, and performance with a trained skill.

Method A single blind randomised controlled between-subjects design was used, with feedback provider credibility (high-low) as independent variable and examination of hearing abilities as the task. First year medical studentsâ&#x20AC;&#x2122; (n = 68) satisfaction, self-efficacy, and performance were the dependent variables and were measured both directly after the intervention and after a three week delay.

Results Credibility did not significantly affect immediate or delayed self-efficacy. Students receiving feedback from a high-credibility source were more satisfied with the feedback. They did not perform significantly better immediately after the feedback intervention, but did so three weeks after the intervention. High credibility was associated with a perception of a negative feedback message and an unsocial feedback provider.

Conclusions Feedback provider credibility impacts satisfaction with feedback and delayed performance. If feedback is not effective in clinical settings, feedback providers may reconsider their credibility.

6.1 Introduction

121 Feedback providers’ credibility impacts students’ satisfaction with feedback and delayed performance

edical students receive formal and informal feedback during clerkships from a variety of sources: attendings, residents, nurses, paramedics, other clerks, and even patients. If done well, feedback providers observe students’ task performances, to compare this with implicit or explicit standards, and based on this comparison give feedback often with the intention to improve skills or behavior.1 Feedback providers influence the degree to which the students accept feedback,2-8 their decision to ask for feedback,9 and their strategy to seek feedback.10 Not everyone is perceived as a credible feedback provider. Murdoch-Eaton and Sargeant found that students’ perceptions of credible providers change over time.11 Young students value feedback chiefly from senior staff while more mature students also see their peers as credible feedback providers. Sargeant et al. show that physicians find feedback from other physicians not always credible particularly if these colleagues have limited opportunity to observe their practice and to make informed assessments.6 There is no shared operational definition of feedback provider credibility, and the dimensions of the credibility concept have been labeled differently over time.12-13 Older age, male gender, substantial experience, professional background, high status, and friendly attitude (sociability), have been considered features that enhance feedback provider credibility.2, 13-18 Source credibility is both viewed as a unidimensional19-21 and a multidimensional construct.12-13, 15, 22 Dimensions that have been used to operationalize credibility include expertise, experience, trustworthiness accuracy, believability or reliability, intentions, dynamism, personal attractivity, fairness, convincingness, informativeness, age, reward power, accessibility, gender, status, caring, composure, sociability and extraversion.2-3, 9-10, 12-18, 20, 22-23 Much social science research on credibility has been done using experiments where participants judge paper scenarios containing only written information given by a (feedback) source.9, 17, 24-25 Outcome measures are often the subject’s perception or opinion about the source or the source’s message.14, 20-21, 23, 26 Participants’ intended behaviors are also used as outcome variables.19, 27-28 Psycho-motor behaviors, relevant variables in the medical context, are rarely mentioned as outcome measures. In addition, outcomes are usually measured immediately after the intervention and not over the long-run.2, 19, 26-29 These experimental studies show that feedback sources having high credibility receive higher ratings on characteristics like trust and satisfaction compared to feedback providers with low credibility.17, 20 This finding conforms with the outcomes from the metaanalyis of Finn and colleagues.13 Concerning feedback content, the message from a credible source is more often judged as accurate,14, 28 favorable,20 and helpful.14 In general, positive messages are better accepted.2 Source credibility has less influence on acceptance of a negative feedback message.21 Several studies show an association between message content and source credibility.2, 19-20, 23, 25, 27 After receiving a negative message from a high-credibility source, recipients are less likely to attribute the criticism to negative dispositions of the feedback provider, compared to when the provider has low credibility.17 Several studies on feedback recipients’ satisfaction, intentions, self-efficacy and behav-

Chapter 6

122

ior, show that recipients are more satisfied with high-credibility feedback.14, 30 However, Fedor et al. did not replicate these findings.25 In classroom studies feedback recipients are more satisfied with and have greater intentions to use feedback from a high credible source than from a low one.14, 25, 28, 30 Bloom and Hautaluoma however did not find any influence of credibility; neither on affective reactions towards the feedback message nor on recipients’ intentions to improve.23 Podsakoff and Farh found that recipients’ self-efficacy was not influenced by source credibility.27 Feedback from a high-credibility source appears to improve performance, especially when the message content is negative.12 Negative feedback messages from a high-credibility source result in more performance improvement than negative messages from a low-credibility source.27 High-credibility communicators induce more opinion change, compared with mildly credible communicators.19, 24 Finn et al. showed in her metaanalysis that teacher credibility positively affected student outcomes such as motivation, learning and student’s communication with teachers.13 In short, social science research outcomes indicate that feedback from a high-credibility source is perceived as accurate and messages are better accepted especially when the message is positive. Feedback from a high-credibility source is likely to alter performance. There is also an interaction between message value (positive - negative) and credibility (high - low). Findings are ambiguous about the influence of credibility on feedback recipient’s satisfaction, self-efficacy, and intentions to change.

Feedback provider credibility is also reported to be important in medical training.5, 7-8, 15, 31 Sargeant et al. describes source credibility as the perceived ability to observe and assess well.31 Bing-You et al. interviewed 12 residents, and found that feedback provider’s characteristics, resident’s observation of feedback provider’s behavior, the content of their feedback, and their method of delivering feedback all play a role are perceived as characteristics of credibility.15 In two qualitative studies Watling et al. identified credibility as a key factor in determining whether learning cues, such as feedback, prove influential.7-8 We did not find experimental studies that investigating the effect of feedback provider credibility as a mediator of the effect of feedback given in a medical education context.

Given the uncertainty of credibility effects of the feedback provider on several relevant outcome measures, our question in this study was: How does the feedback providers’ credibility influence students satisfaction with feedback, their self-efficacy, and their subsequent quality of performance, and more specifically, is feedback from a more credible feedback provider more effective than feedback from a less credible feedback provider?

6.2 Method Design A single blind randomised controlled between-subjects design was used with feedback provider’s credibility (high-low) as the independent variable. This design and the outcome variables were used in a previous study. The choice of the dependent variables was based upon Kirkpatrick’s model of evaluation.32 On the ‘reaction’ level we measured student satisfaction with the feedback (SAT) and self-efficacy regarding their performance of the tuning fork test (SE). On the ‘learning’ level we measured student’s task performances on the tuning W&R test (P).

Credibility In both conditions students received a negative feedback message consisting of two points for improvement in both conditions. In the high-credibility condition the feedback provider was a 49-year old male actor, trained to act as a professor of Ear, Nose and Throat surgery, wearing a white coat, with a visible identification tag. He was instructed to refer to his clinical practice experience while introducing himself to the student.16-17 In the low-credibility condition, the feedback provider was a 20-year old female third year medical student, i.e. two years ahead in medical school compared to the subjects, employed as a teaching assistant, informally dressed without a white coat.

The actor and the teaching assistant received a one-hour training in which the aim of the study and the logistics were explained and trained. In this training the protocol with similar standardized feedback provision method for both conditions was also practiced.

Participants All first year medical students from the University Medical Center Utrecht (UMCU) were invited to voluntarily participate. Students were asked to send an email if they did not choose to participate. From the 313 students we approached, 35 declined for several reasons. From the remaining group, 68 students were assigned to the credibility experiment.

All subjects were informed about one week before the experiment during a thirty-minute plenary meeting, a letter, and several emails. The voluntary nature of their participation was stressed and also that neither being in the study nor scores received would have any effect on exams or grades. A 7.50 Euro reward was promised after completing the full experiment. We explained that the data would be only accessible by researchers and would be stored under code in UMCU. Names and student numbers were removed after the data analyses were concluded. The true nature of the experimental conditions could not be disclosed because this would interfere with the aim of the investigation.

All participants were asked to sign an informed consent form. Subjects were provided with addresses and telephone numbers to ask for additional information if desired.

During the experiment one investigator was in the examining room to make sure the process conformed with the protocol.

Procedure This experiment was conducted parallel to other studies related to feedback provision.

Other students from a large pool of volunteers were asked to participate in a different study. The current study was carried out in the UMCU skills lab, during a regular skills course on physical examination of joints, not related to the W&R task of our study.

123 Feedback providersâ&#x20AC;&#x2122; credibility impacts studentsâ&#x20AC;&#x2122; satisfaction with feedback and delayed performance

Task The task for students was to practice the Weber and Rinne tuning fork procedures (W&R task) to test hearing deficiencies on Standardized Patients (SPs) according to guidelines provided by an otorhinolaryngologist in an instructional video. This test is used in the clinical setting in both the otolaryngology and neurology.33-34 For a detailed description see a short instructional video.35 This task is suitable for an experiment because it is not time consuming, standardized instruction can be given with a videotape, the pathology is easy to simulate by SPs, it has obvious relevance for medical training, and it is new for first year medical students.

Chapter 6

124

Figure 6.1 shows the flow chart of randomisation and data collection according to the CONSORT convention.36 Students were randomised over the two conditions by drawing lots (at time T0). Next, students were asked to rate their self-efficacy in performing the W&R task (SE1). Then, the student watched an instructional video about the W&R task procedure (T1). Subsequently, each student carried out the W&R task (P1) on a SP while being observed by the feedback provider for the specific condition. Directly after this, the student would rate his or her self-efficacy again (SE2). Next, negative feedback consisting of two points for improvement was provided for each condition (T2). The student performed the W&R task a second time on the same SP (P2), and rated selfefficacy a third time (SE3) (T3). Three weeks later the students were asked to rate their self-efficacy for the fourth time (SE4), and to perform the W&R task for the third time (P3). All performances were video recorded.

At the first opportunity that students were in the skills lab, two weeks after finishing the experiment, they received an additional questionnaire to control for other variables: how the feedback was perceived (positive or negative) and how they perceived the feedback provider in general and how they rated the sociability of the feedback provider.13 Students were asked to explain their answers.

The experiment was carried out between December 17th 2007 and January 11th 2008. A debriefing session was organized to explain the experimental aim after the study ended.

Figure 6.1. Flow-chart of the chronological procedure of the trial and the amount of participants who participated in each stage and were included in the analyses

Data analysis When assumptions of normality, linearity, and/or homogeneity of variances, were violated, a Mann-Whitney U test was performed, otherwise a linear mixed model analysis or t-test was used to compare credibility effects on Satisfaction (SAT), Performance (P1, P2, P3), and Self-efficacy (SE1, SE2, SE3 and SE4) between experimental and control group. ‘Post hoc’ the differences in students’ perceptions of feedback content and feedback providers’ sociability were tested with Chi-square.

A balance between type I and type II error was sought by setting α = 0.10, because of the relatively small group. Applying a Bonferroni correction for multiple statistical testing on the outcome variables, this resulted in a significance level of (0.10/3=0.03) α = 0.03.38 Effect sizes (ES) were reported using partial η2 and r. Small, medium, and large effect sizes are respectively 0.01, 0.06, and 0.14 for partial η2 (for those unfamiliar with partial η2 this is equal with d 0.2, 0.5. and 0.8) and 0.1, 0.3, and 0.5 for r.39 6.3 Results Participants Fifty-one students (75%) had a complete dataset without extreme outliers. Incompletion was caused by partial missing data on performance and VAS-scores (n = 4), a broken video camera (n = 8), not appearing at the follow-up due to illness (n = 1) or unknown reasons (n = 4). The students’ mean age was 18,9 (SD = 1.3), and 69 % were women. No significant difference in age and sex distribution between conditions was found.

Quality of the instruments The satisfaction scale showed an internal consistency of Cronbach’s α = 0.84.

The VAS for measuring self-efficacy is a one item instrument and cannot yield reliability estimates. Five independent raters used the fourteen-item checklist to score the video recordings of the students’ performance. The average inter-rater reliability on the total score was high (ICC = 0.92).40 The questions on perceived source sociability and perception of the feedback message were two one-item questions. In an open answer format students could elaborate on their judgment.

125 Feedback providers’ credibility impacts students’ satisfaction with feedback and delayed performance

Instruments Student satisfaction with the video instruction, training opportunities, and the received feedback was measured using a five item Likert scale (1 = totally disagree, 5 = totally agree).

A visual analogue scale (VAS) was used to measure student self-efficacy on detecting hearing loss by using the tuning fork tests (0 mm = ‘I am extremely unconfident’, 100 mm = ‘I am extremely confident’). This procedure for measuring self-efficacy was validated by Turner et al. for medical skills.37 Videotapes of students’ performances were rated by five independent raters using a 14item observational checklist. The total score was used as a performance measure (0 = minimum, 18 = maximum). The W&R observational checklist was validated among medical students (others than those who participated in this study) and in close collaboration with an otorhinolaryngologist. The checklist was adjusted after piloting and discussions with the raters.

Assumptions The data in the low-credibility condition on SAT was not normally distributed therefore we performed a Mann-Whitney U test.

Chapter 6

126

Effect of feedback provider credibility on satisfaction A Mann-Whitney U test revealed a significant difference in satisfaction between conditions U = 396, Z = -2.23, p < 0.026, r = -.27. Students in the low-credibility feedback condition (Mdn = 3.30) were more satisfied with the feedback than students in the high-credibility condition (Mdn = 2.80).

Effect of feedback provider credibility on self-efficacy The self-efficacy scores increased after the first task performance (T2). They further increased in the low-credibility condition and decreased in the high-credibility condition (T3), and at T4 the scores decreased for both conditions to become lower than the initial baseline score (figure 6.2). A linear mixed model analysis showed only a main Table 6.1. Means (M), Standard Deviations (SD) and correlations according to Pearson product moment correlation coefficient for self-efficacy1 (SE1), performance1 (P1), self-efficacy2 (SE2), self-efficacy3 (SE3), satisfaction (SAT), self-efficacy4 (SE4), and performance3 (P3). For performance2 (P2) Spearman correlation coefficients are reported Variable N M SD SE1 SE1 67 37.3 19.5 P1 65 9.4 2.0 -0.10 SE2 68 53.4 15.6 0.18 P2 66 11.1 2.6 -0.24 SE3 67 53.9 19.1 0.02 SAT 68 3.1 0.7 -0.12 SE4 63 29.3 20.8 0.15 P3 55 9.7 3.2 -0.06 ** ) Correlation is significant at 0.01 level (2-tailed) * ) Correlation is significant at 0.05 level (2-tailed)

P1

SE2

P2

SE3

SAT

SE4

P3

0.15 0.52** 0.29* 0.27* -0.11 0.14

0.14 0.72** 0.45** 0.32* -0.09

0.21 0.28* 0.02 0.36**

0.68** 0.31* -0.06

0.40** 0.11

0.25

-

Table 6.2. Means (M), Standard Deviations (SD) and 90% confidence intervals (90% CI) of satisfaction, self-efficacy, and performance outcomes by feedback provider’s credibility condition at the four time points (T1-T4) Satisfaction Self-efficacy Performance N Mdn 90% CI N M (SD) 90% CI N M (SD) 90% CI High credibility T1 Baseline 34 39.1 (18.4) 30.2-42.6 T2 Pre-test 35 51.5 (15.4) 45.8-56.0 33 9.1 (2.3) 8.0-9.8 33 10.8 (3.0) 9.8-12.1 T3 Post-test 35 2.9† 2.7-3.1 34 50.4 (19.8) 41.1-56.3 9.6-12.1 T4 Three weeks 32 30.2 (19.1) 25.9-39.4 25 10.9 (3.5)† Low credibility T1 Baseline T2 Pre-test T3 Post-test T4 Three weeks

33

3.3†

3.1-3.5

33 33 33 31

35.4 (20.6) 55.4 (15.9) 57.5 (17.9) 28.4 (22.7)

31.3-44.1 51.3-61.7 52.1-63.8 21.4-35.9

32 33 30

9.6 (1.7) 11.4 (2.2) 8.6 (2.7)†

†) These values significantly differ in the high and low credibility feedback condition (p<0.03)

9.2-10.2 10.7-12.2 7.9-9.6

effect of time on self-efficacy F (3,65) = 37.5, p < 0.001. The effect of credibility and the interaction between time and credibility was not significant.

Information questionnaire There was a significant association between the feedback condition and the perception of the feedback content χ2(1) = 8.78, p = 0.003. In the high-credibility condition 81% of the students perceived the feedback content as being negative, compared to 36% in the low-credibility condition. In the open comments students mentioned: “Only my negative points were mentioned.” “He did not say what I had to change.”

There was also a significant association between the feedback condition and the students’ perception of the feedback provider’s sociability (χ2(2) = 15.76, p = 0.00). In the high-credibility condition students perceived the feedback provider as having a neutral (24%) or unsocial (76%) attitude. Students perceived the teaching assistant in the lowcredibility condition as having a social (12%), neutral (69%), or unsocial (19%) attitude.

The correlation between students’ perception of the message and their perception of sources’ sociability was high (r = 0.67, p <0.01).

6.4 Discussion This study investigates the effect of feedback provider credibility - operationalized as high status, older age, substantial content experience, and male gender - on feedback effects in student satisfaction, self-efficacy, and performance, by using a single blind randomised controlled between-subject design.

In the high-credibility condition students were more satisfied and, their performance score was stable after receiving feedback. They performed significantly better after three weeks than students receiving feedback from the low-credibility source, with effect size medium to high. As expected, in both conditions self-efficacy decreased over time. In line with others, we did not find differences on student’s self-efficacy between conditions.27 Students apparently feel less confident in performing the W&R task after three weeks without repetition of the skill.

How can these findings be explained? From the questionnaire we learned that most students from the low-credibility condition perceived the source as neutral and the message as positive, while most students from the high-credibility conditions perceived the source as unsocial and the message as negative. This might have influenced the outcomes on two grounds.

127 Feedback providers’ credibility impacts students’ satisfaction with feedback and delayed performance

Effect of feedback provider credibility on performance The performance scores increased after the first task performance (T3). They decreased in the low-credibility condition and they were stable in the high-credibility condition at T4 (figure 6.3). A linear mixed model was used to evaluate the main effects of time and credibility, and the interaction effect between time and credibility on the improvement of performance. We found an effect of time on performance F (2,62) = 19.7, p < 0.001. An interaction effect between time and credibility F (2,62) = 6.3, p < 0.003 was found. Consequently, we tested the differences between the groups at the specific time points. After three weeks (T4) the group in the high-credibility condition had a significantly higher performance score than the group in the low-credibility condition t (53) = 2.81, p < 0.007, ES partial η2 = 0.13. We did not find significant differences at T2 and T3.

Chapter 6

128

1) Responses towards feedback may have been more affected by the perceived content of the feedback message than the credibility or expertise of the provider.19, 23, 27 A negative message from a high-credibility source may have more impact than a negative message from a low-credibility source.2, 27 The inclination to change behavior as a result of feedback from a low credible source will probably will not be very strong. Likewise, positive feedback statements from such sources may also not increase performance.23 We did not investigate this, but it could be done in future studies.

2) A second explanation for our findings is the discrepancy between the message and the recipients’ initial beliefs. Past research suggests that when the discrepancy between students’ beliefs about their performance and the given feedback message is large, subjects are only influenced by high-status sources.2, 24, 27 The discrepancy between the message and recipients’ initial beliefs can also affect students’ perception of the feedback provider. Albrigth et al. found that when there is a positive discrepancy (the message is more positive than was expected) the feedback provider is perceived as more Figure self-­‐efficacy positive.20 We did not6.2 collect data on students’ own beliefs about their performance,

High credibility

65

Low credibility

60 55 50 45 40 35 30 25 20

T1

T2

T3

T4

Figure 6.2. Self-efficacy by condition over time Figure 6.3 measure performance

12

High credibility

11.5

Low credibility

11 10.5 10 9.5 9 8.5 8 7.5 T2

T3

Figure 6.3. Performance measure by condition over time

T4

Conclusion and implications A credible feedback provider does not guarantee more effective feedback. But our study does show that receiving feedback from a credible source may enhance performance after a short delay, even when students find it less satisfactory. It also illustrates the value of using follow-up measurements because some effects are only visible over time.

Further qualitative research could address students’ perceptions of feedback providers’ characteristics related to feedback acceptance and feedback effect in an authentic clinical context. It is also important to investigate the effect of positive and negative feedback content and students’ own expectations of the feedback message in relation to source credibility in the clinical setting.

A practical lesson is that credible feedback providers have a learning effect even when the feedback is appraised retrospectively as negative. If feedback is not effective in the clinical setting, feedback providers may realize that their credibility could be an explanation for this effect.7

129 Feedback providers’ credibility impacts students’ satisfaction with feedback and delayed performance

hence we do not know if these affect study outcomes. This could be studied in further research.

This study has strengths and limitations. First, we used a randomised experiment where subjects received face-to-face feedback from real persons. Second, feedback was provided on students’ (psycho-motor) performance, close to what happens in the clinical setting. Although it was still an artificial situation, this supports the ecological validity of this study compared to studies in which paper scenarios are used to provide feedback and only intended reactions and perceptions are used as outcome measures.

A limitation is that credibility as perceived by the subjects was not measured, but only operationalized on criteria from the literature. Theoretically our intended casting may not have come across. This is an inherent limitation, as we did not want to prompt the subjects’ opinion about it as that may have contaminated the experimental effect. Asking their opinion afterwards would have left unclear whether the feedback message would have affected their opinion, as had been found in the Albright et al. study cited above.20 Also, the whole experiment may have been viewed as a role play, but we did not receive any clue from students that they had noticed that the training actor was playing a role.

It is conceivable that the actor in the high credibility condition, not being medically trained, may have been more unsure about providing feedback while the teaching assistant in the low credibility condition knew what she was talking about, as she was a medical student. Theoretically, insecurity may have compromised the credibility of the actor but again, we have no indication that this showed in any way while he was playing the role. Using a real subject-matter expert as feedback provider could however be considered in a replication of this experiment.

References 1.

2.

3.

Chapter 6

130

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.

23.

24.

van de Ridder JMM, Stokking KM, McGaghie WC, ten Cate ThJ. What is feedback in clinical education? Med Educ 2008;42:189-197.

Halperin K, Snyder CR, Shenkel RJ, Houston BK. Effects of source status and message favorability on acceptance of personality feedback. J Appl Psych 1976;61:85-88.

Ilgen DR, Fisher CD, Taylor SM. Consequences of individual feedback on behavior in organizations. J Appl Psych 1979;64:349-371.

Sargeant J, Mann K, Ferrier S. Exploring family physicians’ reactions to multisource feedback: perceptions of credibility and usefulness. Med Educ 2005;39:497-504.

Veloski J, Boex JR, Grasberger MJ, Evans A, Wolfson DB. Systematic review of the literature on assessment, feedback and physicians’ clinical performance: BEME Guide No. 7. Med Teach 2006;28:117-128.

Sargeant J, MacLeod T, Sinclair D, Power M. How do physicians assess their family physician colleagues’ performance? Creating a rubric to inform assessment and feedback. J Contin Educ Health 2011;31(2):87–94.

Watling C, Driessen E, van der Vleuten CPM, Lingard L. Learning from clinical work: the roles of learning cues and credibility judgements. Med Educ 2012a;46:192–200.

Watling C, Driessen E, van der Vleuten CPM, Vanstone M, Lingard L. Understanding responses to feedback: the potential and limitations of regulatory focus theory. Med Educ 2012b;46:593–603.

Vancouver JB, Morrison EW. Feedback inquiry: the effect of source attributes and individual differences. Organ Behav Hum Decis Process 1995;62:276-285.

Lee HE, Park HS, Lee TS, Lee DW. Relationship between LMX and subordinates’ feedbackseeking behaviors. Soc Behav Pers 2007;35:659-674.

Murdoch-Eaton D, Sargeant S. Maturational differences in undergraduate medical students’ perceptions about feedback. Med Educ 2012;46:711–721.

Pornpitakpan C. The persuasiveness of source credibility: a critical review of five decades’ evidence. J Appl Psychol 2004;34:243-281.

Finn AN, Paul Schrodt P, Witt PL, Elledge N, Jernberg KA, Larson LM. A meta-analytical review of teacher credibility and its associations with teacher behaviors and student outcomes. Commun Educ 2009;60(1):75-94.

Bannister BD. Performance outcome feedback and attributional feedback: interactive effects on recipient responses. J Appl Psych 1986;71:203-210.

Bing-You RG, Paterson J, Levine MA. Feedback falling on deaf ears: resident’s receptivity to feedback tempered by sender credibility. Med Teach 1997;19:40-44.

Solomon GB, DiMarco AM, Ohlson CJ, Reece SD. Expectation and coaching experience: is more better? J Sport Behav 1998;21:444-456.

Leung K, Su S, Morris MW. When is criticism not constructive? The roles of fairness perceptions and dispositional attributions in employee acceptance of critical supervisory feedback.

Hum Relat 200;54:1155-1187.

LaPlante D, Ambady N. Saying it like it isn’t: Mixed messages form men and women in the workplace. J Appl Psychol 2002;32:2435-2457.

Orpen C, King G. Effects of superiors’ feedback, credibility, and expertise on subordinates’reactions: an experimental study. Psychol Rep 1989;64:645-646.

Albright MD, Levy PE. The effects of source credibility and performance rating discrepancy on reactions to multiple raters. J Appl Soc Psychol 1995;25:577-600.

Hurley AE. The effects of self-esteem and source credibility on self-denying prophecies.

J Psychol 1997;131:581-594.

Giffin K. The contribution of studies of source credibility to a theory of interpersonal trust in the communication process. Psychol Bull 1967;68:104-120.

Bloom AJ, Hautaluoma JE. Effects of message valence communicator credibility, and source anonymity on reactions to peer feedback. J Soc Psychol 1987;127:329-338.

Aronson E, Turner JA, Carlsmith JM. Communicator credibility and communication discrepancy as determinants of opinion change. J Abnorm Soc Psychol 1963;67:31-36.

131 Feedback providersâ&#x20AC;&#x2122; credibility impacts studentsâ&#x20AC;&#x2122; satisfaction with feedback and delayed performance

25. Fedor DB, Eder RW, Buckley RM. The contributory effects of supervisor intentions on subordinate feedback responses. Organ Behav Hum Decis Process 1989;44:369-414.

26. Alder GS, Ambrose ML. An examination of the effect of computerized performance, monitoring feedback on monitoring fairness, performance and satisfaction. Organ Behav Hum Decis Process 2005;97:161-177.

27. Podsakoff PM, Farh JL. Effects of feedback sign and credibility on goal setting and task performance. Organ Behav Hum Decis Process 1989;44:45-67.

28. Roberson QM, Stewart MM. Understanding the motivational effects of procedural and informational justice in feedback processes. Br J Psychol 2006;97:281-298.

29. Bochner S, Insko CA. Communicator discrepancy, source credibility, and opinion change.

Pers Soc Psychol Bull 1966;4:614-621.

30. Steelman LA, Rutkowsky KA. Moderators of employee reaction to negative feedback. J Manage Psychol 2004;19:6-18.

31. Sargeant J, Mann K, Sinclair D, van der Vleuten C, Metsemakers J. Challenges in multisource feedback: intended and unintended outcomes. Med Educ 2007;41:583-591.

32. Kirkpatrick DL, Kirkpatrick JD. 2006. The four levels: an overview. In: Kirkpatrick DL, Kirkpatrick JD. Evaluating training programs. The four levels. San Fransisco: Berett-Koehler Publishers, pp. 21-65.

33. Bagai A, Thavendiranathan P, Detsky, A.S. Does this patient have hearing impairment?

JAMA 2006;295(4):416-428.

34. Boatman DF, Miglioretti DL, Eberwein C, Alidoost M, Reich SG. How accurate are bedside hearing tests? Neurology 2007;68:1311 -1314.

35. Ear, Nose & Throat Examination - Medi-Vision Films 12. 2010. Retrieved January 5, 2013. http://www.youtube.com/watch?v=dsS7d6_k1F8.

36. Moher D, Schulz KF, Altman D. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. JAMA 2001;285:19871991.

37. Turner NB, van de Leemput AJ, Draaisma JMT, Oosterveld P, ten Cate TJ. Validity of the visual analogue scale as an instrument to assess self-efficacy in resuscitation skills. Med Educ 2008;42:503-511.

38. Stevens J. Applied multivariate statistics for the social sciences. Mahwah NJ, Lawrence Erlbaum Associates Publishers 1996.

39. Cohen J. Statistical Power Analysis for the Social Sciences. Mahwah NJ, Lawrence Erlbaum Publishers 1988.

40. Downing SM. Reliability: on the reproducibility of assessment data. Med Educ 2004;38:10061012.

Conclusion and discussion

Chapter 7

134

7.1 Summary of research A chronological feedback model is used in this thesis to represent the feedback process. This model reflects elements important in communication, such as the message, sender and recipient. It also reflects psychological aspects which occur during interactions between people.

In this feedback model the differences between feedback, the feedback process, and the feedback effect are visualized and clearly distinguished from each other. When we use the word feedback we refer to the message which is transferred from the feedback provider towards the feedback recipient. We speak about the feedback process when we refer to the process in which task performance, observation, feedback providing and feedback receiving, phases A, B, C, and D in the model are embedded. When we refer to the difference between feedback recipient’s first and the second performance (A2 – A1) we use the term feedback effect. The word ‘effect’ does not imply that the effect is positive but that it has an impact, in contrast to non-effective feedback where there is no detectable difference between the two performances.

In phase A1 the feedback recipient performs a task which has certain standards. In phase B the feedback provider observes the task from phase A1 and interprets task performance by comparing it to implicit or explicit standards. In phase C the feedback provider communicates the feedback consisting in the difference between the feedback recipient’s task performance and the standard. In phase D the feedback recipient receives and interprets the feedback, and in phase A2 the task performance is repeated.

This model is used as a tool to structure the outcomes of the studies in this thesis, especially of chapters 3 and 4.

The research question ‘Which variables influence the feedback process and the feedback effect?’ was studied in chapter 2, 3 and 4. In chapter 2 we focused on the term feedback.

135 Conclusion and discussion

lthough feedback has a long research tradition, the topic has been researched in different fields, and the research topics are very diverse. This means that comparatively little is known about what influences the feedback process and the feedback effect.

The purpose of this thesis was to integrate feedback literature from all the various paradigms and fields, and to explore some of the variables that affect the feedback process and the feedback effect. The central research questions in this thesis were: 1) Which variables influence the feedback process and the feedback effect according to both the published literature and the perceptions of medical students and residents? 2) How do these variables related to the communication of feedback and how do the characteristics of the feedback provider affect a medical student’s perception and performance while performing a physical examination task?

These questions were answered by performing two literature studies (chapter 2 and 3), a survey (chapter 4), and two experimental studies (chapter 5 and 6).

This final chapter will first summarize the research findings, evaluate the goals of this thesis, discuss findings, theories, strengths and limitations, and give suggestions for future research. Finally, we close with some practical implications for feedback research and guidance for giving and receiving feedback in daily practice.

Chapter 7

136

Figure 7.1. Feedback model which represents the feedback, the feedback process and the feedback effect

In chapter 3 we focused on the question of why feedback is not always as effective as expected, and in chapter 4 we described the variables that influence the feedback process and the feedback effect according to medical students’ and residents’ perceptions.

The purpose of chapter 2 was to clarify the feedback concept. Therefore we focused on the conceptual and operational definition of feedback. Feedback definitions were searched in dictionaries, encyclopedias, lexicons and handbooks pertaining to different scientific fields. Furthermore, we searched in the Education Resources Information Center (ERIC), PsycINFO and MEDLINE databases for articles in which feedback was a defining theme. We identified three concepts which are used in feedback definitions: information, reaction, and cycle. Feedback as information represents the content of the feedback message. Feedback as a reaction embodies the interaction between the feedback provider and the feedback recipient with the information. In the cycle concept, the focus is on receiving information, responding to the information, and improving the response quality. In both social sciences and medical education literature the information concept is mostly used, followed by the reaction concept. Few articles use the cycle concept. We chose to use the information concept in this thesis, which is in line with most of the current feedback literature.

A comparison of feedback definitions gives insight into the common characteristics of the various feedback definitions. These characteristics relate to the sender of the message (feedback provider, source, preparation), to the receiver of the message (feedback recipient), to the message itself (content, aim, form), and the environment in which the communication takes place (communication conditions and contextual factors).

Based on our analyses we proposed a research based, operational definition of feedback: ‘Specific information about the comparison between a trainee’s observed performance and a standard, given with the intent to improve the trainee’s subsequent performance’. Key elements in this definition are: The feedback providers give feedback with the intention to improve the learner, and only after observation. The feedback recipients are trainees. The feedback message is about performances and tasks. It de-

137 Conclusion and discussion

scribes a comparison between a performance and the standard. The standards are mentioned, the information is specific, and it is given with the aim to improve the learner.

The context considered within this study is clinical education.

This definition was evaluated taking into account the literature about concepts and operational definitions.1-5 To give feedback according to the definition requires that the following steps should be taken: a) to gather information the trainee’s performance should be observed; b) the content of the message should be about an observed performance and a standard; c) the direction of the provided information is from the clinical teacher towards the trainee; d) and the intention of providing the information is to improve trainee’s performance. These actions are considered to be measurable, reproducible, and clear.2 In chapter 3 we explained why feedback is not always as effective as we expected. The literature does not provide consensus about the impact of feedback. Some reviews and meta-analyses report small to moderate impact6-7 while others report large feedback effects.8 Furthermore, the effect of feedback can be positive and as a consequence the performance improves, but it can also be negative resulting in a decrease in performance. In this chapter we brought feedback research from various disciplines together and two questions were answered: a) Which variables influence the feedback process and the feedback effect according to reviews and meta-analyses? b) In which direction do these variables influence the feedback process and the feedback effect? The study was prepared using the reporting conventions described in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement. A literature search was performed in the database ERIC, PsycINFO and MEDLINE. This search was limited to meta-analyses and literature reviews published in peer-reviewed journals over the period from January 1986- February 2012. The databases yielded 1101 publications. Based on exclusion criteria we narrowed this down to 203 publications. A total of 46 articles (22 meta-analyses and 24 literature reviews) met the inclusion criteria.

We identified a total of 33 independent variables that affected either the feedback process, the feedback effect, or both. We noticed that all four phases of the feedback process (A, B,C and D) and also the feedback effect were influenced by the independent variables. Amongst the variables (n = 3) that influence tasks, standards and first task performance (phase A) are the feedback recipient’s skills and also the subject-matter of the task. The variables (n = 16) that influence the feedback provider’s observation and the interpretation of feedback recipient’s performance (phase B) include task complexity and also the feedback recipient’s cultural background. Independent variables (n = 3) influencing the feedback provider’s communication of the feedback message (phase C) are training content and method, and rubrics used. Culture and context and the content of the feedback are the variables (n = 2) influencing reception and the interpretation of the feedback message (phase D). In total we found 16 variables influencing the feedback effect.

The reviews and meta-analyses that identified the 33 independent variables did not always give a clear, unambiguous indication of how they influenced the feedback process or feedback effect. Due to the many and diverse interactions with other variables, it seemed in some studies they had a positive effect on an outcome variable and in other studies they gave ambiguous results.

However, from six variables that influenced the feedback provider’s observation, interpretation and rating (phase B), the literature was unequivocal. Rating high complexity

Chapter 7

138

tasks results in low inter-rater agreement, feedback providers with high task familiarity have more agreement on ratings, training feedback providers in using observation instruments reduces rating error, and the use of rubrics increases the reliability of scoring and facilitates the feedback communication. Furthermore, when feedback providers and feedback recipients have a similar cultural background this results in higher performance ratings, and when feedback providers have more time to build a relationship with the feedback recipient, the higher the correlations are between a subjective and an objective performance measure. We also identified some variables that influenced the feedback effect in only one direction. When feedback recipients have low initial skill level this results in large feedback effects. The feedback recipients’ self-esteem influences the feedback effect and when feedback recipients set themselves goals this increases the feedback effect too. When feedback is part of a multi-facetted intervention (form) this also increases the feedback effect. In addition, when the feedback content is encouraging, elaborate and specific the feedback effect increases, and frequent feedback enhances the feedback effect too.

The other 21 variables influencing the feedback process and the feedback effect do not always provide a consistent impact. One example is the ‘timing of feedback’. Timing is an important variable, but when feedback should optimally be provided depends on the nature of the task. Also the feedback provider influences the feedback effect, but it is not clear in which way: the effect depends on the level of feedback provider’s expertise and also on the interaction between the feedback provider and the feedback recipient.

Regarding the framework, our study shows that a) variables from each phase of the feedback process influence the next phase in the feedback process, and b) all four phases in the feedback process influence the feedback effect. The consequence for clinical practice is that we should not only focus on how well the feedback provider communicates the feedback (phase C), but also on the performed task (phase A), and the observation and rating (phase B), and FR’s reception and the perception (phase D).

In conclusion we can say that it is questionable if we should look at feedback processes from an abstract point of view. It is much more informative to focus on the feedback process and feedback effect in its context. Focusing on specific feedback situations with specific feedback providers and feedback recipients and specific tasks and specific context will give us much more useful information.

Nevertheless, this study gives us a bigger picture and a clear understanding of why the effect of feedback can be small or even negative. In all four phases of the feedback process different variables influence the outcomes. These phases also influence each other and the feedback effect.

Chapter 4 focuses on the question: Which variables influence the feedback process and the feedback effect according to medical students’ and residents’ perception? It is not obvious that variables mentioned in the literature as influencing the feedback process and affect, are perceived in the same way by medical students and residents. Exploring perceptions of feedback is important because perceptions influence behavior.9-10, 38, 43-44 The goal of this study was to develop an instrument grounded in theory and literature to measure the feedback recipients’ perceptions of the educational value of variables embedded in the feedback process within clinical settings.

Based on the aforementioned feedback model (figure 7.1) a questionnaire was developed: the Perceptions towards Feedback in Clinical Settings (P-FiCS) scale. We

139 Conclusion and discussion

followed the approach as described by Comrey11 Gorsuch12 and Wetzel13 and first determined what needs to be measured based on an organizing framework. The focus of our measurement covers a sample of ten cells from a blueprint.

These ten cells reflect the most important categories of a feedback recipients’ perception of the feedback process. An item pool was generated, the format of measurement was determined, the item pool was reviewed by experts, validation items were included, and the items were administered to a development sample consisting of Dutch (n = 382) and American (n = 292) residents and clerks. The questions were answered on a seven point scale ranging from 1 = totally disagree to 7 = totally agree. We then evaluated the items. The data was factor analyzed for each of the four components from the organizing framework, using principal components analysis with varimax rotation. In this way we determined the number of relevant factors. Cronbach’s alpha was used to evaluate the internal consistency of the resulting scales.

In addition, we collected personal and professional data such as age, gender, medical students’ year of study and resident’s specialty. Bivariate correlations and ANOVA were used to assess if differences in P-FiCS scale scores were explained by age, gender, nationality, seniority, and residents’ medical specialty.

Factor analysis produced nine scales containing 46 items in total. Three scales relate to feedback provider’s teaching behavior: Purposeful and Trustworthy, Involved, and Detached. Two scales relate to feedback recipient’s self-evaluation of task performance.

Both a high and low self-evaluation of task performance contributed to perceiving feedback as valuable. One scale was about Message Clarity. Three scales are associated with feedback context: Privacy, Formative Nature of Task, and Critical Nature of the Task. The internal consistency of the scales varied from α = .61 to .76. Group differences in scale scores are found for age, gender, seniority, nationality, and medical specialty.

This sample was used to develop the P-FiCS, so the first outcomes may not be used as results of this measurement. Can we answer the question based on the results of this study? The P-FiCS scale scores suggest that the educational value of feedback was perceived to be high when a) the feedback provider shows involved teaching behavior, b) a feedback recipient evaluates oneself as low on task performance, c) and when a clear feedback message was received. Contrarily, a) when the feedback provider showed detached teaching behavior b) and when the feedback was not given in private the educational value of feedback was perceived as low.

Based on these studies the research questions ‘Which variables influence the feedback process and the feedback effect according to the literature and the perceptions of medical students and residents?’ can be answered. Many different variables influence each of the four phases of the feedback process, and the feedback effect. Task variables, cultural and contextual variables, and feedback recipient characteristics influence phase A - task, standard and first task performance. Phase B - observation and interpretation -, is influenced by observation related variables, instrument and method related variables, time to build a relationship, and also feedback recipients’ characteristics. Phase C -the communication of the feedback message-, is influenced by the characteristics of the person who communicates and through the means employed, the feedback message, the timing, frequency and the intensity of the feedback. The interval between the feedback and the next performance, the activities that take place in this interval, together with feedback recipient’s characteristics affect phase D -the reception and interpretation of the feedback message.

Chapter 7

140

Most of these variables did also affect the feedback effect, except the variables mentioned in phase B. The results from the Dutch and American participants on the P-FiCS indicates that medical students’ and residents’ perceptions of the educational value of feedback in the clinical setting is influenced by a) the teaching behavior of the feedback provider, b) their own self-evaluation of the task performance, c) the message clarity, and d) task and context aspects.

In chapter 5 and 6 the following question is key: ‘How do the variables related to the communication of feedback and the feedback provider medical student’s satisfaction, self-efficacy, and performance influence the performance of a physical examination task?’

Two single blind controlled trails were carried out. We focused on feedback framing (chapter 5) and on feedback provider’s credibility (chapter 6) as independent variables.

The task in both experiments was to practice the Weber and Rinne (W&R) tuning fork test. This procedure tests hearing deficiencies in patients. Students performed this task on Standardized Patients (SPs) according to guidelines provided by an otolaryngologist on an instructive video. The outcome variables are student’s self-efficacy about task performance, satisfaction and performance.

The following procedure was carried out for each of the two experiments. At baseline (T1) students were randomized over two conditions, and they were asked to rate one’s self-efficacy in performing the W&R task. During the pre-test (T2), students watched an instructive video about the W&R task procedure. Subsequently, each student carried out the W&R task on a SP while observed by the feedback provider. Directly after the first task performance, the student rates his or her self-efficacy again. Then, the feedback provider provided feedback about the performance in either condition. During the post-test (T3), the student performed the W&R task a second time on the same SP, and rated self-efficacy a third time. Two to three weeks later (T4) each student was asked to rate their self-efficacy for the fourth time and to perform the W&R task for the third time. All performances were videotaped. The videotapes were analyzed with an observational checklist.

In chapter 5 the focus is on one specific aspect of feedback communication: the feedback framing. Framing refers to the process by which people develop a particular conceptualization of an issue or reorient their thinking about an issue.14 15 Message framing is a variable that is used in many studies outside medical education, such as communication, and it has been shown to have an effect for example on opions, belief of concent, judgements.16-18 In this study framing refers to the valence of the feedback message: whether it is positive or negative.19 Positive framing and negative framing are the two conditions in this experiment. In the positive condition the feedback was framed as: “You did this well, some tips are…” and in the negative condition: “This is not well done: you should change…” The data of 59 first year medical students were used for analysis. Students in the positively framed feedback condition were significantly more satisfied and had significantly higher self-efficacy as measured directly after the performance. There were no measurable differences between conditions in performance either immediately after the intervention or after a delay of two weeks. Over time in both conditions performance and self-efficacy decreased. Self-efficacy differences disappeared after two weeks.

7.2 Evaluation of the goals of this thesis By answering the research questions in this thesis we set the following goals: a) clarify the feedback concept, b) connect research from different disciplines, c) determine whether or not variables being effective in a non-medical context are perceived effective in a medical context, and d) contribute to the understanding why feedback is not always effective. To what extent have we met these goals?

141 Conclusion and discussion

In chapter 6 the emphasis is on one specific aspect of the feedback source: the credibility of the feedback provider. Sargeant describes credibility is as the ability to observe and assess well.20 The feedback provider’s characteristics, the resident’s observation of feedback provider’s behavior, the content of feedback provider’s feedback, and their method of feedback delivery all play a role in being perceived as a credible feedback provider.21 In this experiment high and low credibility are the two conditions. In the high credibility condition feedback was given by a 49-year old male training actor, acting as a professor of Ear, Nose and Throat surgery, wearing a white coat, with a visible identification tag. He was instructed to refer to his clinical practice experience, while introducing himself to the student. In the low-credibility condition, the feedback provider was a 20-year old female third year medical student, two years ahead in medical school compared to the subjects, employed as a student assistant, informally dressed without white coat. The feedback in both conditions was about two good points and two points for improvement. Based on the data of 68 medical students we concluded that credibility did not significantly affect immediate nor delayed self-efficacy. Students receiving feedback from a high-credibility source did not perform significantly better immediately after the feedback intervention, but did so three weeks after the intervention. Although the feedback provider’s credibility did not enhance self-efficacy and satisfaction it did influence the feedback recipient’s performance. The task performance in the low credibility condition was better compared to the high credibility condition directly after the given feedback. But after three weeks this effect was gone.

The group receiving feedback in the high-credibility condition initially did not perform better than the group in the low-credibility condition, but three weeks later their performance score was the same as directly after the given feedback, and it was significantly higher than the group in the low-credibility condition.

The answer on the second question ‘How do variables related to the communication of feedback affect the feedback provider medical student’s satisfaction, self-efficacy and performance while performing a physical examination task?’ is, that both framing the feedback message and the feedback provider’s credibility affect some of these variables.

Framing feedback in a positive manner enhances students self-efficacy and satisfaction with the feedback message on the short term. Receiving feedback from a highly credible feedback provider results in a higher satisfaction and higher performance score on the long term compared to receiving feedback from a less credible feedback provider.

Table 7.2 gives an overview of the variables reported in this thesis, that influence the feedback process or feedback effect according to the meta-review (chapter 3), learner‘s perception (chapter 4), and empirical studies (chapter 5 and 6). Variables with an asterisk influence the feedback process or feedback effect in a certain direction. From the variables without an asterisk we know that they are influential, but it is not always clear in which direction.

Chapter 7

142

In this thesis the feedback concept is clarified. Three different feedback concepts are distinguished - feedback as ‘information’, ‘reaction’ and ‘cycle’. We were able to explain the changes in feedback concepts over time.

When feedback was first used in social sciences it was seen as a ‘cycle’ connecting input and output, and over times the feedback concept changed to ‘information’, in which the message content is key. We also described the similarities and differences between the three concepts. Beside a conceptual definition, also an operational definition based on the feedback concept ‘information’ is proposed. Because an operational definition is based on a conceptual definition, it is important to make a deliberate choice about the feedback concept which will be used in a research project. Mixing up different feedback concepts, or using different feedback concepts interchangeably will be a hindrance for developing feedback theories.22-25 Kluger and DeNisi describe that due to the diversity in fields and paradigms feedback research is carried out in ‘single pockets’.6 As a consequence feedback researchers do not always use the information which is present from studies within different research fields - such as teaching and learning, labor and management, therapy, and communication sciences. Bringing research fields deliberately together, results in additional information on variables that influence the feedback process and effect, and it contributes to theory building regarding feedback.

In this thesis these four research fields are present. The integration becomes clear in the meta-review about the variables that influence the feedback process and the feedback effect (chapter 3) and the perception study about variables affecting the educational value of feedback according to medical students and residents (chapter 4).

In our thesis many aspects from the labor and management literature are presented in the phase of observation and interpretation (phase B): the type of instruments used for collecting information, the aspects of reliability and validity of the instruments, but it is also reflected in literature about cultural differences between feedback providers and feedback recipients and how different types of feedback providers give feedback in a different way. The study on the credibility of the feedback provider fits also in this category. Further issues related to workplace culture and the feedback climate fit in this category.

Research in the communication field sheds light on how feedback messages are transmitted and its effect on one’s behavior. Several issues in this thesis stem from this field.

In the meta-review we found variables related to the communication of the feedback message influencing how the feedback recipient received the feedback and also influencing the feedback effect. From the P-FiCS we learned that medical students and residents perceive the clarity of the feedback message as important. And the question about the message framing is also rooted in a communication tradition.

The above explanation may suggest that these fields are isolated. Nevertheless, in practice the fields are intertwined. In many feedback studies at least two of the four perspectives are present. This makes the feedback research diverse in research approaches and rich in content.

The variety of literature in medical education is illustrated by the description of research themes in a literature sample from the period 2008-2013 on feedback in medical education.

Due to the many research traditions and the many research fields in which feedback is used, interdisciplinary overview studies will help research, advance, and bridge knowl-

Training content and method (3)

Rubrics (3)***

Provider

Feedback communication

Phase C

Culture and context (3)

Content (3)

Feedback provider’s purposeful and trustworthy (4)* Involved (4)* Detached (4)* Message clarity (4)* High self-evaluation on task (4)* Low self-evaluation on task (4)* Privacy (4)* Formative task nature (4)* Critical task nature (4) *

Recipient

Reception & interpretation

Phase D

Task complexity (3) Task nature (3) Culture and context (3) Activities (3) Time (3) Timing (3)

Form (3)*** Medium (3) Intensity (3) Frequency (3)*** Framing (5)**

Content (3)*** Complexity (3)

Feedback recipient’s self-esteem (3)*** Feedback recipient’s goal-setting behaviour (3)***

Feedback source (3) Feedback provider’s Credibility (6)**

Recipient

Task performance2

Phase A2*

* The feedback effect is included in the second performance in phase A2. By comparing the first performance and the second performance after receiving feedback, the difference will be visible when the same type of task is carried out. The effect can be described as: *) based on learner’s perception; **) based on an empirical study; ***).based on meta-analyses and reviews

Task complexity (3)*** Task Nature (3) Culture and context (3) Training content and method (3)*** Nature of the instrument (3) Assessment method (3) Purpose of observation (3) Intensity of observation (3)

Feedback context

Feedback recipient’s cultural background (3)***

Feedback provider’s position (3) Feedback provider’s task familiarity (3)*** Feedback provider’s cultural background (3)*** Time to build relation (3)***

Provider

Observation & interpretation

Rubrics (3)***

Feedback recipient’s skills (3)***

Recipient

Task performance1

Feedback form

Task variation (3) Task subject matter (3)

Task & standard

Phase B

Focus of observation (3) Type of standard (3)

Actor >

Activity >

Phase A1

Feedback content

Feedback Recipient

Feedback provider

Component

Phase >

Johnson’s Mr. A. talks to Mr. B.

Conclusion and discussion

Lasswell Formula

Table 7.1. Organizing framework based on the Johnson’s Mr. A talks to Mr. B model and the Laswell formula. filled with variables reported in the meta-review (3) the P-FiCS (4), the framing study (5) and the credibility study (6)

143

Table 7.2. Overview of variables from the meta-review and their connection with the means (M) and standard deviatsions (Sd)from items from the P-FiCS as perceived by American and Dutch learners (n=674) Variable metareview

Corresponding items P-FiCS

M

Sd

Phase A Task & standard, and task performance Feedback provider’s I-1.‡ Assigns the tasks for me to perform position

Chapter 7

144

4.78 1.40

IV-1. It is given in front of patients† Culture and context/ IV-2. It is given in front of residents Intensity of IV-3. It is given in front of attending physicians Feedback IV-4. It is given in front of medical students† IV-5. It has no consequences for my grades IV-9. It concerns a task that can be performed in several ways Task complexity IV-10. It concerns a task which does not have to be done perfectly IV-11. It concerns a task that has no negative consequences for the patient Phase B Task observation and interpretation

2.52 3.48 3.77 2.95 4.64 4.77 4.58 3.85

1.37 1.41 1.42 1.36 1.45 1.28 1.23 1.37

Type of standard

I-4. Compares my current performance with my past performance I-7. Is usually the same person Time to build relation I-8. Knows me well I-17. Does not know me very well† I-3. Explains the task for me before I start Feedback provider’s I-5. First demonstrates the task before I start task familiarity I-9. Is an expert on the task I have to perform* I-16. Is not an expert on the task I have to perform† I-2. Observes me directly I-10. Observes my performance in person* Intensity of I-11. Observes me discreetly observation I-12. Observes me several times I-15. Observes me just once† Phase C Feedback communication

5.76 4.56 5.14 2.90 5.36 5.12 6.13 2.54 5.05 6.16 4.98 5.91 2.71

1.10 1.39 1.37 1.30 1.39 1.31 0.90 1.19 1.82 0.91 1.35 0.97 1.18

Feedback frequency IV-8. It is given at regular intervals I-6. Tells what I have to change during the observation Feedback timing IV-6. It is given during my clinical activities IV-7. It is given directly after completing a task III-4. Contains a description of my performance III-6. Goes into details Feedback form III-7. Allows me to ask questions and make comments* III-8. Addresses weaknesses in a constructive way* I-13. Gives feedback directly, not through a third person* I-19. Gives feedback through a third person† Intensity of feedback III-2. Focuses on main points III-3. Addresses both strong points and points for improvement* III-5. Presents facts and examples to illustrate the main points*

5.00 4.61 3.93 5.57 5.26 5.76 6.14 6.00 6.43 2.07 5.35 6.22 6.06

1.52 1.70 1.80 1.18 1.14 1.11 0.87 1.09 0.84 1.10 1.28 0.98 0.88

edge gaps between fields. For example, providing an overview of instruments used in these four research fields which measure different aspects of the feedback process and the feedback effect would be beneficial in promoting research on feedback perception.

Due to the different research contexts, it is necessary to determine whether or not variables effective in a non-medical context are also perceived as effective in a medical context. To address this we compared outcomes of the meta-review (chapter 3) with the outcomes of the P-FiCS as described in chapter 4 (table 7.2).

As said before, in the meta-review we determined 33 variables that influence different phases in the feedback process and the feedback effect. The P-FiCS consisted of 46 items that are perceived as influencing the instructive value of feedback. Given the fact that we used the outcomes of the meta-review also as input for the P-FiCS, we should be able to connect the variables from the meta-review to the items of the P-FiCS. Fourteen variables from the meta-review (table 7.2: first column) could be related to the variables from the P-FiCS (table 7.2: second column). The aspects of these fourteen variables reflected in the items of the P-FiCS are perceived as contributing to the educational value of feedback according to feedback recipients in the medical field. We were not able to relate the scales to the variables. The content of the scales referred often to more than one item from the meta-review.

Four items from the P-FiCS could not be related to the variables from the meta-review.

Two items relate to a feedback provider characteristic: to what extent feedback providers feel at ease in giving feedback. The two other items point to feedback recipient’s feedback seeking behavior. These two variables did not turn-up in the meta-review, which means that this has not been much researched, and also in the review of the literature between 2008-2013 as described in the introduction, it was not a theme that received a lot of attention.

From the meta-review 17 variables are not mentioned in table 7.1. These variables -for example task nature and task subject matter (phase A), purpose of observation and focus of observation (Phase B), feedback source and feedback medium (Phase C), and the time and activities in between the feedback and the second performance (phase D)- are not consciously perceived by feedback recipient’s as effective in the medical

145 Conclusion and discussion

Phase D Feedback reception and interpretation Feedback recipient’s III-1. Is tailored to the goals of my training 5.94 1.03 goal setting II-1. That I find difficult 5.54 1.19 4.91 1.44 Feedback recipient’s II-3. My performance is worse than most of my peers II-4. I am convinced that I could have done better 5.16 1.37 skills Feedback recipient’s II-5. I can complete the task with ease 4.07 1.34 self-esteem II-7. My performance is better than most of my peers 4.24 1.25 II-8. I am satisfied with my performance 4.64 1.27 Not in meta-review Feedback provider I-14. Is someone who enjoys giving feedback 5.52 1.15 I-18. Is someone who is reluctant to give feedback† 2.48 1.21 characteristic II-2. I ask for it 5.16 1.55 Feedback seeking II-6. It is unsolicited by me 4.62 1.31 ‡ The item numbers refer to the codes form the blue print as described in chapter 4 * Items with a score higher than 6.00 on a seven point scale. † Items with a score lower than 3.00 on a seven point scale. These specific boundaries are subjectively chosen.

Chapter 7

146

context. This does not mean they are not important but they are not recognized as being effective by medical students and residents.

This thesis gives insight into the complexity of the feedback process, and it offers explanations of why feedback is not always effective. Firstly, a lack of uniform terminology, a clear ‘feedback language’ may hinder feedback research. Secondly, many variables influencing the feedback process and feedback effect interact with each other and it is not always clear how they interact. Thirdly, perceptions influence behavior and we do not yet fully understand the perception-behavior link.

It is important to notice that this thesis -in particular chapter 2, 3 and 4- does not only focus on one phase in the feedback process or exclusively on the feedback effect, but gives attention to the feedback process as a whole. By paying attention to the feedback process in relation to the feedback effect, it is possible to draw lines which could be interesting for further research. The less researched areas become visible. It does not give insight why feedback may not be effective, but it does indicate that many questions related to feedback effectiveness are still unanswered.

We integrated literature from different perspectives. This integration gives insight into the complexity of the ‘feedback language’. Many different terms are used for similar variables and concepts, probably due to the different traditions and theories of the fields in which feedback research is carried out. This is a significant hindrance in furthering feedback research and may be a possible explanation for the fact that we only see slow progress in the field of feedback research, and why the knowledge about the feedback effect, the variables that influence the feedback process and their effect is small.

In the attempt to identify variables that affect the feedback process and the feedback effect, we found about thirty variables. In many studies these variables interact with other variables. Due to these interactions it is not possible to clearly disentangle what the feedback effect is and could explain why the effect sizes in reported feedback research are often small to moderate.

The insight from the feedback recipient’s perceptions gave an idea of the complexity of feedback processing. When the message is provided as clear, concise and as precise as possible, there is still no guarantee that there will be a positive feedback effect. The given feedback will always be ‘filtered’ and it is hard to predict in which way this will be.

Particularly the influence of feedback perception in relation to feedback effectiveness on performance is a part of the feedback field which has not been studied thoroughly.

Only in a few studies do we see the acknowledgement that perceptions matter.9, 40 7.3 Discussion, limitations and strengths We will view the results of this thesis in the light of the communication theory and the social perception theory: are these theories able to offer an explanation for our findings? Further assumptions, strengths and weaknesses will be discussed.

Results in the light of the communication perspective In this thesis we gave attention to communication aspects such as the message, the sender, the receiver and the dialogue. It is assumed that the effectiveness of oral feedback partly depends on how the feedback is communicated by the feedback provider, the dialogue between the feedback provider and the recipient, and also how the feedback recipients receive this information. We integrated two models from communica-

Feedback in relation to theory of social perception Many theories reported in the current feedback literature have a cognitive background.

In these theories conscious and intentioned actions have a central place. These actions cause certain effects, behaviors, or judgments. Into this line of thinking feedback fits in well: feedback contains information which can be processed and transferred into ac-

147 Conclusion and discussion

tion theory, the Lasswell formula and ‘Mr. A talks to Mr. B’ to create a model which was helpful in approaching our research from a communication perspective.

Many variables related to the feedback message influence the feedback process and the feedback effect. Message clarity is one of the scales of the P-FiCS. Feedback recipients seem to perceive this scale as affecting the educational value of feedback. The message framing study (chapter 5) shows that it also influences a feedback recipient’s performance -especially on the long term- and perceptions.

The meta-review, the P-FiCS and the credibility study (chapter 6) show that also feedback provider-related variables are important. Scales relating to the feedback provider’s teaching behavior affect the feedback recipient’s perceptions of the feedback message.

Furthermore we see from the experimental studies that the feedback provider’s credibility affects the outcome of the subsequent performance, in particular on the long term.

The meta-review shows feedback recipient related variables which influence the feedback effect. This is confirmed by the scales from the P-FiCS: feedback recipients perceive certain circumstances in the feedback process as more valuable than others, especially self-evaluation. However, we did not investigate the relationship between the feedback recipient’s perceptions and the effectiveness of the feedback.

Besides the fact that feedback recipients perceive asking questions and making comments as affecting the educational value of feedback, no new information on the feedback dialogue is included in this thesis.

Recent research in the field of communication is in line with the studies here carried out. In communication research attention is given to the feedback recipient’s perceptions; not only a feedback recipient’s perceptions related to their task performance,26 but also how they perceived the feedback provider’s behavior,27-30 the instructive value of feedback31 and the learning climate in which feedback was given.32 Based on the results of the empirical studies in this thesis new insights in communication research cannot be claimed, but we have added significant evidence on these topics in the context of medical education.

The communication perspective is especially valuable in unraveling the feedback process, and it is helpful in categorizing variables that influence the feedback process and the feedback effect. Its contribution lies predominantly in showing the complexity of the feedback process. Is this approach able to offer a clear explanation of why feedback is not always effective in the way that we think it should be?

Johnson described in his model the filters that are present in the communication process.33-34 When the sender receives information it is filtered by the sender’s perception.

Based on the perceived information the recipient acts and the information that the recipient receives is filtered too. We need to realize that these two filters also apply to our research: the listener hears a message and filters the information and the feedback provider does the same thing. There is always a bias of some kind due to these two filters: what is heard and what is said might be different. Social perception theory37-41 might further help to elucidate this issue.

Chapter 7

148

tion. Furthermore, when feedback recipients consciously listen, they hear what needs to be changed and are able to act upon the feedback. However, many studies showed that the effects of feedback are small to moderate6, 35-36 and have much variation.6, 8, 36 In the light of the cognitive background it is surprising that feedback has only a small effect: Why do trainees not incorporate the suggested changes?

The social perception theory offers explanations that may help our understanding of why feedback has limited effects in general. Social perceptions or social beliefs refer to constructs such as expectancies, categories, stereotypes, prototype schemata, intuitive and implicit theories and hypothesis. Social perception is seen as the end of a categorization process and it is also described as going beyond the information.37 When people perceive for example a dog, in their memory semantically and lexically information is activated. This happens also when social knowledge is perceived. This does not necessarily happen intentionally. This incidentally activated knowledge influences peoples’ judgment. People are unaware of the knowledge that is activated but they are also not aware of how it affects their judgments and impressions.38 In a similar way behavioral responses can also be activated after perceiving a social stimulus. One example is mimicry, when people unintentionally mirror the behavior of their partner during a conversation.38 According to Swann it is the everyday purpose, the goals people have, the people they interact with, and the context in which persons are, that determine the accuracy of the perceptions derived from the information around us.39 The accuracy of our perceptions is also influenced by expectations, and reactions to negativity.40-41 When it comes to the perception of someone’s personality, this is for example influenced by our level of acquaintance, and how much we interact with a person, but people are also able to perceive dominance or warmth from photographs and voices.42 All these aspects are important in social interaction, such as feedback. When social perception influences behavior,38, 40, 43-44 for feedback research it is important to understand how the perceptionbehavior link works. According to social perception theory our behavior is not only shaped by conscious decision-making, but also by an unconscious part. In human-tohuman oral feedback dialogues, both learner’ and supervisors will perceive a situation differently according to their context and frame of reference, this might determine their behavior, and they are probably not aware of this. Perception studies are necessary to understand learner’s and supervisor’s perception of the feedback: the similarities and the differences. Relating perceptions to behavior may help understand the perceptionbehavior link.

How should we evaluate the results obtained in the light of social perception theory? In this thesis we explored learner’s perceptions towards the feedback process (chapter 4, 5 and 6). Given the fact that people are often not conscious about their own perceptions this a challenging area of research.

Feedback recipients perceptions in phase A could relate to the perception of the task such as previous performance, perception of task standards, the feedback provider, the feedback recipient’s own behavior, and the context in which the task needs to be performed. In phase B, observation and interpretation, the feedback provider’s perception could be about the feedback recipient’s former performance, the task, the context, the observation form, etc.

During the communication of the feedback, phase C, the feedback provider perceives the context in which the feedback takes place, the relationship between feedback pro-

vider and feedback recipient, perception of certain non-verbal behavior, perception of bringing the message, perception of their own capabilities regarding the feedback and their own expertise.

When the feedback recipient hears the information, phase D, the feedback recipient’s perception of feedback provider, the feedback message, the interaction, the feedback context, previous experiences, time pressure, can all affect how the message is received.

This is not a complete list of a feedback recipient’s and feedback provider’s perception.

In this area of research many themes remain unexplored. As social perception is a fairly recent field, it is an area in which much still needs to be researched, especially when it comes to feedback.

Definition In chapter 2 we proposed a definition of feedback. By defining a term the description and the concept is made explicit. In the definition we have not mentioned who the feedback giver should be: whether a novice or an expert, or how strictly we will use the

Conclusion and discussion

Assumptions related to models used In chapter 1 we introduced a feedback model in which communication receives a lot of attention. This model was dominant in the meta-review (chapter 3), the development of the P-FiCS (chapter 4) and in the choice of the independent and dependent variables in the framing and credibility experiment (chapter 5 and 6). Choosing a specific model has consequences and one could argue that a model based on a teaching and learning perspective might have been more appropriate for studies relating to medical education. We chose for a communication model because we assumed that the communication process would receive significantly more attention in this model.

Another concern was the choice of the phases and the description of the phases in the model, especially given the fact that the studies on which the models are based are old; published between 1940-1950. When models are compared, different phases are described. Phase D of the model, the reception and perception, contains, according to Shrauger, the following steps: (a) reception and retention of evaluative information; (b) assessment of the information source; (c) attribution of responsibility for the outcome obtained; (d) changes in self-evaluation; (e) satisfaction with feedback; (f) changes in task performance.45 A phase can always be subdivided into smaller sub-phases. Although the content in the sub-phases can be described in different ways, similarities between the other models do occur.46-50 In chapter 4 and 5 we have chosen outcome variables (self-efficacy, satisfaction and performance) based on Kirkpatrick’s model, which is often used in the medical context.51-52 This model was developed in 1959, to evaluate training outcomes in business organizations.53-54 The model simplifies the evaluation process into four levels on which a training can be evaluated: the level of reaction, learning, behavior and organizational outcomes.53-54 The literature points out several limitations:53-56 the model is incomplete –for example cultural factors influencing the learning climate are missing,53 56 it assumes causal linkages between the four levels which are not confirmed, and it assumes that the highest evaluation level provides the most useful information.53-54 We used the model to categorize the outcome variables in the experimental studies. We did not use it for the purpose for which it was developed -evaluating trainings for the improvement of the study programs or its effect on organizations. The mentioned limitations of Kirkpatrick’s model have not consequence for our findings.

149

Chapter 7

150

definition. When the concept of a cycle is used, it can only be determined after the event whether a process of giving and receiving information was a feedback process or not, so the use of the concept of feedback as a cycle is limited. In contrast, the information concept can be used whenever information is transmitted. So the application of the information concept of feedback is broader. When the cycle concept is used, many of the feedback articles would need to be excluded from the study, because the cycle concept is not often used. When the information concept is used, there is a wealth of theory and literature available for review.

Since the publication in 2008, this definition is often cited. This could be an indication that the nformation concept useful for their research and practice.

The empirical and theoretical studies used in this thesis do not always define the term feedback properly. We used the information concept of feedback as a definition in our study, but we are not sure if this conceptual definition is always used in the articles we have cited of referred.

Task choice: WR tuning fork tests In the context of our experiments (chapter 5 and 6) we choose the W&R tuning fork task. This task is used in the clinical setting in both the otolaryngology and neurology fields for detecting hearing impairments in patients.57-58 Although electronic devices such as the audio scope have been developed, the W&R tuning fork test offers support to modern audiometric testing. Therefore, in spite of being an old technique,59 it is still taught in medical schools.60 Task choice is important when experiments on performance are at stake, especially when a task needs to be observed. Would a different task choice have led to different results?

The following reasons had led us to our task choice. We needed a task which had the following features: a) new to the students, b) observable, and c) having a certain level of difficulty. The W&R tuning fork test met these criteria.

The experiments took place with first year medical students who were untrained in using the tuning fork tests. A task such as measuring blood pressure might be already familiar to some students, especially if they had a nursing or a paramedic background before they entered medical school. This task was likely to be more specific. When we measured the studentsâ&#x20AC;&#x2122; self-efficacy before task performance: no outliers were detected which suggests that students were not previously acquainted with the task.

From the W&R tuning fork test most aspects were observable. Aspects which were not observable such as whether or not the tuning fork was placed on the correct place on the mastoid and if it was struck to the bone firm enough were made observable by giving the SPs instructions to give a sign when this had happened.

Lastly, the W&R tuning fork test has many sub-tasks which makes the task difficult to a certain extent. When the tuning fork is vibrating too loud, it will produce overtones, which disturb the test. It is hard to place the tuning fork in the proper position by the ear in the Rinne test,60 and to strike the tuning fork in the right way on bones especially when a person has a lot of hair or legs from eyewear behind their ear. Furthermore, it is not easy to understand the reasoning behind the W&R-task,61 which makes it hard to correctly explain the test and its results to patients. So, in the W&R tuning fork test there are many different aspects that can be incorrectly performed. A study on the W&R tuning fork test performance of second year medical students gave insight into which components students had difficulty with.62

Ecological validity A potential threat to experimental studies (chapter 5 and 6) is the lack of ecological validity.63-64 The experimental studies were carried out in a simulated clinical outpatient office in the skills lab, and not in the hospital. Feedback providers were portrayed by a third year medical student, a resident and a professional training actor, but not an otolaryngologist. The medical students in this study were volunteers, while in a normal situation all students receive feedback and there is not much choice. This leads to the questions whether or not the results of this experiment may be generalized to feedback situations in the real clinical setting.

We believe that the outcomes of the experiment can be generalized to the clinical setting because similarities between the created and the real context are present. Students performed the W&R task on standardized patients they had not met before. This is similar with a first patient encounter in the real setting: in both cases students have no existing bond with the patient. The medical student, the resident and the training actor were also new to the students, which is often also the case in the clinical setting: it takes a while before medical students know all the physicians. We have copied the clinical atmosphere as much as we could: for example the professional training actor used a copied UMCU name tag with the title ‘professor in Ear, Nose and Throat Surgery’, they were dressed with a formal clothing (tie, ironed shirt, no sneakers), a white coat, and a stethoscope. The students’ oral reactions, the written comments and reflections made clear to us that they were impressed with the situation. First year medical students have, in this stage of their studies, not experienced working and learning in a clinical setting, so they cannot compare the created situation to the real hospital setting. Therefore we believe that the alternative setting would not affect their performance to the extent that they would be able to compare and or notice the difference.

By using the created context we were able to control many variables such as the difficulty of the patient, the time for performing the tasks, privacy and reactions. This leads us to surmise that the feedback effect might be smaller when these experiments are repeated in the clinical context.

151 Conclusion and discussion

In clinical practice slight variations in performing the W&R tuning fork task exists, therefore a ‘golden’ standard for task performance was not available. Together with a otolaryngologist, we developed a standard based on the performance in the instructive video. The standard was captured in a observational checklist. In the two experimental studies students were asked to perform the task as close as possible to what they had watched on the videotape. Feedback providers referred in their feedback to the videotapes as the standard of performance.

We only can speculate if the results of the experiment would lead to a similar outcome if a different task was used. From the results of the meta-review and the P-FiCS we know that tasks with a critical nature or formative nature influence the perception of the feedback value in feedback recipients. Task familiarity also affects performance.

However, if a different task would meet the features described above, is not critical in nature and can be used in a formative way, we assume the outcomes should be similar if the experiment is carried out under similar conditions and circumstances, supporting its generalizability.

Chapter 7

152

Limitations and strengths The limitations in this thesis relate to the generalization of the results, and sources of bias found in this study. To what extent are we able to generalize the results of our studies? The literature reviews from chapter 1 and 3 have respectively 2013 and 1012 as their end date. New publications could have influenced the outcomes.

Many feedback studies are carried out in a non-medical context. Therefore it was necessary to explore whether or not the variables in the non-medical setting could be transferred to the medical setting. From the perception study we noticed that several variables were perceived as influential in the medical setting, but we need to further explore if they also influence performance the medical or clinical setting.

Both American and Dutch medical students and residents participated in the P-FiCS. Although two different countries were involved, it does not mean that the results can be generalized to other countries. Cultural aspects play an important role in communication and further work on feedback needs to be performed in this area. The data-set in this study was used for identifying scales. The scales were meaningful and were carefully used as indicators of what students and residents perceive as influencing the instructive value of feedback. To use the questionnaire further validation of the P-FiCS is necessary, for example with measurements of the feedback environment, or during specific activities performed in the clinical context.

Reviews and meta-analyses suffer from publication bias.65-67 Studies with negative results are difficult to publish, and this affects the balance of reviews and meta-analyses.

These reviews and meta-analyses were used as input for our meta-review and therefore it also affects this study.

Students volunteered to take part in the P-FiCS study and also in the experiments. This freedom to volunteer could also have affected the results,64, 68 but given the large sample size we do not think this has greatly affected the results of our study.

The major strengths of this thesis are the relevance of the topic and the integration of the literature from different fields about the feedback process and the feedback effect.

The relevance of this thesis that it is that medical students, residents, medical educators, and indeed nearly everyone in the medical education field perceive this topic as important. Within the curricula in medical schools feedback receives attention, feedback training is provided for residents, and feedback is an integral part of faculty development courses. The attention to feedback is also illustrated by the growing number of publications on this topic. In spite of the long research tradition, many questions related to the effectiveness of feedback remain unanswered. By focusing both on the feedback, the feedback process and the feedback effect the broader context also receives attention. This gives more insight into variables influencing the feedback process and effect, and contributes to an awareness of the complexity of the feedback process.

Although we used a model which was rooted in communication theories we heavily relied on theories from other fields within the social sciences. We integrated the feedback literature in particular from: communication, teaching and learning, therapy and the labor and management studies. Bringing knowledge from different fields together led to a bigger picture of feedback research. It gives an insight into how history has affected feedback research, the different angles from which feedback research can be approached and the different research topics within feedback research. The outcomes of the integrated literature were verified in the medical education context and selected variables were explored in depth.

7.4 Research suggestions and practical implications for researchers

153 Conclusion and discussion

The research suggestions discussed in this section are directed towards theories which can be further used in feedback research, ideas on how different fields within the feedback research can be connected, topics which do not receive sufficient attention, and suggestions on the terminology used in feedback research.

As was shown in the first chapter each research field accentuates different aspects of feedback research. In communication research, the focus is on the transmission of feedback messages, while in the organizational and management theory employees’ and employers’ behavior is changed due to organizational development and feedback.

A research field is often connected to certain theories, so the choice of research field directs theoretical choices.

Furthermore, many feedback studies are dominated by cognitive theories about how we memorize and learn new information. A central notion in the cognitive theories is the fact that learners can deliberately influence their own learning process, for example by practicing, by memorizing and with other techniques of deliberate practice.69 Within feedback research not much attention is paid to the unconscious processes that also affect our learning, in which social perception is an important element. This became clear in both chapter 3 and 4 for our thesis. Choosing the social perception theory as a point of departure would lead to different questions and different feedback research.

This will be a valuable addition to the research theories that dominate today’s feedback research. Theories of social perception allow explanations of the variables that implicitly influence the feedback process and the feedback effect. Research on this would provide further clarification on the perception-behavior link within feedback research.

Overview studies, such as literature reviews and meta-analyses, are beneficial to connect fields and theories, and for providing an overview of the development of a feedback thread throughout time. Both types of synopses are important.

Research in medical education can benefit from the theoretical perspectives from the fields of management and labor, communication, teaching and learning and therapy.

Medical education research integrates aspects of the four aforementioned fields. Feedback research in medical education can profit more from the long history of interdisciplinary research. The diversity in feedback research should be an advantage instead of a barrier when overview studies are able to make connections between the different fields. Literature studies of feedback research’s history will also disclose that many aspects have a long tradition: the effectiveness of positive versus negative feedback dates from 1970’s, and also research on the feedback order dates from that time. Developments in the field of feedback are not so rapid; many questions that have been asked in the past, are often still unanswered. Researchers tend mostly to focus on current literature because it shows recent developments. For feedback research however it can be valuable to include older publications also. In the 1980’s feedback received a lot of attention followed by a decline, and from 2005 onwards the topic is considered again.

Valuable theoretical articles from the past help researchers to understand the current state. Often they are neglected, such as Fedor, Franzen, Greller,70 Ilgen,71 Herold, and Larson72 If researchers paid more attention to feedback history, they would realize that certain aspects we consider innovative, have been mentioned before.

Chapter 7

154

Focus on feedback process and feedback effect Certain trends can be identified in feedback research. In feedback research the focus is often on the feedback effect, and the feedback process receives less attention. Many articles focus on the effectiveness of different feedback interventions: Does feedback work? What is the effect? and, Under which circumstances is it effective? As described in chapter 3 the results are diverse and often contradictory. Investigating the relationships between variables within the feedback process will help us to understand better how the feedback process alters the feedback effect. Research questions that could receive attention are: How do variables related to learner’s first task performance, such as task perception, task complexity, task nature, the culture and context, learner’s age and learner’s skills influence the feedback giver’s communication and feedback receiver’s feedback reception? When reliable and valid observation instruments are used, does this influence the feedback communication, the leaner’s reception of the feedback, and does this influence the effectiveness of the feedback? How do variables related to the communication of the feedback, like feedback source, feedback medium, feedback complexity influence the feedback dialogue and the learner’s reception and perception of the message? How do variables such as learner’s self-esteem and their goal-setting behavior affect their reception and perception of the feedback?

A second trend is a lack of research in certain areas such as feedback communication, feedback perception and the influence of personal variables on the feedback effect.

This is illustrated in the meta-review (chapter 3), where many blank spots are found in table 3.2. In particular this can be seen with the personal variables and the interaction between the feedback provider and feedback recipient. Understanding how these influence feedback perception will give us valuable insight into why feedback is not always effective. Examples of research questions could be: What is the relationship between learner’s feedback seeking behavior and feedback providers’ feedback skills?

Does a learner’s feedback seeking behavior affect the feedback effectiveness?

Research questions in relation to personal variables and perception are: When feedback provider’s feel competent in giving feedback how does this affect the feedback dialogue, the perception of the feedback process of the learner and the provider, and the effectiveness of feedback? The research focus should change from effectiveness to towards the feedback recipient’s perception. Perception is one of the final filters the feedback message has to pass, before it will be brought into practice.

Feedback ‘language’ In feedback research it is important to pay attention to feedback concepts and to terminology in which the feedback process and feedback effect are described. In some cases the meaning of terminology depends on the feedback concept. One example is the adjectives ‘positive’ and ‘negative’ in relation to feedback. In the cycle concept, positive and negative refers to the feedback loop. In a negative feedback loop the balance in a system is maintained, and the loop is closed. While a positive feedback loop leads to a distortion in a system, the loop is open.73 In this example positive and negative refer to feedback effects. When positive and negative are used in the reaction concept the adjectives refer to the interaction between the feedback provider and feedback recipient: i.e. the communication process, the perceived impact, the perception of the process, or the tone of the communication. In the information concept the adjective refers to the form and the content of the message: was the information positive (good points) or negative (points for improvement). If a feedback concept is not consistently used, these

7.5 Practical implications for supervisors and trainees The literature describes many guidelines on feedback provision.74-84 These guidelines are mostly written from a feedback provider’s perspective, and they address the issues of downward feedback: supervisors providing feedback towards learners. Tips on how learners can provide feedback towards supervisors are scarce.74 The guidelines address issues such as the importance of planning for feedback, respecting the feedback recipient’s privacy,75 allowing self-assessment, establishing follow-up plans,76 exploring the discrepancy between the feedback provider’s perception of learner’s behaviour and learner’s view,77 asking recipients to verify the feedback,81 focussing on action and behaviour and not on personality,79 defining consequences if the feedback recipient’s behaviour is not corrected,80 creating a safe environment by evaluating -as a feedback provider- your own state of mind, do you have enough time, are you under stress, are you in the right frame of mind,82 et cetera. Often these articles are based on ‘gut feeling’, and theoretical underpinning of evidence is missing.

Table 7.3 gives an overview of practical implications derived from the literature studies (chapter 3) and the empirical studies (chapter 4, 5 and 6) in this thesis. The four phase feedback model is the backbone for the practical implications. Both the perspectives of the feedback provider and feedback recipient (phase A and D) and feedback provider (phase B and C) are addressed. Both feedback provider and feedback recipient have an active role. Feedback is part of process, it is an activity that can be prepared for by both the feedback provider and the learner.

This chapter will be closed with two practical implications for feedback recipients and feedback providers.

155 Conclusion and discussion

adjectives lead to confusion. Researchers should therefore make a deliberate choice about the feedback concept which they will use, and explicitly describe and define their feedback related terms.

Developing feedback terminology will also help in theory building. In the first chapter we identified themes within feedback research. In one theme the word ‘feedback’ was used for data received from the evaluation of curricula, teaching programs, or instruments. This information was not used to improve a person’s performance but to reflect on curricula, programs, et cetera. For this type of information and purpose, the term ‘evaluation’ would be more appropriate instead of calling it ‘feedback’. Developing the feedback language would make feedback research more consistent and enhance the effectiveness of literature searching.

From the literature about framing the feedback message (chapter 5) we noticed that many of the terms used to describe feedback content are not clearly described e.g.: valence, sign, feedback tone, tone of voice, frame, package. As a consequence it is difficult to estimate what is meant by each word and precisely how these words relate to each other.

This can also be said about the following sequence of ambiguous terms: audit, appraisal & assessment, assessment feedback, assessment for learning, evaluation, feedback, formative assessment, formative evaluation, formative feedback, peer-assessment, peer-evaluation, peer-feedback, performance evaluation, performance appraisal, personnel evaluation, summative feedback, et cetera.

So consensus studies within the medical setting about feedback terminology and concepts is important.

Practical implications for feedback recipients and feedback providers In medical education learners are the owners of their learning process. They have to be active and acquire their own feedback. Feedback recipients can influence many aspects of the feedback process and their interaction with the feedback provider, for example by engaging actively in the feedback dialogue and by creating a follow-up on the received feedback. When feedback providers checks learner’s perceptions and explore how to tailor the message to their goals, they increase the chance that feedback will be effective.

Chapter 7

156

Dialogue Not all feedback providers feel comfortable in giving feedback. They are afraid of harming the relationship with the feedback recipient, and especially when it comes to providing feedback about points for improvement.85 When feedback recipients encourage feedback providers to be specific, to let them know that they appreciate receiving feedback, to give them a clear invitation to be specific and open, and to give feedback providers the idea that feedback is welcome, this encourages feedback providers to deliver a specific, elaborate and clear message.

By asking questions about the feedback content and the underpinning arguments the feedback recipient creates a dialogue and also has an influence on the feedback conversation. If a feedback provider does not make the standards very clear the feedback recipient needs to ask for the reference points, examples, particular descriptions of what went well, what could be improved and tips for improvement. When feedback providers are not able to make standards explicit it is necessary to ask probing questions: ‘What do I need to do to receive a 4 on a 5 point scale?’ ‘What should I do to achieve a score above expectations?’ This forces the feedback provider to think through the relationship between the observation and scoring the scale. Sometimes the standards are not even clear in the mind of feedback providers and the enquiry by the feedback recipient feedback provider will assist in making standards explicit to both parties.86 A clear feedback message is perceived by the feedback recipient as having a high educational value and when the feedback content is encouraging, elaborate and specific it increases the feedback effect.

Follow-up When feedback recipients discuss with the feedback provider how they are going to apply the feedback then feedback recipients are able to work on the goals they have set for themselves. Possible approaches for follow-up are: discussing the application of the feedback suggestions, discussing when in the next weeks feedback will be given again about a similar performance, discussing how to engage in deliberately practicing the feedback.69 Formulating new goals based on the received feedback and making it part of the Personal Development Plan is another way of creating a follow-up. When feedback recipients are able to formulate goals specifically, it gives them a focus for asking feedback at a later time.6-7, 36, 87 Exploring how the given feedback can become part of a multi-facetted intervention, is also a way of creating a follow-up. When feedback is part of a multi-facetted intervention this increases the effect on performance. When the feedback is embedded in a rich learning context, -such as education about a specific topic, a plenary about it, reading literature is required, and tests need to be performed,- then the feedback changes form

Table 7.3. Overview of actions for Feedback Providers in general and in phase B task observation and interpretation and phase C feedback communication of the feedback process and for Feedback Recipients in general and in phase A, task, standard, task performance and phase D feedback reception and interpretation of the feedback process Feedback process

Task, standards, task performance and observation and interpretation

Feedback communication and feedback reception

Feedback provider

Reflect on why you want feedback (3, 4); Choose a feedback provider carefully (purposeful and trustworthy, detached, involved, credible) (4, 6); Determine on which topics you want feedback: set goals (4); Determine which form of feedback works best, for example your feedback preference (3, 4, 5).

Make explicit that you will observe and give feedback (3, 4); Explore recipient’s feedback preferences, expectations and goals; feedback recipient’s skills: low more effect, feedback recipient’s self-esteem (3, 4): Explore task dimensions (3, 4); Be aware of the level of task and task complexity (3, 4); Be aware of the standard you use (4); If possible use a framework in which feedback can be provided (4).

Phase A: Task, standards and Task performance Reflect in action (4); Invite supervisors to give feedback (4); Be aware of formative nature of the task and the critical nature of the task (3, 4).

Phase B: Task observation and interpretation Use instruments which are reliable and valid (3); Develop and use rubrics (4); Observe a task frequent in different situations (3,4); Observe and make notes (3,4); Be aware of interaction between feedback provider and feedback recipient which influences variables (3, 4, 5, 6); Stay focused on the points to be observed (3, 4); Be aware of your task familiarity (3, 4, 6).

Phase D: Feedback reception and interpretation Check perceptions (3, 4); Ask for further clarification on standards tips, examples and descriptions of performance, et cetera (4, 5); Summarize the feedback and discuss followup and try to related it to goals set (3, 4); Encourage feedback providers to be specific and to let them know that you appreciate receiving feedback (3, 4); Weight the given feedback and decide what you are going to do with it (credibility) (6); When feedback did not reach you, try to reflect on the reasons why it did not (self-evaluation) (3); Practice deliberately applying the feedback (goals) (4).

Phase C: Feedback communication Communicate rich feedback (3, 4, 5); Place feedback in perspective of the goals mentioned by the feedback recipient (3, 4); Observe feedback recipient during communication of feedback, especially on nonverbal language (3); Check feedback recipient’s perception of the message (3, 4); Engage in a dialogue (framing) (3, 4, 5); Embed feedback in a multi-faceted intervention (4); Address a follow-up (frequency) (3, 4) Give it in private (4).

157 Conclusion and discussion

Preparation

Feedback recipient

a single-facetted to a multi-facetted intervention. Based on the literature this makes feedback more effective. A feedback recipient’s progress does not merely depend on feedback.

Chapter 7

158

Check perceptions As discussed, perceptions influence behavior.38, 43-44 It is therefore of crucial importance to check the feedback recipient’s perception of the message. Which parts of the feedback came across well, and which parts did not come across. Often feedback providers think their ‘duty’ is done after feedback is communicated. Exploring how the message comes across, and how the feedback influences the relationship can also strengthen the intrapersonal bond between feedback provider and feedback recipient. The feedback provider might get some valuable insights about their own communication style, valence, tone of voice, or the influence of certain words s/he used. Often these conversations are useful in further adjusting the feedback style to the feedback recipient’s needs.

Summarizing the feedback aloud is a way to check if the feedback recipient selected the points which the feedback provider intended to give. Feedback recipients sharing their intentions with the feedback provider for what they intend to change in a subsequent performance is also a method to check perceptions of the feedback. The two mentioned methods also help the feedback recipient to reflect to what degree the received feedback matched their goals.

Tailor feedback to goals Beforehand feedback providers can explore a feedback recipient’s feedback preference, and expectations regarding the observation and the feedback. Asking whether or not the feedback recipient has certain goals s/he would like to accomplish and whether they have specific points for improvement which they are working on is generally perceived as beneficial. When feedback recipients have goals in mind, feedback is more effective. When a feedback provider knows a feedback recipient’s goals it makes it easier for them to tailor the feedback, and the feedback provider can assume the feedback will be welcome. This in turn makes it easier to communicate feedback. When a feedback provider knows the goals a feedback recipient has in mind, this background information helps the feedback provider to tailor the message towards the feedback recipient. Feedback recipients accept feedback better when the message refers to the goals s/he has set.6, 8, 36 In this instance feedback is generally more acceptable and it motivates the feedback recipient to act upon the given feedback.

Creating a feedback culture Archer explains it is not useful having feedback only on a single occasion but a culture of feedback is necessary.88 Feedback giving and receiving has to become ‘normal’ and integrated in the daily activities.

By engaging in a feedback dialogue, creating a follow-up, and by discussions about perceptions regarding giving and receiving feedback, important steps in creating this culture will be set. Most importantly, these advantages should lead to avoiding risk and errors, further improvement of learning and teaching in the clinical setting, and the quality of patient care.

References 1.

2.

3.

4.

5.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.

23.

24.

25.

26.

159 Conclusion and discussion

6.

Fodor JA, Garrett MF, Walker ECT, Parkes CH. Against definitions. Cognition 1980;8:263267.

Dodd SC. Operational definitions operationally defined. Am J Sociol 1943;48(4):482-491.

Medin DL, Lynch EB, Solomon KO. Are there kinds of concepts? Ann Rev Psychol 2000;51:12147.

Clement DA, Frandsen KD. On conceptual and empirical treatments of feedback in human communication. Commun Monogr 1976;43:11-28.

Frandsen KD, Millis MA. On conceptual, theoretical and empirical treatments of feedback in human communication: Fifteen years later. Commun Reports 1993;6:79-91.

Kluger AN, DeNisi A. Effects of feedback intervention on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychol Bull 1996;119(2):254284.

Jamtvedt G, Young JM, Kristoffersen DT, O’Brien MA, Oxman AD. Audit and feedback: Effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2006 Apr 19;(2)(2):CD000259.

Hattie J, Timperley H. The power of feedback. Rev Educ Res 2007;77(1):81-112.

Watling CJ, Lingard L. Toward meaningful evaluation of medical trainees: The influence of participants’ perceptions of the process. Adv Health Sci Educ Theory Pract 2012;17:183-194.

Leung K, Su S, Morris MW. When is criticism not constructive? The roles of fairness perceptions and dispositional attributions in employee acceptance of critical supervisory feedback.

Hum Relat 2001;54(9):1155–1187.

Comrey AL. Factor analytic methods of scale development in personality and clinical psychology. J Consult Clin Psychol 1988;56:754-761.

Gorsuch RL. Exploratory factor analysis: Its role in item analysis. J Pers Assmnt 1997;68:532560.

Wetzel AP. Factor analysis methods and validity evidence: A review of instrument development across the medical education continuum. Acad Med 2012; 87, 1060-1069.

Chong D, Druckman JN. Framing theory. Ann Rev of Political Sciences 2007;10:103-126.

Entman RM. Framing: Toward clarification of a fractured paradigm. J Commun 1993;43(4):5158.

Van Gorp B. The constructionist approach to framing: Bringing culture back in. J Commun 2007;57:60-78.

Druckman JN. On the limits of framing effects: Who can frame? J Polit 2001;63(4):1041-1066.

Scheufele DA. Framing as a theory of media effects. J Commun 1999;49(1):102-122.

Levin IP, Schneider SL, Gaeth GJ. All frames are not created equal: A typology and critical analysis of framing effects. Organ Behav Hum Decis Process 1998;76:149-188.

Sargeant J, Mann K, Sinclair D, Van der Vleuten C, Metsemakers J. Challenges in multisource feedback: Intended and unintended outcomes. Med Educ 2007;41:583-591.

Bing-You RG, Paterson J, Levine MA. Feedback falling on deaf ears: Resident’s receptivity to feedback tempered by sender credibility. Med Teach 1997;19:40-44.

Lundberg GA. Operational definitions in the Social Sciences. Am J Sociol 1942;47(5):727-745.

Kendler TS. Concept formation. Annu Rev Psychol 1961;12:447-472.

Williams RL. Operational Definitions and assessment of higher-order cognitive constructs.

Educ Psychol Rev 1999;11(4):411-427.

Wacker JG. A theory of formal conceptual definitions: developing theory-building measurement instruments. J Oper Manag 2004;22:629–650.

Hinton JS, Kramer MW. The impact of self-directed videotape feedback on students’ selfreported levels of communication competence and apprehension. Commun Educ 1998;47(2):151-161.

Chapter 7

160

27. Kerssen-Griep J, Trees AR, Hess JA. Attentive facework during instructional feedback: Key to perceiving mentorship and an optimal learning environment. Commun Educ 2008;57(3):312-332.

28. Trees AR, Kerssen-Griep J, Hess JA. Earning influence by communication respect: Facework’s contributions to effective instructional feedback. Commun Educ 2009;58(3):397-416.

29. Martin L, Mottet TP. The effect of instructor nonverbal immediacy behaviors and feedback sensitivity on Hispanic students’ affective learning outcomes in ninth-grade writing conferences. Commun Educ 2011;60(1):1-19.

30. Witt PL, Kerssen-Griep J. Instructional Feedback I: The interaction of facework and immediacy on students’ perceptions of instructor credibility. Commun Educ 2011;60(1):75-94.

31. King PE, Schrodt P, Weisel JJ. The instructional feedback orientation scale: Conceptualizing and validating a new measure for assessing perceptions of instructional feedback. Commun Educ 2009;58(2):235-261.

32. Dannels DP, Housley Gaffney AL, Norris Martin K. Students’ talk about the climate of feedback interventions in the critique. Commun Educ 2011;60(1):95-114.

33. Johnson W. People in quandaries. The semantics of personal adjustment. New York: Harper & Row Publishers; 1946.

34. Johnson W. Speech and personality. In: Bryson L, ed. The communication of ideas. A series of addresses. New York: Cooper Square Publishers;1964;53-78.

35. Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, O’Brien MA, Johansen M, Grimshaw J, Oxman AD. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;6:CD000259. doi: 10.1002/14651858.

CD000259.pub3.

36. Shute VJ. Focus on formative feedback. Rev Educ Res 2008;78:153-189.

37. Jussim L. Social perception and social reality: A reflection-construction model. Psychol Rev 1991;98(1):54-73.

38. Ferguson MJ, Bargh JA. How social perception can automatically influence behavior. Trends Cogn Sci 2004;8(1):33-39.

39. Swann WB. Quest for accuracy inperson perception: a matter of pragmatics. Psychol Rev 1984 9 (1): 457-477.

40. Higgins ET, Bargh JA Social cognition and social perception. Annu Rev Psychol 1987;38:369425.

41. Fiske ST. Social cognition and social perception. Annu Rev Psychol 1993;44:155-194.

42. Berry DS. Accuracy in social perception: contributions of facial and vocal information. Pers Soc Psycho 1991b;61:298-307.

43. Bargh JA, Chartrand TL. The unbearable automaticity of being. Am Psychol 1999;54(7):462479.

44. Bargh JA, Ferguson JM. Beyond behaviorism: On the automaticity of higher mental processes. Psychol Bull 2000;126(6):925-945.

45. Shrauger JS. Responses to evaluation as a function of initial self-perceptions. Psychol Bull 1975;82(4):S81-S96.

46. Lasswell HD. Describing the effects of communications. In: Smith BL, Lasswell HD, Casey RD, editors. Propaganda, communication, and public opinion. A comprehensive reference guide. Princeton: Princeton University Press; 1946.

47. Ilgen DR, Barnes Farrell JL, McKellin DB. Performance appraisal process research in the 1980s: What has it contributed to appraisals in use? Organ Behav Hum Decis Process 1993;54(3):321-68.

48. Gallagher JJ. Teaching and learning: new models. Ann Rev Psychol 1994;45:171-95.

49. Mory EH. Feedback research revisited. In: Jonassen DH, editor. Handbook of Research on Educational Communications and Technology. Mahwah: Lawrence Erlbaum Associates; 2004.

161 Conclusion and discussion

50. Smither JW, London M, Reilly RR. Does performance improve following multisource feedback? A theoretical model, meta-analysis, and review of empirical findings. Pers Psychol 2005;58:33-66.

51. Smidt A, Balandin S, Sigafoos J, Reed VA. The Kirkpatrick model: A useful tool for evaluating training outcomes. J Intellect Dev Disabil 2009;34(3):266–274.

52. Sargeant J, Borduas F, Sales A, Klein D, Lynn B, Stenerson H. CPD and KT: Models used and opportunities for synergy. J Contin Educ Health Prof 2011;31(3):167–173.

53. Alliger GM, Janak EA. Kirkpatrick’s levels of training criteria: Thirty years later. Pers Psychol 1989;42:331-342.

54. Bates R. A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence. Eval Program Plann 2004;27:341–347.

55. Stokking KM. Levels of Evaluation: Kirkpatrick, Kaufman and Keller, and Beyond. Hum Resour Dev Q 1996;7(2):179-183.

56. Kaufman R, Keller JM. Levels of Evaluation: Beyond Kirkpatrick. Hum Resour Dev Q 1994;5(4):371-380.

57. Boatman DF, Miglioretti DL, Eberwein C, Alidoost M, Reich SG. How accurate are bedside hearing tests? Neurology 2007;68:1311-1314.

58. Bagai A, Thavendiranathan P, Detsky AS. Does this patient have hearing impairment? JAMA 2006;295(4):416-428.

59. Pearse JMS. Early days of the tuning fork. J Neurol Neurosurg Psychiatry 1998;65:728.

60. Turner JS. The ear and auditory system. In: Walker HK, Hall WD, Hurst JW, editors. The clinical methods . The history, physical and laboratory examinations. 3rd ed. Boston: Butterworth; 1990. p. 609-611.

61. Hall CM, Croutch C. Tuning forks revisited: Theory, use, and interpretation of results. Hearing Review 2010;17(3):26-30.

62. Kruisheer EM, van de Ridder JMM, Meeuwsen A. Evaluatie van knelpunten bij het vaardigheidonderwijs van het Otologisch onderzoek. [Students’ perception of difficulties in learning tests for detecting hearing loss.] Utrecht, University Medical Center Utrecht, 2006 (unpublished report).

63. Schmuckler MA. What is ecological validity? A dimensional analysis. Infancy 2001:2;419436.

64. Rossi PH, Freeman HE. Evaluation a systematic approach. New York Sage publications 1993.

65. Delgado-Rodriguez M. Systematic reviews of meta-analyses: applications and limitations. J Epidemiol Cummun H 2006;60:90-2.

66. Sharpe D. Of apples and oranges, file drawers and garbage: why validity issues in metaanalysis will not go away. Clin Psychol Rev 1997;17(8):881-901.

67. Chambers, EA. An introduction to meta-analysis with articles from the Journal of Educational Research (1992-2002). J Educ Res 2004;98(1):35-44.

68. Callahan CA, Hojat M, Gonnella JS. Volunteer bias in medical education research: an empirical study of over three decades of longitudinal data. Med Educ 2007;41:746–753.

69. Ericsson KA, Charness N, Feltovich PJ, Hoffman RR. The Cambridge handbook of expertise and expert Performance. New York: Cambridge University 2006.

70. Greller MM, Herold DM. Sources of feedback: A preliminary investigation. Organ Behav Hum Perform 1975;33(1):244-256.

71. Ilgen DR, Fisher, CD, Taylor MS. Consequences of individual feedback on behavior in organizations. J Appl Psychol 1979;64(4):349-371.

72. Larson JR. The performance feedback process: A preliminary model. Organ Behav Hum Perform 1984;33(1):42-76.

73. Gregory D, Johnston R, Pratt G, WattsM, Whatmore S. The Dictionary of Human Geography [e-book]. Hoboken: Wiley; 2009 [cited 2015 Feb 11]. Available from: Ebook Library Utrecht University.

Chapter 7

162

74. Ambrose L, Moscinski P. The power of feedback. Healthc Exec 2002;17(5):56-57.

75. Kaprielian VS, Gradison M. Effective use of feedback. Fam Med 1998;30:406-407.

76. Moorhead R, Maguire P, Thoo SL. Giving feedback to learners in the practice. Aust Fam Physician 2004;33:691-694.

77. Lucas JH, Stallworth JR. Providing difficult feedback: TIPS for the problem learner. Fam Med 2003;35:544-546.

78. Gordon J. ABC of learning and teaching in medicine. One to one teaching and feedback.

BMJ 2003;326:543-545.

79. Hesketh EA, Laidlaw JM. Developing the teaching instinct: Feedback. Med Teach 2002;24:245-248.

80. Anders R. Feedback without Fear. AORN J 2000;74:882-884.

81. Wood BP. Feedback a key feature of medical training. Radiology 2000;21:S17-19.

82. Adam Z. It’s good to talk. Nurs Times 1999;95:66-67.

83. Geis GL. Formative feedback: The receiving side. Performance and instruction journal 1986;25(5):3-6.

84. Branch WT, Paranjape A. Feedback and reflection: Teaching methods for clinical settings.

Acad Med 2002;77:1185-1188.

85. Fedor DB, Davis WD, Maslyn JM, Mathieson K. Performance improvement efforts in response to negative feedback: the roles of source power and recipient self-esteem. J Manage 27 2001;27:79–97.

86. Gabsdil M. Clarification in Spoken Dialogue Systems. AAAI Technical Report SS-03-06.

2003. Retrieved March 8, 2013. http://aaaipress.org/Papers/Symposia/Spring/2003/SS-0306/SS03-06-006.pdf 87. Veloski J, Boex JR, Grasberger MJ, Evans A, Wolfson DB. Systematic review of the literature on assessment, feedback and physicians’ clinical performance: BEME Guide No. 7. Med Teach 2006;28(2):117-128.

88. Archer JC. State of science in health professional education: Effective feedback. Med Educ 2010;44:101-108.

Appendices A Overview of studies included in Chapter 3 B Supplementary Material with Chapter 3

1966-1992

-

1950-1994

1950-1991

2. BangertDrowns et al; 1991

3. Bommer et al; 1995

4. Conway et al; 1997

B. Time span literature

1. Azevedo et al; 1995

A. Author, year

ESr

ESr (corrected mean correlations)

ESd

ESd weighted mean effect sizes (wmes)

C. Outcome measure

Phase B

Effect

Effect

E. Target phase in the FB process

Refs=159 Phase B (177 samples; 281 coefficients) Ss=28999

Refs=40

Refs=40 ESs=58

Refs=22 ESs=43

D. Included studies (Refs), Effect Sizes (ESs), or subjects (Ss) G. Participants (Ss) and tasks

Assessed the relationship between objective and subjective performance measures, and estimates the population correlation between these measures.

Examine the psychometric properties of subordinate, supervisor, peer and self- ratings of job performance.

Effects of mediated intentional feedback on post treatment performance on achievement tests

Ss: Subordinate, supervisor, peer and self-ratings of job performance Tasks: on-the-job performance ratings

Ss: Sales people and others.

Tasks: n.m.

I. Independent variables

J. Variables not influencing the process and effect of feedback according to this specific study

Moderators: Job type, Objective measures, Content rating method, and rating format.

-Job type

-Counts for grade -Subject assignment -Study duration -Grade level -Subject-matter content -Source of report -Year of report -Error rate during instruction

a) Strength of the Rating source, job relationship between type, rating type, and and within raters dimension type. b) Correlation of inter- and intra rater reliability

Correlations between objective and subjective measure of performance

Correct answer, repeat until correct, right/wrong, explanation

Comparison of Computer presented experimental group feedback and control group on immediate post test and delayed post test data.

H. Dependent variable(s)

Ss: Students (college, Comparison of elementary, economics, nursing, experimental group medicine) and control group Tasks: Tests-like events based on texts, testing, programmed instruction, computer assisted instruction

Judge the relative Ss: Primary, secondary and effectiveness of college students feedback in general Tasks: not mentioned (n.m.) and based on various computer-based instruction typologies.

META-ANALYSIS

F. Research questions or aim of study

Table S1. Overview of study characteristics of included meta-analyses (n=22) and reviews (n=24) in alphabetical order

Appendices

164

ESr

1956-1986

7. Harris et al; 1988

Refs=54 (70 correlations)

Refs=30 ESs=217 Ss=1,058

ESg

6. Fukkink et 1973-2009 al; 2011

D. Included studies (Refs), Effect Sizes (ESs), or subjects (Ss)

Refs=16 ESr (Corrected mean (53 samples: 25 published effect size rpb) 28 unpublished)

C. Outcome measure

5. Ford et al; <-1986 1986

B. Time span literature

Phase B

Effect

Phase B

E. Target phase in the FB process Ss: Among them are: Firefighters, police officers, production workers, bank tellers, nurses, clerical workers Tasks: n.m.

G. Participants (Ss) and tasks

What is the average Ss: n.m. correlation between Tasks: n.m. self-supervisor, selfpeer and peersupervisor ratings?

Can the variance be explained by moderators such as: job type, rating format and rating scale?

1. What is the effect Ss: Students and professionals of VF interventions on Task: application of a skill the interaction skills of professionals?

2. Which methodological and pedagogical characteristics correlate systematically with the results of experimental studies into VF?

Examine the relation between effect sizes for objective and subjective ratings of performance, and investigating differences in race effect sizes in different types of objective criteria.

F. Research questions or aim of study

J. Variables not influencing the process and effect of feedback according to this specific study

Moderators: -Type of outcome measure (positive â&#x20AC;&#x201C; negative); -Type of outcome measure (micro molar); -Observation of target skills (without coding form - with coding form)

Video feedback intervention

-Rating format -Rating scale

- Design - Randomisation - Number participants - Professional level - Extra instruction

Ratee races.

Moderators: Type of criterion: performance indicator, cognitive, and absenteeism criterion

I. Independent variables

Correlations between Moderators: rating self supervisor, peer- format, rating scale, supervisor and self- and job type peer ratings

Interaction skills (verbal, non-verbal, and paralingual behavior), and for receptive, informative, and relational skills

Correlations between objective and subjective measures of performance

H. Dependent variable(s)

Appendices

A. Author, year

165

Standardized variance components

10. Karelaia 1955-1999 et al; 2008

1980-2006

9. Huang; 2009

C. Outcome measure

ESr

B. Time span literature

8. Heneman; 1927-1983 1986

A. Author, year

Phase A

Phase B

E. Target phase in the FB process

Refs=86 Effect 249 indep datasets

Refs=50 130 indep. Data sets

Refs=23 Ss=3178

D. Included studies (Refs), Effect Sizes (ESs), or subjects (Ss) G. Participants (Ss) and tasks

Examine the effect of Ss: n.m. experience trough Tasks: judgement tasks learning across experimental trials in studies with multiple blocks of judgments Aim: quantify learning and compare effects of different types of feedback

Conduct a metaSs: n.m. analysis of Tasks: paper and pencil tests, generalizability non- paper and pencil tests studies based on performance assessment to assess task-associated variation.

Examining the Ss: n.m. convergent validity Tasks: n.m. between supervisory ratings and objective measures of performance.

F. Research questions or aim of study

Moderators: Subject area and Research design

Moderators: rating method and rating format

I. Independent variables

Learning: Type of feedback Judgemental Noise in context achievement Judgemental consistentcy Matching Linear cognitive ability Residual correlation

Task-sampling variability: Person-task interaction Person-rater interaction

Correlation between objective and subjective measures of performance

H. Dependent variable(s)

Appendices

166 -

Moderators: -Scoring strategy -Assessment method -Publication status

-Rating format

J. Variables not influencing the process and effect of feedback according to this specific study

11. Kingston 1988-2010 et al; 2011

B. Time span literature

Esd (wmes)

C. Outcome measure

Refs=13 ESs=42

D. Included studies (Refs), Effect Sizes (ESs), or subjects (Ss) Effect

E. Target phase in the FB process G. Participants (Ss) and tasks

1. What is the Ss: from an academic K-12th average effect size of grade setting. formative assessment on educational achievement (EA)?

2. Is the average effect size of formative assessment on EA moderated by grade or content area?

3. Is the average effect size of formative assessment on EA moderated by specific formative assessment practices?

F. Research questions or aim of study

I. Independent variables

J. Variables not influencing the process and effect of feedback according to this specific study

Student achievement Using formative Moderator: assessment practices Grade level such as Professional Development; CurriculumEmbedded Assessments; Computer-Based Formative System; Specific Use of Student Feedback; Other (Classroom assessment activities, Student reflection, Assessment conversations); Moderators: Content Area Treatment Type

H. Dependent variable(s)

Appendices

A. Author, year

167

Refs=53 ESs=52

As much as possible

14. Kulik et al; 1998

ESd

ESr Refs=12 (weighted mean (25 samples) correlation)

1960-1989

Refs=131 ESs=607 Ss=12652

D. Included studies (Refs), Effect Sizes (ESs), or subjects (Ss)

13. Kraiger et al; 1990

C. Outcome measure

ESd

B. Time span literature

12. Kluger et <-1992 al; 1996

A. Author, year

Effect

Phase B

Effect

E. Target phase in the FB process

Do changes in immediate and delayed feedback have small, medium or large effects on learner performance?

H. Dependent variable(s)

Ss: Students (college, military, senior high) Tasks: Classroom quizzes, acquisition on test content, list learning tasks

Feedback intervention (FI) sign -Correct-incorrect -Attainment level -Normative information -Norms -Written FI -Graphical FI -Public FI -Group FI -Task novelty -Time constraint -Time duration -Creativity -Quantity-quality -Performance rating -Transfer measure -Latency measure -Reaction time -Knowledge task -Vigilance task -Rewards-punishment -Experimental control -Lab-field

J. Variables not influencing the process and effect of feedback according to this specific study

-Number of different tests or lists to be learned -Number of trials or times each test was taken -Subject assignment -Duration of treatment -Class level -Year of the report -Source of the study

Ratee race effects in -Job-knowledge job performance ratings and job knowledge ratings

Feedback interventions not confounded with other manipulations

I. Independent variables

a) Comparison Immediate and between experimental delayed feedback and control; b) Comparison between subjects receiving immediate and subjects receiving delayed feedback.

Correlations between objective and subjective measure of performance

Ss: n.m.

Comparison between Tasks: reading errors, puzzles, control and (quasi) memory retention, reaction time, experimental group motor performance, maintenance jobs

G. Participants (Ss) and tasks

Comparing Ss: n.m. supervisory ratings to Tasks: n.m. objective indices of job knowledge and job performance for black and white employees.

To quantify the variability of feedback effects and to rule out artifact-based explanations to feedback interventions effect variability.

F. Research questions or aim of study

Appendices

168

ESd

ESd

16. Lyster et ≥ 1980 al; 2010

C. Outcome measure

15. Li; 2010 1988-2008

B. Time span literature

Effect

E. Target phase in the FB process

Refs=15 Effect ESs= 43 betweengroup contrasts; 33 within group contrasts; Ss=827

Refs=33 Ss=1773

D. Included studies (Refs), Effect Sizes (ESs), or subjects (Ss)

1. How effective is corrective feedback (CF) on target language development in L2 classrooms?

2.To what extent does CF effectiveness vary according to the following variables? a) Types of CF; b) Types and timing of outcome measures; c) Instructional setting (Second language (SL) vs. Foreign Language (FL) classroom): d) Length of treatment (brief vs. short-to-medium vs. long), and e) Learners’ age.

What is overall effect of corrective feedback on second language (L2) learning? Do different feedback types impact L2 learning differently?

Does the effectiveness of corrective feedback persist over time?

What are the moderator variables for the effectiveness of corrective feedback?

F. Research questions or aim of study

Ss: Students Task: Language related, such as: regular and irregular past tense, question forms and about indefinite articles.

Ss: Adults; In three studies children between 10-12 were included.

Tasks: learning Foreign or Second Language

G. Participants (Ss) and tasks

1. Free constructedresponse measures; 2. Constrained constructed-response measures; 3. Selected-response measures; 4. Metalinguistic judgments;

Comparisons between experimental and control. If an experimental and control group were lacking these were replaced by ‘alternative’ experimental and control groups

H. Dependent variable(s)

Instructional setting (second vs. foreign language classroom),

Oral corrective feedback Types of CF, Types of outcome measures, Treatment length, Learners’ age.

Time interval between performances

J. Variables not influencing the process and effect of feedback according to this specific study -Timing of posttest.

-Mode of delivery; -Outcome measure; -Publication type; -Target language; -Learners age, -Year of publication

I. Independent variables

Feedback; Feedback types; Moderators: research context; research setting Task type; Treatment length; Interlocuter type;

Appendices

A. Author, year

169

-

18. Norton; 1992

19. Pritchard 1987-> et al; 2008

1975-1984

B. Time span literature

17. Murphy et al; 1986

A. Author, year

ESd

ESr corrected for measurement and sampling error

Average ESd

C. Outcome measure

Refs=17 ESs=83

Refs=32 ESs=56 Ss=11327

Refs=111 (146 samples)

D. Included studies (Refs), Effect Sizes (ESs), or subjects (Ss)

Effect

Phase B

Phase B

E. Target phase in the FB process G. Participants (Ss) and tasks

Update the data on effectiveness of the Productivity Measurement and Enhancement system (ProMES), determine the effects over time, the effects in different settings, and focus on moderator variables that determine the degree of effectiveness.

Review via metaanalyses the issue of the validity of peer assessment, and to identify factors that may moderate the relationship between peer-assessment and other measures of performance.

Subjects: employees in cardboard manufacturing, traffic police, poultry processing, insurance sale, university research support, nurses psychiatric hospital Tasks: n.m.

Ss: Military service, college students, insurance agents, managers, factory workers.

Tasks: n.m.

Determine whether Ss: n.m. performance Tasks: n.m. appraisal research using paper case descriptions of people leads to outcomes systematically different from those resulting when raters directly or indirectly observe performance.

F. Research questions or aim of study

I. Independent variables

J. Variables not influencing the process and effect of feedback according to this specific study

Comparison of productivity in the workplace under baseline and after feedback.

Correlation between subjective measure of performance by a peer and an objective measure of performance

ProMES system Moderators: degree of match, quality of feedback, changes in feedback system, interdependence, centralization.

-Prior feedback -Trust -Number of personnel in unit -Turnover -Complexity -Management support -Stability of organizationâ&#x20AC;&#x2122;s environment trust

Moderators: Purpose -Occupation of using, time, -Dimension occupation, dimension, criterion type

Comparison between Rating direct or -Rater and ratee characteristics ratings of paper cases indirect observations -Rating scale format and observations versus written -Training effects performance -Performance level vignettes

H. Dependent variable(s)

Appendices

170

<-1991

21. Travlos et al; 1995

22. <-1995 Viswesvaran et al; 1996

1974-1992

20. Tang et al; 1995

B. Time span literature

R

ESd (wmes)

ESd

C. Outcome measure

Refs=215

Refs=17 ESs=60

Refs=50 ESs=256

D. Included studies (Refs), Effect Sizes (ESs), or subjects (Ss)

Phase B

Effect

Phase D

E. Target phase in the FB process Ss: Only outcomes on 10-12 grade and college students are reported.

Tasks: Puzzles, drawing

G. Participants (Ss) and tasks

Investigate the Ss: reliability of peer and supervisory ratings of various jobperformance dimensions; Compare inter rater agreement and intra rater consistency in the reliability of ratings.

To assess the effects Ss: n.m. of temporal locus of Tasks: motor tasks.

Knowledge of Results (KR) on learning and performance of motor tasks. the effect of KR-delay and post-KR interval on acquisition and retention phase and the effect of interpolated activity during these intervals on acquisition and retention in motor primary tasks.

Determine the conditions in which an over justification effect occurs.

F. Research questions or aim of study

J. Variables not influencing the process and effect of feedback according to this specific study

Duration of the time-span of the (KR) delay and Post-KR interval.

Verbal and motor interpolated activities during KR-delay and post KR-interval.

-

-Retention -Acquisition with motor interpolated activity -Retention with motor interpolated

Reward: paper certificate, praise, material rewards, bonus points, money

I. Independent variables

Strength of the 10 job dimensions: relationship between e.g. quality, and within raters leadership.

Communication competence, effort, job knowledge

Comparison between groups with long and short KR-delay and post KRâ&#x20AC;&#x201C;interval during acquisition and retention of motor learning tasks.

Comparison between groups with verbal and motor interpolated activities during KR-delay and post KR-interval during acquisition and retention of motor learning tasks.

Comparison of rewarded group with control group on the over justification effect. This is measured by time spend on a task and questionnaires

H. Dependent variable(s)

Appendices

A. Author, year

171

ESd

Narrative

25. Clariana et al; 2005

26. Claus et al; 2009

1885-2005

Narrative

24. Cheraghi-Sohi et al; 2008

C. Outcome measure

Narrative

B. Time span literature

-

23. Balzer et al; 1989

A. Author, year

Refs=56

Refs=20 ESs=35

Refs=9

Refs=21

Phase ABD Effect

Effect

Effect

Effect

D. E. Included Target studies (Refs), phase in Effect the FB Sizes(ESs), process or subjects (Ss) G. Participants (Ss) and tasks

Assess and advance the conceptual understanding op Performance Management from a cross-border perspective.

Examine the relative effects of multiple-try feedback (MTF) to other forms of feedback in higher order and verbatim posttest outcomes.

Assess the efficacy of the feedback of (a) patient assessments, (b) brief training, (c) interventions combining both feedback and brief training, on the interpersonal skills of primary care physicians.

Evaluating empirical studies that test efficacy of cognitive Feedback (CFB). Does CFB influence performance? Which specific CFB component influences performance?

Outcomes on judgment tasks on the level of reaction, behavior and result.

H. Dependent variable(s)

Ss: Employee’s and employers form a crossborder perspective Tasks: n.m.

Performance dimensions Purpose of Performance Assessment; Motivation Self-efficacy Performance

Ss: Mainly graduate and Higher order- learning undergraduate students and lower order-learning Tasks: print-based texts outcomes and questions, Multiple choice questions, Computer based instruction, lectures and test reviews

General practitioners and a Interpersonal skills of mixture of primary care primary care physicians. physicians, trainee physicians, medical residents and experienced practicing physicians.

Tasks: First contact and on-going care to patients.

Ss: Physicians, students, negotiators, interviewers, volunteers, engineers Tasks: judgment tasks

QUALITATIVE REVIEWS

F. Research questions or aim of study

Appendices

I. Independent variables

J. Variables not influencing the process and effect of feedback according to this specific study

Culture Rater’s cultural background Ratee’s cultural background Feedback

Comparison between MTF intervention and other feedback interventions: correct response feedback (KCR), knowledge of response (KR), delayed feedback (DF)

-

-

Patient based feedback, Brief training brief training and their combination. (trainings such as: communication skills, interpersonal skills to increase trust, medical interviewing, and increasing awareness of patient agenda)

Cognitive feedback (CFB) -Cognitive information; -Functional validity information.

172

Refs=42

Narrative

Refs=7

29.Hepplestone et 2001-2010 al; 2011

Narrative

Phase D

Phase B

Effect

D. E. Included Target studies (Refs), phase in Effect the FB Sizes(ESs), process or subjects (Ss)

Narrative: Refs=40 aspects of utility index are evaluated.

1980-2009

C. Outcome measure

28. Hamilton et al; 1998-2004 2007

27 Craig, 2010

B. Time span literature

Ss: Nurses, midwives, professionals form health care management and medicine Tasks: n.m.

Ss: Postgraduate (PG) medical staff ranging from interns (first PG year) to senior EM residents, Task: direct observation in emergency. department

G. Participants (Ss) and tasks

Aims to address the comment by Ss: Students Burke (2009, 42) in which the author claims “it appears that we currently have a blind spot in relation to strategies for students making effective use of feedback” and supports Whitelock (2009, 199).90 91

Explore evidence relating to the use of performance assessment in health care professionals; Examine practice of performance assessment in health care professionals in Northern Ireland

Describes characteristics of published models that have been applied to postgraduate medicine (PGM) in Emergency Departments (EDs).

Identify other models of direct observation (DO) that haven been applied to PGM in Emergency Medicine (EM).

Develop recommendations for practice of DO in PGM in EM. Suggest for areas for further research.

F. Research questions or aim of study

-

Student engagement with Technological feedback interventions used by tutors to encourage students to engage with received feedback 1 Electronically published FB 2 Electronically produced FB 3 Adaptive release of grades 4 Computer assisted assessment 5 Peer assessment

Utility of assessment instruments

Reliability Validity Educational impact Cost-effectiveness Acceptability

J. Variables not influencing the process and effect of feedback according to this specific study -

I. Independent variables

Outcome data on level Direct observation 2-4 of Kirkpatrick’s model: 2 (Learning) Residents, 3 (Behavior) Patients and 4 (Results) EDs.

H. Dependent variable(s)

Appendices

A. Author, year

173

Refs=24

% studies reporting progress

34. McAfee et al; 1989

1971-1987

Refs=57

Refs=22 Ss=57775

Refs=118 (88 comparisons)

Refs=31

Effect

Phase B Effect

Phase B

Effect

Effect

D. E. Included Target studies (Refs), phase in Effect the FB Sizes(ESs), process or subjects (Ss)

33. Jonsson et al; No time limit narrative 2007 set

ESd

32. Jawahar et al; 1997

Narrative

C. Outcome measure

% increase / decrease of risk between intervention and control study

-

B. Time span literature

31. Jamtvedt et al; <-2006 2006

30. Jaehnig et al; 2007

A. Author, year G. Participants (Ss) and tasks

Ss: Health care professionals responsible for patient care Tasks: n.m.

I. Independent variables

Reliability of scoring Validity of judgments of performance assessments Promote learning

Comparison of performance appraisal ratings on performance appraisal purpose(PAP) effect

-

-Purpose of rating (decisionperformance) -Type of rating scale

-Type of intervention -Complexity of the targeted behavior -Seriousness of the outcome -Study quality

-No feedback -Knowledge of Results (KR) -Knowledge of correct Responding (KCR) -Review feedback -Extra instructional Consequences

J. Variables not influencing the process and effect of feedback according to this specific study

Incentives and feedback -

Use of rubrics

Purpose of performance appraisals (administrative or research), research setting, and measurement

Objective measured Audit and feedback professional practice and healthcare outcomes.

Overall impact on Feedback criterion test performance Elaboration feedback Delayed feedback

H. Dependent variable(s)

Investigate the use of incentives Ss: Employees: weavers, Safety conditions and (positive reinforcement) and polices, coal mine workers, workplace accidents feedback for safety improvement. metal fabrication workers Tasks: earplug use, eye accidents rate, bodily injuries

1. Does the use of rubrics Ss: n.m. enhance the reliability of scoring? Tasks: n.m.

2. Can rubrics facilitate valid judgment of performance assessments?

3. Does the use of rubrics promote learning and/or improve instruction?

Does the purpose of appraisal, Ss: Students and such as for administrative professional raters purposes, research, feedback or Tasks: n.m. employee development, determine a direct effect on appraisal leniency or level of performance ratings.

Assess the effect of audit and feedback on the practice of healthcare professionals and patient outcome

What impact do different types of Ss: n.m. feedback have on the Tasks: programmed effectiveness of programmed instruction instruction?

F. Research questions or aim of study

Appendices

174

â&#x2030;¤ 2010

1994-2004

1966-2004

35. Miller et al; 2010

36. Rout et al; 2007

37. Rowe et al; 2005

B. Time span literature

Narrative Low effect: <10% progress; Large effect: >25% progress

Narrative

Narrative

C. Outcome measure

Refs=11

Refs=56 (referring to 52 different peer review situations)

Refs=16

Effect

Phase BCD Effect

Effect

D. E. Included Target studies (Refs), phase in Effect the FB Sizes(ESs), process or subjects (Ss)

Which interventions are most effective (or cost effective)?

Not education related interventions are left out: Integration of services, community participation, drugs program, telemedicine, job aids, economic intervention, and community case management.

Investigate and evaluate the process and outcomes of the assessment interview, and to evaluate the feasibility of introducing peer review of the clinical assessment interview in acute clinical settings

What is the evidence that workplace based assessment influences physician education and performance?

F. Research questions or aim of study

Use of peer review in clinical assessment interview

Performance improvement

H. Dependent variable(s)

Ss: Health-care workers in Health-worker low- and middle income performance outcomes countries Tasks: n.m.

Ss: preregistration nurses and midwives Tasks: peer review of clinical assessment interview

Ss: doctors from all levels of training and from different specialties.

Task: n.m.

G. Participants (Ss) and tasks

Appendices

A. Author, year

Could be identified: the included studies missed vital information about the use of peer review Could not be identified -Printed guidelines -Computer based training -Education intervention (training, seminars) -Combination: education and management approach -Self-assessment -Distance learning

Could be identified: the included studies missed vital information about the use of peer review

Feedback interventions: e.g. guidelines, supervision, selfassessments, audit and feedback, trainings

J. Variables not influencing the process and effect of feedback according to this specific study Mini-clinical evaluation exercise Direct observation of procedural skills Multiple assessment methods

I. Independent variables

Multisource feedback Mini-clinical evaluation exercise Direct observation of procedural skills Multiple assessment methods

175

B. Time span literature

2002-2008

-

-

A. Author, year

38. Sasanguie et al; 2011

39. Shute; 2008

40. Smith; 1986

Refs=141

Refs=18

F. Research questions or aim of study

G. Participants (Ss) and tasks

Phase B

Phase D Effect

Influence of Rater training (lecture, group discussion practice and feedback) on rating accuracy and specific rating errors.

FR’s Goal orientation FR’s Ability( to correct answers) FR’s Uncertainty FR’s Cognitive load FR’s Use of task strategies FR’s Performance FR’s Response certitude

Lecture-student Relationship; Learning environment; Student’s learning strategy

H. Dependent variable(s)

Ss: Students, army officers, Leniency error, halo error, and rating accuracy. managers, supervisory engineers, supervisors.

Tasks: Rating of hypothetical job candidates, performances of employees, performances of college instructors, vignettes of hypothetical 1st line supervisors

Present findings from an Ss: n.m. extensive literature review of Tasks: n.m. feedback to gain better understanding of features, functions , interactions and links to learning.

Apply the findings form literature review to create a set of guidelines relating to formative feedback.

Phase BC What are the (educational) Subjects: n.m. arguments with respect to the Tasks: n.m. segregation or combination of supportive and evaluative roles of lecturers in higher education?

Does disentangling these roles improve the lecturer–student relationship?

Does disentangling the teacher’s roles result in a safer learning environment for the students?

Will the students learn more ‘deeply’ when they do not know the examiner, and thus do not know exactly what to expect from the examination?

D. E. Included Target studies (Refs), phase in Effect the FB Sizes(ESs), process or subjects (Ss)

% studies Refs=24 reporting progress (% calcu-lated by JMMvdR)

Narrative

Narrative

C. Outcome measure

Appendices

I. Independent variables

J. Variables not influencing the process and effect of feedback according to this specific study

Normative feedback

Content of rater training (RET, PDT, and PST) and the method of rater training (lectures, discussion, practice and feedback)

Formative feedback Feedback complexity Feedback specificity Feedback timing

Segregation or combination of supportive and evaluative roles of lectures.

176

Narrative

% of change measured from median (median effect)

Narrative

42. Steinman et al; 1975-2001 2006

43. Van Gennip et 1990-2007 al; 2009.

C. Outcome measure

41. Soumerai et al; 1970-1988 2006

B. Time span literature

Effect

Refs=15 Ss=1201

Effect

Refs=26 Effect clinical trials=33

Refs=44

D. E. Included Target studies (Refs), phase in Effect the FB Sizes(ESs), process or subjects (Ss)

To what extent are the outcomes of peer-assessment (PA) on learning related to interpersonal variables?

To what extent are the outcomes of PA on learning related to structural features of the PA format.

Assessment of interventions which are most effective at improving the prescribing of recommended antibiotics for acute infections.

What is known about the effectiveness and efficiency of approaches to improve prescribing practices in ambulatory settings?

F. Research questions or aim of study

Ss: university students, undergraduate students, lectures of business school.

Tasks: written papers, essays, oral presentations, quality of lessons, interface design, posters, etc.

Ss: Clinicians Tasks: Prescribing antibiotics

Ss: Physicians Tasks: Drugs prescribing

G. Participants (Ss) and tasks

J. Variables not influencing the process and effect of feedback according to this specific study -Printed educational materials, -Reports of patientspecific lists of prescribed materials, -Group education, -Clinical pharmacy services -Trial design -Patient sample size -Secular trend in antibiotic use -Baseline rate of recommended antibiotic use -Intervention frequency -Clinician education -Control group -Specific disease target -Children targeted Effect of Value diversity and interdependence could not be identified, due to lacking information.

I. Independent variables

Feedback methods: e.g. printed educational materials, group education, feedback of physician-specific prescribing patterns.

Quality Improvement strategies: clinician education, education and audit and feedback, audit and feedback,

Peer assessment Interpersonal variables (psychological safety, trust, value diversity, interdepence) Structural features of peer assessment format

Prescribing behavior

Quality of antibiotic selection for acute outpatient infections. This is operationalized by measuring: adherence to prescribing guidelines, % of total prescriptions for recommended prescription, changes in type of prescriptions.

Learning (objective learning benefits, learning benefits as perceived by students, beliefs)

H. Dependent variable(s)

Appendices

A. Author, year

177

% studies reporting progress

Narrative

1966-2003

44. Veloski et al; 2006

45. Wanguri; 1995 1980-1990

C. Outcome measure

B. Time span literature

A. Author, year

Refs=113

Refs=220

Phase BCD

Effect

D. E. Included Target studies (Refs), phase in Effect the FB Sizes(ESs), process or subjects (Ss)

Review and integrate crossdisciplinary empirical research on performance appraisals, evaluations and feedback

What are the features and characteristics of feedback that influence physicians’ clinical performance?

F. Research questions or aim of study

Physician’s clinical performance: medical records, billing records, test ordering behavior

H. Dependent variable(s)

Feedback alone and feedback strategies combined with other interventions

I. Independent variables

Ss: Managers, civil service No specific interventions Feedback, appraisals and personnel, faculty evaluations university, air force officers, nurses, employee insurance company

Ss: Physicians, house staff, other units of analysis: hospitals, patient populations Tasks: n.m.

G. Participants (Ss) and tasks

Appendices

178 -Participating in instrument (BARS) construction does not influence cognitive complexity; -Feedback providers do not perceive consequences for themselves after conducting thorough appraisal; -No differences in employee attitudes towards the overall satisfaction and consequences between trait-rating scales and performance standard; differences;

-Physicians’ involvement in design feedback system.

-Type of standard.

-Detailedness feedback.

-Feedback reports made public -Written/verbal feedback

J. Variables not influencing the process and effect of feedback according to this specific study

46. Woehr et al; 1994

1949-1992

B. Time span literature

ESd

C. Outcome measure

ESs=71

Refs=29 (26 published articles) Phase B

D. E. Included Target studies (Refs), phase in Effect the FB Sizes(ESs), process or subjects (Ss) Providing a preliminary quantitative review of the effectiveness of rater training across four rater training approaches

F. Research questions or aim of study

H. Dependent variable(s)

Ss: n.m.

Comparison between Tasks: Performance ratings experimental and control group on halo error, leniency error, rating accuracy and observational accuracy

G. Participants (Ss) and tasks

Appendices

A. Author, year I. Independent variables

J. Variables not influencing the process and effect of feedback according to this specific study

(combinations of) four training approaches: Rater error training (RET), Performance dimension training (PDT), Frame of reference training (FOR), and Behavioral observation training (BOT)

179

SUPPLEMENTARY MATERIALS Introduction

Appendices

180

The article Variables that influence the process and outcome of feedback: a meta-review of findings provides an overview of variables that the literature suggests may influence the quality of the feedback process and its effect.

The number of referenced variables is so great that a complete structured overview would be exhausting. We chose to keep the article lean and readable and provide background materials in this supplementary document about the following topics: (I) Information about the feedback model, (II) description of the eight step approach used to select the variables and to synthesize the results, (III) explanation of the effect sizes used in meta-analysis, (IV) description of results in each phase of the feedback process, (V) description of the feedback results according the narrative review of Wanguri, and (VI) a list of abbreviations.

I Information about the feedback model For simplicity, we divided the feedback process into four phases. However a closer look reveals that each phase has sub-steps. In phase B, for example, the feedback provider (FP) (a) observes a performance, (b) interprets the performance, (c) unconsciously or consciously the FP compares the performance with a standard, (d) this interpretation results in a mark or score when the FP (e) rates the performance.

More than one step was placed in a phase to keep the model simple and useful. As a consequence, it is possible that an independent variable connected with a phase can also influence a dependent variable connected with the same phase. For example the quality of FP’s observation (focused, observed during task performance, took notes) might influence how the FP scores the performance (with doubt, deliberate, with clear examples in mind). In this example ‘observation’ is seen as independent variable belonging to phase B and influencing the dependent variable ‘scoring of performance’, also belonging to phase B. II Description of the eight step approach used to select the variables and synthesize results We took the following steps to select the variables from the 46 studies: a) Each study was categorized in one or more phases of the feedback process or the feedback effect. Therefore, we first determined the dependent variables or the outcome variables of the literature reviews and the meta-analyses (table 1 column H in the article). The descriptive model was used as a tool to categorize the studies (figure 3.1 in the article). b) The dependent, independent, moderator, mediator and confounding variables were determined that were used to answer the research question (table 1 column I in the article). The moderator, mediator and confounding variables elucidate the causal process by which the independent variable influences the dependent variable. The mediator and moderator variables give insight about aspects of the independent variable that contribute to the effect.1-2 Mediator variables are intervening variables that are necessary to complete a cause-effect link between the independent and dependent variable.1-2 According to MacKinnon, Krull and Lockwood a confounding variable is

III Explanation of the effect sizes used in meta-analyses Authors of meta-analyses use the effect sizes of primary studies to calculate the overall mean effect size of the meta-analysis. The effect sizes from the primary studies often receive additional codes. For example an effect size can be coded for (a) research type, a study is carried out in a laboratory versus field study; (b) for feedback timing, feedback was given immediate versus delayed; (c) for task type, feedback was provided on a physical task versus a memory task. Additional analysis are performed with the help of

181 Appendices

defined as: ‘a variable related to two factors of interest that falsely obscures or accentuates the relationship between them’.1 A moderator is an interaction variable that influences the direction, strength, or both of the relationship between an independent and dependent variable. Stratification by the moderator variable will show different strengths of relationships in subgroups between the independent and dependent variables. c) We assessed the unit of measurement for each study. In meta-analyses the effect sizes are the usual units of measurement. Effect size d (ESd) indicates the extent to which means differ. ESd=0.2 is considered a small, d=0.5 a medium, d=0.8 a large effect.

Effect size r (ESr) indicates the size of correlation effects between variables. ESr=0.1 is considered a small, r=0.3 a medium, and r=0.5 a large effect.3 Table 1 gives an overview of the effect sizes used in the meta-analysis.

In the literature reviews, units of measurements could be the number of studies that report a change, or percentages of studies in which effects were found. We computed percentages of Smith’s study.4 Narrative reviews just describe differences. We only reported variables which were clearly indicated as effective, but the reader should be cautious that we did not always find indications of the significance of these effects. d) We identified and selected those variables which have a reported impact on the feedback process and feedback effect. In meta-analyses we only considered variables with a significant effect. In the quantitative and narrative reviews only those variables that were clearly reported as influencing the direction of the effect are discussed. e) Variables that did not influence the feedback process and the feedback effect were not described but are mentioned in table 1 column J in the appendix. f) To detect which phase of the feedback process and feedback effect was influenced, we 1) considered which dependent variable was influenced and, 2) in which phase of the feedback process or the feedback effect this variable could be classified.

Variables which have an impact in one study but where impact is lacking in the other study are mentioned as well. These variables are marked with † in table 1 in the article. g) To create a comprehensive overview, we clustered variables with similar meanings or similar content. A description of each variable can be found in table 2 from the article. h) We were able to decide the direction of the influence from the variables that influenced the phases of the feedback process or effect. All variables are described in the Supplementary Materials (Section IV). Only variables with a clear direction receive attention in the article.

The results of the narrative review by Wanguri are often based on a single study so the evidence is not convincing.5 Results of this study are enclosed in the Supplementary Material Section V. Only when results of Wanguri’s study supported results of other studies they are included in the article.

these codes. Based on these codes the effect sizes used in the meta-analysis can be grouped on task type, on research type, feedback timing, etc. Now effect sizes on feedback timing can be compared, e.g., are effect sizes for immediate feedback different than effect sizes for delayed feedback? The additional analysis makes clear under which conditions or influences the overall mean effect size of feedback effects might become weaker, stronger, or stay the same. In table 1 in the article, the moderators are mentioned in column I and J. IV Description of results in each phase of the feedback process

Appendices

182

We first describe the main effect, i.e., what does the literature report on the effectiveness of providing feedback. All variables are then discussed in the order and under the phase where they appear in table 1 column E from the article.

Main effect of feedback Several studies described a main effect of feedback. The outcome variables ranged from learner characteristics, job performance, specific task performance, and physician and clinical tasks.

Feedback effect on learner’s characteristics Formative feedback, information intended to modify learner thinking or behavior to improve performance can reduce learner uncertainty about the level of performance.

Formative feedback can reduce the cognitive load of a novice learner and provide useful information to correct misconceptions or improper task strategies.46 Learner goal orientation can change from performing to learning due to formative feedback. Goal orientation may give the learner insight about the process that ability and skills can be developed through practice and that making mistakes is part of skill acquisition. Learners with a learning orientation show more positive effects than learners with a performance orientation. Learner’s goal orientation can change from formative feedback.46 Feedback effect on specific task, job or educational performances Kluger and DeNisi report that feedback has a moderate impact on performance improvement (d=0.41). However, 38% of the effect sizes are negative so feedback may decrease performance.16 A meta-analysis involving 83 field studies in jobs like sales, manufacturing, and service roles ( i.e., education, health care, military) showed that productivity under feedback is higher (d=1.16) than productivity at baseline. In this study feedback was a performance score on a list of organizational objectives. In a feedback meeting the organization discussed the results. In this field study 18% (n=15) effect sizes are negative.24 Feedback increases job satisfaction and job performance.5 Based on a study of O’Reilly and Anderson, Wanguri concludes that the relationship between feedback and jobsatisfaction is stronger than between feedback and job-performance.6 The correlation between job satisfaction and feedback after it is conveyed is stronger than the correlation between job performance and feedback.5 Feedback and incentives positively influence safety related performance in workplaces.

All included studies reported safety enhancement and accident reduction.9 In computer based instruction feedback had a large effect (d=0.80) on feedback recipients (FRs) who received feedback compared to controls who did not receive feedback.

Feedback on physicians’ and clinicians’ performances In clinical trials, 67% (n=22) of the studies providing feedback on clinician’s antibiotic prescribing positively changed the absolute volume of recommended antibiotics, or the percentage of patients treated with the recommend antibiotics.11 Receiving feedback positively changed clinician’s performance in 70% (n=29) of the studies.42 The evidence concerning the effect of patient based feedback on primary care physicians’ interpersonal skills is very limited. A review about the impact of feedback and brief training included only two studies about patient feedback. A study involving trainees showed a positive effect for patient feedback on patient satisfaction, but the study involving experienced physicians did not show an effect. The author’s explanation is that trainees are more flexible in adapting their behavior in line with the feedback compared to experienced clinicians.8 In many studies, receiving feedback is effective, and improves performance.9-12, 16, 19, 42 The impact of feedback is often small to moderate.19, 16, 21, 25 A: Effects of tasks, standards and first task performance on later phases of feedback process and effect We found influences on variables connected to the task performance (phase A), observation and interpretation (phase B), and the reception and interpretation of the feedback message (phase D) and the feedback effect. No influences were found on phase C (table 3.2 in the article).

Task variation When task variation is large, the assessment scores reflect not only students’ abilities but also task characteristics. More tasks are needed to reliably assess student performance. A meta-analysis on task-sampling variability in performance assessment showed that 12% of task variance resided in differential task difficulty among learners.13 Task complexity Feedback providers in the workplace agree more about observations of low (blue collar, non-managerial) complexity tasks (r=0.60) than high (white collar, managerial) complexity tasks (r=0.48).14 Murphy, Herr, Lockhart and Maguire found also differences in

183 Appendices

On delayed posttests the effect was still evident (d=0.35).12 A systematic review by Jaehnig and Miller on the effect of feedback on outcomes in programmed instruction showed similar results. From the 33 included studies, 19 studies had a feedback condition that was compared with a non-feedback condition as control group. In 11 studies the feedback condition was more effective than the condition without feedback.47 Cognitive feedback, i.e., task information, cognitive information, and FR’s perceptions of the relation between task and cognitive information, improves judgment tasks.10 Receiving and withdrawing corrective feedback was compared in second language acquisition (SLA). Receiving corrective feedback had a medium effect (d=0.64).25 Lyster also studied this topic and the outcome corroborates Li’s results. He found medium effect sizes (d=0.74) in between-group contrasts, and large effect sizes (d=0.91) in within-group results.7 Formative assessments (assessment for learning) show an increase in educational achievement (d=0.28) when a control group without formative assessment is compared with a group that received formative assessment.22

Appendices

184

task complexity. They compared groups that received evaluations based on observations with groups that received evaluations based on paper vignettes. They found that effect sizes for task complexity were higher when FPs had to base their observations on paper vignettes the effect (d=0.98) than observations based on real behavior (d=0.64).15 Feedback on complex tasks seems less effective than easy tasks. Effect sizes for tasks with high complexity are small (d=0.03). Tasks with low complexity that receive feedback have moderate effect sizes (d=0.55).16 Recently, Fukkink performed a meta-analysis on the use of Video Feedback (VF) and its effect on interaction skills of professionals and students. Task complexity (high specific versus general) moderates feedback effects. He compared micro versus molar skill tasks. A micro skill in his study is defined as a highly specific skill such as the number of questions asked during a conversation. A molar skill is a broader skill like the degree of empathy or responsiveness shown during a conversation. Effect sizes are smaller in micro skills (g=0.32), compared to molar skills (g=0.52).17 In line with Kluger and DeNisi is a study of Li about SLA, where task type was used as a moderator variable. Effect sizes in tasks characterized as ‘mechanical drill’ were high (d=0.89). Effect sizes in complex communication tasks have moderate to high impact (d=0.69). The mean effect sizes in the mechanical drill tasks was significant larger than the effect size generated by communicative activities.25 Clariana compared lower-order and higher-order tasks about whether or not Multiple Try Feedback (MTF) was more effective than such other feedback options as Delayed Feedback (DF), Correct Response Feedback (KCR), or Knowledge of Response Feedback (KR). Examples of lower-order tasks are remembering facts and stating knowledge, while higher order tasks involve laboratory tasks, trouble-shooting tasks, or tasks which needs knowledge application. After receiving MTF in computer based education, effect sizes are negative for lower-order outcomes (d=-0.22). Effect sizes are positive for tasks with higher-order learning outcomes (d=0.08) which shows that MTF is more effective than other types of feedback on these task types.18 Teamwork is essential when employees are interdependent with each other to achieve results. Teamwork requires interactions among employees to achieve results. The degree of interdependence moderates the effectiveness of feedback (r=-0.30). Higher interdependence yields less effective outcome feedback on tasks. In a context with high degrees of interdependence process feedback might be more effective than feedback on production outcome.24 Jamtvedt’s study, however, where task complexity was used as a moderator variable to explain variation in results across studies, no differences between groups were found.19 In a study about workplace productivity, productivity scores were compared at baseline and after feedback. The effect sizes for job types – blue collar, technical, academic and managerial- are high (d>1.00). However for highly routine jobs, such as clerical jobs effect sizes are much lower (d=0.27). It is not clear in these jobs if more frequent feedback should be provided. Perhaps due to the repetition and routine it is impossible to improve task strategies.24 Task nature Observations of communication tasks produced lower agreement in ratings between FPs (r=0.45) than observations of the quality of task performance (r=0.63). In peer observation, there is more agreement on ratings of FR’s rules and authority compliance (r=0.71) than on rating of FR’s job knowledge (r=0.33).20 Interrater reliabilities between

Table 1. Gives an overview of the different effect sizes which are used in the meta-analyses Used statistic

Authors Azevedo R, Bernhard RM; 1995 Bangert-Drowns RL, Kulik C-LC, Kulik JA, Morgan MT; 1991

d

d

The groups which are compared when Effect Size d or g was used.

The variables that correlated in case Effect Size r was used.

Compared experimental group receiving computer-presented feedback with a control group receiving no computer presented feedback.

Compared groups of students who received identical instruction, except that one received feedback on answers to questions and others didn’t.

Overall correlation in sample reporting multiple objective and/or subjective measures of performance.

Clariana RG, Koul R; 2005

-

Compared studies with Multiple Try Feedback with studies in which other feedback forms were used.

Conway JM, Huffcutt AI; 1997

r

Interrater reliabilities between various sources, including: subordinates, supervisors and peers. Correlations among the four different sources of ratings: subordinates, supervisors, peers and the ratees themselves.

Fukkink NG, Trienekens N, Kramer Hedges’g LJC; 2011

Compared experimental groups that received Video feedback intervention with control groups that received no intervention.

Ford JK, Kraiger K, Schechtman SL; 1986

rpb

Point-biserial correlations between ratee racee (1=White, 0=Black) and objective indices of performance and subjective ratings.

Harris MM, Schaubroeck J; 1988

ρ

Peer-supervisor, self-supervisor, and self-peer correlations.

Heneman RL; 1986

r

Relationship between supervisory ratings and results-oriented measures of performance (corrected mean correlation).

Huang C; 2009

σ2pt and σ2t

The effect size is the standardized variance component for tasks (σ2t) and for person-task interaction (σ2pt ).

Jawahar IM, Williams CR; 1997

d

Subtracting the mean of the control group (ratings obtained for research purposes) from the mean of the treatment group (ratings obtained for administrative purposes), and then divided by the standard deviation of the control group.

Karelaia N, Hogarth RM; 2008

r

ra= achievement index: the correlation between criterion and judgement; G= matching index: correlation between the prediction of two models; Re= Environment predictability: multiple correlations between the models of the environment and the judge; Rs=consistency with which the judge executes the decision rule: multiple correlations between the models of the environment and the judge; C=correlation between the error-terms of two models.

Kingston N, Nash B; 2010

d

The difference between the treatment and control group means on the posttest outcome variable divided by the pooled standard deviation. The difference between the group differences at pretest and posttest divided by the pooled standard deviation.

Kraiger K, Ford JK; 1990

r

Correlations between job performance, job knowledge, and supervisory ratings for white and black ratees.

Kulik JA, Kulik C-LC; 1988

-

Compared groups that received feedback immediately and after a delay interval.

185 Appendices

Bommer WH, Johnson JL, Rich GA, Podsakoff PM, Mackenzie SB; r 1995

Used statistic

Authors

Appendices

186

The groups which are compared when Effect Size d or g was used.

The variables that correlated in case Effect Size r was used.

Compared experimental groups that received feedback with control groups that received no feedback.

Compared the mean between experiment group and control group In cases where there were no pretest scores. Compared the mean change score of the experiment and control group in cases were both pretest and posttest were reported.

Compared mean difference between experimental and control groups at the time of posttests (between group).

Compared mean differences between pretests and posttests (within group).

Kluger AN, DeNisi A ; 1996

d

Li S; 2010

d

Lyster R, Saito K; 2010

d

Murphy KR, Herr BM, Lockhart MC, Maguire E; 1986

d

Compared the groups in which (in)direct observation was used to evaluate performance with groups in which raters read performance vignettes and then rate performance of hypothetical ratees.

Norton SM; 1992

r

Correlations on peer assessment on ability and performance.

Pritchard RD, Harrell MM, DiazGranados D, Guzman MJ; 2008

d

The mean difference in the overall effectiveness scores between the feedback and baseline periods. studies. This difference was then divided by the pooled standard deviation.

Tang S-H, Hall VC; 1995

d

Travlos AK, Pratt J; 1995

-

Viswesvaran C, Ones DS, Schmidt r FL; 1996

The difference between the means of the rewarded group and a nonrewarded control group, divided by the pooled standard deviation of this difference.

Compared “short” versus “long” time intervals Compared effects of interpolated activity in a specific KR length versus an empty KR time interval of the same length.

Each effect size was corrected according to the group’s sample size and the weighted means effect size (WMES).

Interrater and intrarater reliabilities between of supervisory, and peer and supervisory ratings of job-performance.

different types of FPs (supervisor and peer) are higher for non-managerial tasks, respectively r=0.54 and r=0.39, than for managerial tasks, r=0.44 and r=0.36.14 To determine whether task nature influenced the feedback effect, Kluger and DeNisi evaluated three moderator variables related to task nature: (a) physical tasks (scored yes vs. no), (b) memory tasks (scored yes vs. no), and (c) procedural tasks (scored yes vs. no). All three moderator variables were significant. Results suggest that the feedback effect was stronger in situations where participants performed a non-physical task (d=0.36) rather than a physical task (d=-0.11). Feedback effects are weaker when participants performed an algorithmic task (d=0.19) rather than a non-procedural task (d=0.36). Feedback effects are stronger when memory tasks (d=0.69) were performed rather than other task-types (d=0.30).16 However, the value of Kluger and DeNisi’s information is limited due to lack of specific information about task types. What do the authors mean by ‘other task types’ and ‘non-physical task’?

A comparison of experimental and control groups in test-like events, gives medium effect sizes on multiple choice items (d=0.48) while they are small on completion items (d=-0.02), mixed items (d=0.27) and short answer items (d=0.29).21 Within SLA different tasks can be performed, and each task has its own outcome measure. Constructed-response tasks require learners to produce the target language freely without constraints. Constrained constructed-response tasks require learners to complete tasks in which the use of the target features are necessary. Metalinguistic tasks requires learners to judge the grammatical correctness of target structures. A comparison of pre- and posttest and experimental and control group on these different SLAtasks shows a significant difference on task outcomes. The effect sizes were higher (d=0.97 between-groups, d=1.25 within-groups) for the free response tasks, compared

to the other two task types. Effect sizes were the lowest for the meta-linguistic tasks (d=0.45 between-groups, d=0.70 within-groups).7 Task perception Both FP’s and FR’s can perceive the task in certain ways. Task perception moderates the effectiveness of feedback. When Fukkink used video feedback, he noticed that when outcome variables of a task are perceived as positive behavior (active listening, authenticity), are compared to outcomes that are perceived as negative behavior (nervousness, passivity), effect sizes are higher for positive perceived behavior (g=0.41) compared to negative perceived behavior(g=0.28).17

Culture and context Culture influences how performance dimensions and job roles are defined. Both cultural specific and universal perceptions determine how job roles are viewed. For example in the UK a performance dimension is: ‘minimum supervision required’ while in China ‘obedience’ was used as a job dimension.23 Culture also influences motivation and personal values. USA FPs expect FRs to be extrinsically motivated, while Latin American FPs expect FRs to be intrinsically motivated. A comparison between Hong Kong and USA FRs revealed that Hong Kong FRs scored higher on values that reflect collectivism, while US participants gave higher importance to cultural values of individualism.23 It is not clear how culture influences the effect of feedback. Feedback has different effects on performance in the USA and England. Both praise and criticism improved performance in the USA; in England only praise increased the outcome. Possible mediators are factors such as power distance, uncertainty avoidance, trust in the supervisor and the perceived importance of the feedback.23 In a centralized organization culture decisions are made by a few managers and employees have little autonomy. Employee involvement in centralized organizations to develop and implement a feedback system improves the performance outcome (r=0.26).

We hypothesize that employees autonomy (from involvement) will also produce outcome gains. 24 Study context is important in SLA and it moderates the feedback effect. When students learn a foreign language in their own country, this is called a foreign language setting: a French student learning Spanish in France A second language setting is one in which the learner’s target language is the primary language of the linguistic community: a French student studying Spanish in Spain. For foreign languages settings feedback ef-

187 Appendices

Task subject matter In performance assessment the subject area of a task (math, science, listening, reading, writing, foreign language) moderates task sampling-variability. For example, the percentage of variance due to task sampling for writing was 3% and for listening 18%. The percentage of variance for person-task interaction was even larger. The lowest percentage was found in Foreign Language (18%), and the highest percentage was found for Math (37%).13 The content area in which the task was performed (Mathematics, English Language Arts (ELA) or Science) also moderated the feedback effect in formative assessment in students in K-12 grades. Effect sizes were highest for ELA (d=0.32) and lowest for Science (d=0.09). Students who had ELA benefitted more from the formative assessments, than the students in the intervention condition who had Science.22

fects are high (d=0.74), effects are between small and moderate in a second language setting (d=0.41).25 Lyster did not find differences in feedback effects regarding to instructional context. He compared contexts in which a foreign language was learned compared to settings in which a second language was learned.7

Appendices

188

FR’s cultural background Values about job dimensions vary among cultures. A FR’s cultural background predetermines role perceptions. Especially when a learner works in another country and/or has a supervisor with a different cultural background, this will influence task performance. No empirical data is available which explains how cultural background influences task performance.23 The correlation between subjective ‘supervisory ratings’ and objective measures of job performance --the amount of time spend on a task, quantity of production, days not spend at work-- is moderated by FR’s ethnic background. A stronger correlation was found between FP’s ratings and job performance measures for black FRs (r=0.22) than for white FRs (r=0.11). FP’s observations of black FRs are stronger related to FR’s actual performance than observations of white FRs.26 A possible explanation for this difference is that when the FP and the FR share their (ethnic) background, compensatory factors, such as giving each other the benefit of the doubt, are more likely to occur. This may cause low correlations between FP’s observation and objective job performance measures.27 In Ford’s study ethnic background also influences ratings. FP’s ratings for white FRs are higher than for black FRs. White FRs have also a higher performance level on objective performance indices than black FRs.28 In Claus’ study we find a possible explanation for this finding. Ethnic background is connected with cultural background. The differences in cultural background, the norms and values and the related perceptions of both raters and ratees might cause these differences.23 FR’s age The age of the FR can also influence the effectiveness of the feedback. In Lyster’s study on SLA in which learners received corrective feedback, a regression analyses showed a negative correlation in both between- (r= -0.41) and within group (r=-0.51) results between age and outcomes. Younger learners profited more from the corrective feedback compared to older learners.7 FR’s skills When a group of students completes a set of tasks some variance in their scores is determined by the person-task interaction. Person-task interaction means that when all students were ranked for each task the ranking would vary across tasks. When a group of students has a lot of variation between students in skills, the percentage of variance for student-task interaction will be high. When students have exactly the same skills the percentage of variance for student-task interaction will be low.

Scores having a large amount of task-related variation are not reliable and decisions based on such scores are inaccurate. A meta-analysis on task-sampling variability indicates that the percentage of variance for student’s differential performance of the same task was 26%.13 Feedback on clinical practice was more effective in a group FRs who did not meet the required baseline of recommended practice, than in a group who met this requirement.19

FR’s skills did not moderate the effects of VF on performance. Effects in groups with less and more experienced trainees were the same.17 B: Effects of observation and interpretation on later phases of the feedback process and effect Several variables influence other measures connected with observation and interpretation (phase B). Few variables influenced phase C and the feedback effect. No variables influenced phase D.

Nature of instrument Results about how observation instruments influence ratings are mixed. Norton and Wanguri describe the influence of FP’s interpretation of observation categories. When instruments have objective measures, correlations between peer observations and objective measures of performance are higher (r=0.73) than when instruments leave space for personal interpretation (r=0.39).30 The halo effect drops when rating categories are chosen that do not force raters to rely on their overall evaluations of the FR. Behavior Anchored Ratings Scales (BARS) lead to accurate performance observations.5 Heneman found significant differences in rating methods due to the standards used.

This will be discussed under ‘Type of Standard’. However, in other studies, rating format, rating scale, rating scale format, and rating method did not significantly influence the outcomes.15, 31-35 Two examples illustrate why authors did not find a significant differences. Murphy and colleagues studied whether rating scale format influenced rating accuracy in paper cases and real life observations. However, they did not perform detailed analyses to compare if different rating methods influenced the outcomes differently.15 Jawahar controlled whether different ratings scales moderated the effect of observation purpose. Analyses of subgroups of rating scales involves only two or three studies.31 This suggests that findings might be an artifact of authors choices of research design.

189 Appendices

Training content and method Rating errors influence the observation and the feedback message: halo effect, leniency effect, rating accuracy,4-5, 29 and observational accuracy.29 Training about how to work with observation instruments reduces rating errors and increases accuracy.5 Woehr compared four different training approaches. A comparison of experimental and control groups showed the impact of training on task observation. Effect sizes are: rater error training (d=0.25), performance dimension training (d=0.18), frame of reference training (d=0.45), and behavior observation training (d=0.59).29 Performance standard training reduced leniency errors in four of seven studies. Rater error training and performance dimension training reduced halo errors.4, 29 Rating accuracy is increased by a combination of frame of reference training (d=0.83),4, 29 performance dimension training,4 and behavior observation training (d=0.77).29 Observational accuracy also increased via behavior observation training (d=0.49).29 Smith reviewed 24 studies on training methods and showed that training methods also influence rating accuracy. Observation training which use a discussion format reduce leniency errors in 50% of the examined studies; and halo errors in 85%. Training that uses practice and feedback reduces halo errors in 70% and improves rating accuracy in 83%. Lectures reduce halo errors by 60%.4

Appendices

190

Assessment method - utility A review of 46 studies about performance assessment of health care providers shows that eight assessment methods are commonly used: appraisal/performance interview (n=16), self-assessment (n=7), multisource feedback (n=3), portfolio (n=3), critical incident technique (n=2), observation (n=2), reflection (n=1), and supervision (n=1). Six studies report that validity aspects (content, factorial or face validity) are adequate. Five studies reporting adequate reliability. To assess performance of health care providers in a reliable and valid manner a multi-method strategy is suggested.35 Further research is needed to investigate whether or not this multi-method strategy is effective.

Peer review is ‘an organized effort whereby people critically appraise, systematically assess, monitor, make judgments, determine their strengths and weakness, and review the quality of their practice to provide evidence to use as the basis of recommendations by obtaining the opinion of their peers’ (Rout & Roberts 2006). A systematic review on the use of peer review in midwifery and nursing revealed that it was not possible to identify factors for good practice because high quality, published evidence is lacking.36 The type of formative assessment used in Kingston’s study on students achievement in K-12 grades moderated the feedback effect. Professional development, where the trainer explained how to use formative assessments (d=0.30) and the use of a computerbased formative system (d=0.28) was the most effective. The effect size of the category with classroom assessment activities, student reflection, assessment conversations (d=0.10), curriculum-embedded assessments (d= -0.05), and student feedback (d=0.03) were very low.22 Miller reviewed 16 medical education studies. Different assessment instruments were described in these studies: multisource feedback, Mini Clinical Evaluation Exercise (M-CEX), Direct Observation of Procedural Skills (DOPS), and multiple assessments methods. All these studies report a positive impact on educational results in workplace based assessment.37 But similar to Craig’s study, evidence favoring performance improvement is lacking.37, 39 The authors state there is limited evidence that multisource feedback may lead to improved performance. But the quality of facilitation and the feedback context have an important effect on responses.37 Rubrics A rubric ‘is a scoring tool for qualitative ratings of authentic or complex student work.

It includes criteria for rating important dimensions of performance as well as standards of attainment for those criteria.’ (Jonsson & Svingby 2007, (p.131)). The authors suggest that the use of rubrics increases scoring reliability. Little published evidence suggests that rubrics also increase validity, except for consequential validity, but this is based only on two studies.38 The concrete nature of rubrics makes expectations and assessment criteria explicit.

This facilitates providing feedback according to the perceptions of supervisors using the rubrics.38 Purpose of observation When observations are obtained for administrative purposes compared to observations for research purposes the effect size is d=0.32. Observations obtained for an administrative purpose have higher scores than observations for research.31 Correlations between peer observations and other performance measures are higher when observations are made for research (r=0.56) than observations for administrative

decision making (r=0.37).30 It is possible that observer knowledge that an employee will be directly and personally confronted with consequences of observations for annual evaluations will also influence the rating. Observations for a research purpose will not influence employees personally and will be indirect.

The purpose of observation caused significantly different effect sizes from observations based on paper vignettes (d=1.73) versus observations based on real behavior (d=0.31).15 It is much harder to base ratings on a paper description than on real life observation.

Bias is unlikely in paper descriptions. Many influences cause bias in real life, e.g., the relationship among FP and FR, circumstances of observation, bad mood of FP, and others.

Intensity of observation Rating errors are reduced when FPs observe FRs while concentrating on the task being observed.5 Murphy, Herr, Lockhart and Maguire studied the influence of rater and ratee characteristics, the influence of rating purpose, the influence of performance level, the influence of training, and the influence of rating scale format in FR’s observations, rating from people in real life, and from behavioral descriptions in paper vignettes. Observations of FR’s task performance based on real life observations have low effect sizes (d=0.31). The effect size of observations which are based on paper vignettes which provide descriptions of FR’s behavior is slightly higher (d=0.42).15 Intensity of observation is a moderator variable in Jawahar and Williams’ study on the influence of purpose of rating on scores. When mean effect sizes are compared under circumstances of real life observations versus observations based on a video tape, effect sizes are significant different from each other. The effect of observation purpose is higher in real life observation (d=0.41), than in observation based on video tapes (d=0.04).31 In VF the intensity of observation also moderates feedback effects. VF is more effective

Appendices

Focus of observation The correlation between observations from peers as FP and an objective measurement is lower when the observation focus is on FR’s performance quality (r=0.24) than on FR’s performance quantity (r=0.38).32 A comparison of correlations between ratee race and objective measurements-- absenteeism, cognitive criteria, performance criteria-- shows significant larger effect sizes for cognitive criteria (r=0.34) than for absenteeism (r=0.11) and performance effect sizes (r=0.16). Race effects in this study are larger when measuring cognition than performance or absenteeism.28 In addition, Ford showed significant differences in correlations between race and objective performance measurements compared to correlations between race and subjective performance measurements. Effect sizes were higher for the objective indices measuring performance and cognition.28 The relationship between observations from supervisors as FPs and objective jobknowledge ratings is stronger (r=0.22) than the relationship between supervisor observations and objective job-performance ratings (r=0.13).26 In general, it is easier to measure a person’s knowledge than to observe and measure a person’s performance. The degree of bias in job-performance measures, whether subjective or objective, may explain the low correlation between these two measures.

191

when trainers use a structured observation form (g=0.55) than when these forms are absent (g=0.21).17 A review of seven studies that focused on direct observation in the Emergency Department (ED) concluded that direct observation is beneficial for trainee learning according to trainee self-report. Behavior changes were also reported but data on whether behavior change is sustained over time are lacking. It is not clear from these studies that direct observation also improves patient outcomes or improved practice. Direct observation involves supervisors assessing patients and making plans, but this seems to be at the expense of longer patient waiting times.39

Appendices

192

Type of standard Outcome oriented measures of performance, e.g., countable behaviors, job outcomes, should be used to observe FR’s performance objective. Examples include the amount of time spent on a task, quantity of production, or absentee work days. Subjective measures of performance leave more room for FP personal impressions.32 Correlations between objective measures and subjective performance ratings vary, ranging from r=0.27 to r=0.69.30, 32-33 The correlation between FP’s subjective observations and observations in which another employee functions as a standard is higher (r=0.66) than the correlation between FP’s subjective observations and a result oriented measure of performance as standard (r=0.21).33 Time to build relation, trust, and safety When peers in a peer feedback situation had time to get acquainted with each other their observations had higher correlations with an objective measure of performance (r=0.69), than peers who lacked time (r=0.20).30 Van Gennip’s study shows that in peer-assessment (PA) interpersonal variables are rarely studied. Few studies report how trust or psychological safety affects student learning and their perceptions about learning through PA. The outcomes are equivocal; trust in peer-assessor did not increase learning results. Sometimes perceptions about learning through PA were affected positively. 40 Sasanguie was searching for arguments for and against splitting the role of supervisors as teachers and assessors. She found that when students have an external assessor it gives them a liberating feeling, i.e., they feel free to ask questions and have a more open attitude. For students with high test anxiety and poor study skills knowing the supervisor/ examiner is a trust factor. An argument against having the supervisor as assessor is that students can develop strategies to impress the supervisor which can shape the reliability of (formative and summative) assessments. All options influence the reliability of assessments. 41 FP’s position In multi-source feedback situations FRs are observed by supervisors, peers,14, 20, 34 subordinates,14 and oneself.5, 14, 34 Interrater reliability of observations among supervisors as FP is higher than among peers.20, 14 Interrater reliability for supervisors approximates r=0.50, for peers it approximates r=0.40,20, 14 and for subordinates r=0.30.14 Observations from FPs who are subordinates have low correlations with observations from FPs who are peers, supervisors, and self-assessors; correlations between ratings from different FP groups range from r=0.14 to r=0.22.14 Agreement between self-assessors and peers, and self-assessors and supervisors are also low.14, 34 Correlations between

FP’s task familiarity Supervisors as FPs express more observational agreement than peers as FPs on both managerial and non-managerial tasks.14 There was lower agreement between FR’s selfratings and supervisory ratings and self-ratings and peer ratings on managerial jobs ρ=0.27 and ρ=0.31 than on non-managerial jobs ρ=0.42. and ρ=0.40.34 Managerial jobs are high-complexity activities. An explanation for this finding is that employees estimate their skills better in low-complexity activities than in high-complexity activities.

There is more agreement among supervisors as FPs (r=0.53) than among peers as FPs (r=0.33) on the observation of job knowledge, the required knowledge to get a job done.20 Subordinates as FPs agree more on observations of interpersonal dimensions (r=0.34) than cognitive dimensions (r=0.25) of job performance. Supervisors and peers as FPs yield opposite results, i.e., they agree more on cognitive than on interpersonal dimensions.14 Interrater reliability among supervisors as FPs (r=0.43) compared to subordinates as FPs (r=0.25) is higher on cognitive aspects of a task, i.e., diagnosing problems, job knowledge, and judgment ability.14 For external assessors not knowing the situation and the task it can cause uncertainty, which may influence the observational ratings.41 FP’s cultural background FP’s cultural background influences ratings. A comparison between European and USA ratings on one hand and Brazilian and Asian ratings showed that European and USA ratings were higher.23 Cultural background of raters also leads to bias related to the cultural background of the FR or to the type of behavior the FR presents. This bias is based on cultural desirability of behavior.23

193 Appendices

peers and supervisors are higher: ρ=0.6234 and r=0.34.14 Jawahar compared control groups where ratings are obtained for research purposes with treatment groups where ratings are obtained for administrative purposes. He found a difference in effect sizes when student and professionals were rating. The influence of ‘observation purpose’ is higher in professionals’ ratings (d=0.50) than in students’ ratings (d=0.22).31 When supervisors as FPs observe a subordinate, the influence of ‘observation purpose’ is higher (d=0.40) than when subordinates as FPs observe a supervisor (d=0.09).31 A possible explanation for these findings is that students and subordinates have less concern about the consequences of ratings. Thus students do not rate differently when a ratee has to be evaluated for research, administrative reasons, or for the purpose of an annual performance appraisal. Subordinates find it difficulty to evaluate their supervisors. For administrative ratings they might be less critical and this might reduce the difference between the rating types.

Supervisors and assessors are often the same person. Sasanguie explored the consequences of disentangling these roles from the perspective of the supervisor. When roles are disentangled it forces them to make the criteria and content for assessment very clear, so that an external assessor should be able to repeat the assessment. An argument in favor of disentangling roles is that no conflict of interest is present. External assessment increases the quality of the program, instruction, and the assessment producing higher accountability, reliability, and fairness.41

C: Effects of feedback message communication on later phases of feedback process and effect We found several influences on variables connected with receiving feedback and its interpretation (phase D). All showed the feedback effect. No influences were found on phase C (table 3.2 in the article).

Appendices

194

Feedback source The source of feedback in the clinical setting it makes a real difference. In 62% (n=18) of the studies a positive effect was reported when employees received feedback from clinical house staff. Feedback provided by research teams was less effective.42 The feedback source moderated the effect of corrective feedback in a study on SLA. This study compared groups with and without corrective feedback, which is not explicitly stated in the report. Li found that the interlocutor type moderated the effect of corrective feedback. In studies in which feedback was provided by a native speaker effect sizes were significantly higher (d=1.00) than in studies in which feedback was provided by a teacher (d=0.41). No significant differences in effect sizes were found for studies in which feedback was provided by a computer (d=0.83) and native speakers (d=1.00).25 Feedback medium Kluger and DeNisi compared an experimental group that received feedback with a control group without feedback. Both computer mediated feedback and verbal feedback moderated the feedback effect. Studies in which computer mediated feedback has been provided showed stronger feedback effects (d=0.41) than in studies in which feedback was provided by other media (d=0.23). Verbal feedback is less effective (d=0.23) than other feedback forms (d=0.37).16 In this work the information is lacking what ‘otherwise provided feedback’ or ‘other feedback forms’ means.

Hepplestone reviewed how teachers can help students use feedback and act on it using technology. He reviewed 42 studies and reported different ways to engage students: 1) send them feedback electronically, so that they always can reread the feedback; 2) use electronically produced FB, e.g., with “track-change” or “comments” in word-programs or use audio feedback;. 3) release grades when feedback is given and take notice of or withhold grades when feedback is not provided. This is an effective way of engaging students but evidence that this was done electronically Is absent; 4) computer assisted assessments help students to monitor their own learning and foster independence; 5) PA, providing critical comments to other students, and peer FP’s own performance are related. The quality of the peer feedback was higher when FP’s own quality of work was good. From the narrative description we conclude that these five different approaches prompt engagement with the feedback and stimulate trainees to act.43 Feedback form Instruction content with embedded feedback and the combination of a feedback intervention with other educational interventions (e.g., audits, supervision, or education-multi-facetted interventions) influence the effectiveness of feedback.11, 19, 21, 44 Multiple facetted interventions are more effective than single feedback interventions.19 Clinicians’ performances improved in 63% (n=24) of the studies in which feedback is combined with education. Seventy-five percent (n=12) reported improvement when feedback was combined with educational outreach visits and 81% (n=62) reported im-

Feedback content Feedback about successes increases FR’s intrinsic motivation (d=0.34). Feedback with negative connotations did not influence FR’s intrinsic motivation (d=-0.16).45 Feedback that gives general information may shape learner reception and interpretation of feedback. Learners may not be sure about how to respond to feedback or they need more information processing time to understand what the FP intends.46 A discouraging feedback intervention is a moderator variable in Kluger and DeNisi’s study. The effect sizes of studies having a discouraging feedback intervention are small and negative (d=-0.14). When the feedback content is encouraging, effect sizes of performance outcomes are higher and positive (d=0.33).16 Feedback having relevant information, i.e., correct answers (the standard) task information, is more effective than shallow feedback, i.e., ‘right wrong’ or compliments.10, 16, 21, 47 Several moderator variables relate to feedback content in Kluger and DeNisi’s study.

When pre and posttests are compared after receiving feedback, the effect size for feed-

195 Appendices

provement when feedback was combined with other interventions.42 A combination of supervision, audit, and feedback resulted in moderate to large improvement of health-worker performances.44 Steinman studied the influence of feedback on the quality of drug prescription. Feedback combined with clinician education and audit improved the quality of prescriptions for 3.4%. However, when clinicians received only education, quality of drug prescription improved for 13.9%.11 Bangert-Drowns, Kulik, Kulik and Morgan compared groups that received feedback on answers with a group that did not receive feedback. They found that type of instruction mediated the feedback effect. Feedback embedded in programmed instruction (d=-0.04), computer assisted instruction (d=0.22), or in an instruction on text comprehension (d=0.48), was less effective than feedback embedded in a testing situation (d=0.63).21 In Fukkink’s meta-analysis on VF trainees see their own behavior on tape and discuss it with a trainer. Feedback is based on video-observations. They found that VF positively affects the interaction skills with a medium effect size (g=0.40). Effect sizes were slightly higher for verbal skills (g= 0.42), than for non-verbal (g= 0.35) or paralingual skills (g=0.39). The effect sizes for receptive, informative, and relational skills are g=0.44, g=0.47, and g= 0.35, respectively. So among the interaction skills, verbal skills can be the easiest enhanced by VF. From the other skills informative skills which have the highest effect sizes.17 Contrary to Bangert-Drowns, Fukkink did not find an effect of the type of instruction affecting feedback. If no additional instruction is given in a VF situation the feedback is as effective than when instruction is given.17 An explanation is that VF already has such a big impact that more instruction does not make a difference.

Lyster distinguishes three feedback types in SLA. Teachers give learners cues to draw on their own resources to self-repair, they reformulate learners utterance, minus the error, and they provide the correct form. The three feedback types increase the feedback effect in both the between and within group contrasts. When teachers give learners cues for to draw on their own resources to self-repair, effect sizes are respectively d=0.83 and d=1.14. When teacher’s reformulated learners student’s utterance, minus the error it is d=0.53 and d=0.70. When teachers providing the correct form it is d=0.84, d=0.60.7

Appendices

196

back that yields a correct solution is d=0.43. When the correct answer is withdrawn the effect size is d=0.25. Feedback that gives the correct answer is more effective.16 Knowledge of Correct Response (KCR) is a type of feedback where the learner is told the correct answer after an incorrect response.

When pre and posttests are compared after receiving feedback the effect size for feedback only giving praise is d=0.09, when feedback contains praise and more information the effect size is d=0.34. Feedback with praise and additional information is more effective.

When pre- and posttests are compared after receiving feedback on performance change related to previous trials the effect size is d=0.55. The effect sizes after receiving feedback without this information is d=0.28. Feedback with information related to previous trials is more effective.16 Bangert-Drowns, Kulik, Kulik and Morgan’s study compared a control group without feedback with an experimental group with feedback. They found that the type of feedback influenced the effect sizes. When learners received feedback until they performed correctly (repeat until correct feedback), or feedback having an explanation, the feedback effect was medium (d=0.53). Feedback effects were low when feedback only had ‘right’ or ‘wrong’ (d=-0.08) or the correct answer (d=0.22).21 In cognitive feedback on judgment tasks, task information contributes most to the effect of the feedback.10 A meta-analysis on determinants of judgment tasks studied the role of three different types of feedback: 1) outcome feedback knowledge of the criterion value;2) cognitive feedback, refers to information of the judgment process; 3) task information feedback, refers to information about the environment in which the judgment takes place. These findings agree with previous research, i.e., students learn best from feedback that instructs them about the characteristics of the judgment task they have to perform.48 Outcome feedback in these judgment tasks was helpful when the learners where unfamiliar with task requirements, but detrimental when the learners had already acquired some experience with the judgment task.48 According to Shute46 the specificity of the content is important too. When the feedback message contains specific or elaborate information about how to improve performance, the feedback messages will be more effective than feedback involving general information.46 In Pritchard’s study the variable ‘changes in the feedback system’ was a moderator variable. Sometimes the feedback content fails to match feedback effects. Changing the feedback system correlates with the effectiveness of the feedback (r=0.30). When it took several changes to tailor the performance indicators in the feedback to the performance outcomes, feedback was less effective than when no changes in the system were made.24 Another key moderator in Pritchard’s study was the quality of organized feedback meetings. In high quality meetings time was spent discussing feedback reports with participants in the presence of a supervisor. Time was spent on how feedback relates to performance, used to identify problem causes, on improvement strategies, and on future goals. The correlation of this variable with the effectiveness of the intervention is high (r=0.43).24 Feedback complexity It is not clear what the effect is of long and complicated feedback. Shute reports some

negative effects and also reports studies in which no effect was found due to feedback complexity on outcome variables. The findings are inconclusive.46

Feedback frequency Kluger and DeNisi compared an experimental group that received feedback with a control group without feedback. Feedback frequency moderated the feedback effect.

Frequent feedback has stronger effects (d=0.39) than infrequent feedback (d=0.32).16 Continuous feedback reports about drug prescribing behavior by medical doctors changes drug prescribing practice.51 In Pritchard’s study feedback resulted in improved organizational productivity. Pritchard showed that the effect of feedback did not decrease over time even after feedback was given ten times or more.

Feedback intensity Studies in which feedback is provided in the clinical setting for over two years report significantly more effect than studies in which feedback is provided less than one year.42 A similar effect was found for the effect of corrective feedback in SLA. Li does not explicitly state which groups are compared but we infer groups that received corrective feedback were compared to groups that did not receive feedback. The effect of correc-

197 Appendices

Feedback timing Bangert-Drowns, Kulik, Kulik and Morgan’s study compared a control group without feedback with an experimental group that received feedback. They found that the type of feedback influenced effect sizes. Two variables related to timing mediated the feedback effects. The effect of feedback was weak among FRs who could view the feedback before completing the task (d=-0.08). Effects are stronger for FRs who received feedback after task completion (d=0.46).

They found also a significant effect of feedback timing. Immediate feedback after a test is more effective (d=0.72), than delayed feedback (d=0.56). They suggest that feedback timing is a confounding variable related to instruction type.21 Immediate feedback was more effective than delayed feedback in applied tasks in classrooms (d=0.28) and in list-learning experiments (d=0.34).49 When motor performance tasks are learned, immediate feedback is more effective than delayed feedback (d=0.55).50 Kulik and Kulik compared studies in which students received immediate and delayed feedback. The effect of feedback timing is moderated by task type and by the type of additional information which might be presented with the feedback. In a list-learning task, delayed feedback after an item is more effective (d=0.55) than delayed feedback after a test (d=-0.24). In experiments on acquisition of test content immediate feedback is inferior to delayed feedback.49 Further, they found that in list-learning timing had a small effect if the feedback and the items to be learned are presented together (d=0.09). When feedback was presented with FR’s answer, delayed feedback had a negative effect and immediate feedback was more effective (d=0.76).49 Jaehnig and Miller report from a review that in programmed instruction delayed feedback is effective because participants receive an additional exposure to the instructional frame.47 Shute illustrates that the effect of feedback timing is mixed, regardless of the unit of time is used. There is support for both immediate and delayed feedback being effective.

She suggests that most field studies report that immediate feedback is more valuable, whereas laboratory studies show positive effects of delayed feedback.46

Appendices

198

tive feedback is moderated by its duration. When the feedback session was 50 minutes or less, effect sizes were higher (d=1.15), than in sessions of medium length (60-120 minutes) (d=0.46) or in sessions over 120 minutes (d=0.49).25 In clinical practice, intensive feedback-- verbally provided by a supervisor or senior colleague as FP towards the individual FR-- leads to an improvement of 55% in care.

Non-intensive feedback --not verbal feedback, provided by a peer or subordinate as FP towards a group about general information (costs, numbers of tests)-- leads to an improvement of 45% in care. Moderate intensity feedback, i.e., between intensive and non-intensive, leads to an improvement of 11%.19 It is not clear how to explain this. One possibility is that in non-intensive feedback situations learners accept personal responsibility and are intrinsically motivated to change the situation. By contrast, the intensive feedback situation prompts external motivation which leads to a change. The moderate intensity feedback situation provides insufficient motivation to change and too much prompting to be intrinsically motivated.

In Lyster’s study on SLA brief (<1 hour), short-medium (1-3 hours) and long (>7 hours) corrective feedback sessions were compared. The effect sizes of the long sessions were significantly higher in both the between (d=1.13) and the within-group (d=1.11) results compared to sessions with brief and short-medium duration. The variation in effect sizes of the brief and short-duration sessions ranged from d=0.57 to d=0.87.7 D: Receiving and interpreting feedback and its effect We only found influences on variables related to the feedback effect. No influences were found on phase D. Time interval between feedback – next performance Timing in this variable refers to the interval after receiving feedback and before the next performance. In a study on motor-performance, task feedback is more effective when the time interval between feedback and the next performance is more than five seconds. The effect size between FRs with time intervals less than five seconds and more than five seconds is d=0.52.50 The time interval between feedback and next performance also influenced feedback outcomes in a SLA study. When corrective feedback was given and the next performance was within 2-6 weeks effect sizes were higher (d= 0.84, d=1.01) in both the between- and the within-group contrasts compared to the situation where the next task performance occurred within a week (d= 0.63 and d=0.84), but this difference is not significant.7 Activities between feedback – next performance In motor-performance learning Travlos studied the effect when a FR engaged in another activity between the receiving feedback and the next performance. Effect sizes were calculated from studies with and without intervening activities in the FeedbackPerformance cycle. FR’s performance dropped (d=0.44), when a FR performed a verbal task between the received feedback and the next performance. When a motor task instead of a verbal task was performed in the Feedback-Performance cycle, it had little impact on FR’s performance (d=0.13).50 FR’s self-esteem Kluger and DeNisi compared experimental groups that received feedback with control groups without feedback. Feedback as a threat to self-esteem moderated the feedback

effect. When feedback is perceived as reflecting career prospects or intelligence, it was associated with a stronger feedback effect (d=0.47) than feedback which was seen as threatening to FR’s self-esteem (d=0.08).16 FR’s goal setting behavior Goal setting is another moderator in Kluger and DeNisi’s study. When a FR sets goals the feedback effect is stronger (d=0.51), than when no FR goals are set. (d=0.30).16 V Description of feedback results according to the narrative review of Wanguri

A: Effects of tasks, standards and first task performance on later phases of the feedback process or effect In this study we found only influences on phase B and C. No influences were found on phases A, D, and Effect.

FR’s age Young FRs will be rated higher by FPs older than the FRs. Older FRs receive lower ratings.

FR’s skills Feedback from a supervisor towards a well performing FR is supportive, for low performers the need for improvement is stressed. Ratings are higher for well performing FRs than for poor performers. Supervisors are perceived as positive by high scoring FRs. A cynical attitude is observed among low scoring FRs. B: Effects of observation and interpretation on later phases of the feedback process and effect In this study we found only influences on phase B and D. There are no influences on phase C and the feedback effect.

Nature of the instrument Goal directed instruments where objectives are central lead to higher satisfaction with the feedback process than instruments featuring subjective traits. Instruments that are constructed by the FRs generate positive perceptions among FRs toward the feedback process.

Purpose of observation FRs better accept ratings for personal development than ratings used for administration, research, or personal evaluation.

Number of FPs Judgments are perceived as more fair when multiple raters are involved rather than having only one participating rater. Accuracy increases when more FPs are involved in task performance, observation, and rating. Ratings are more accurate in multi-source feedback situations.

199 Appendices

Wanguri’s narrative review reported several variables that were not mentioned previously in the meta-analyses and literature reviews. Personal variables receive more attention, such as FP’s self-esteem, age, involvement, communication skills and attitude, and FR’s age, communication skills, satisfaction and trust (table 2).

This illustrates that the variables we listed before are not exhaustive.

Table 2. The 14 variables mentioned in the study of Wanguri and their effect on the feedback process and effect Dependent Dependent Dependent Dependent Dependent variables variables variables variables variables in phase A: in phase B: in phase C: in phase D: Effect: Independent variables connected with: Phase A: tasks, standards and first task performance Feedback recipient’s age

Appendices

200

x

Feedback recipient’s skills

x

Phase B: observation and interpretation Nature of the instrument

x

Purpose of observation

x

Number of raters

x

Feedback provider’s self-esteem

x

Feedback provider’s age

x

Feedback provider’s involvement Phase C: the communication of the feedback message Feedback provider’s communication skills Feedback provider’s attitude Phase D: reception and interpretation of the feedback message Feedback content

x

x x x x

Feedback recipient’s communication skills

x

Feedback recipient’s satisfaction Feedback recipient’s trust

x x

FP’s self-esteem Self-raters with high self-esteem have more leniency bias than self-raters with low-selfesteem.

FP’s age In general, young supervisors give lower ratings than older supervisors. On interpersonal skills they give higher ratings than older supervisors do. Supervisors rate older colleagues lower than their younger colleagues.

FP’s involvement FRs perceive the feedback process as unhelpful when feedback providers do not pay attention to the feedback process and withhold a clear analyses of FR’s performance.

Supervisors have positive perceptions toward the feedback process and are also more satisfied with it than colleagues who are not supervising.

C: Effects of the feedback message on later phases of feedback process and effect In this study we found only influences on phase C, no influences were found on phase D, the feedback effect.

FP’s communication skills Supervisors and FRs are more satisfied about communication in the feedback process when a participative approach, i.e., asking questions, is used to improve mutual understanding.

D: Effects of reception and interpretation on feedback process and effect In this study we found only influences on phase D and the feedback effect.

Feedback content FRs perceive feedback as unhelpful when a clear analysis of FR’s performance is missing. Feedback messages about future goals result in higher work satisfaction and organizational outcomes.

FR’s communication skills The quality of the dialogue in the feedback communication influences FR’s productivity on the next task performance.

FR’s satisfaction FRs satisfaction with the feedback communication leads to higher job satisfaction and improves job-performance.

Low rated FRs perceive the feedback process to be more unfair than high rated FRs. FR’s trust FRs who trust the relationship with the supervisor perceive the feedback process to be fair.

VI

A list of abbreviations

Throughout this document a few abbreviations are used that warrant explanation: BARS BOT CF CFB DF DO DOPS EA ED

= Behavior Anchored Rating Scales = Behavioral Observation training = Corrective feedback = Cognitive feedback = Delayed Feedback = Direct Observation = Direct Observation of Procedural Skills = Educational Achievement = Emergency Department

201 Appendices

FP’s attitude Supervisors who have a positive attitude are described as more participative in goal setting and less critical during feedback.

Appendices

202

ELA EM ESd ESg ESr FB FI FL FP FOR FR L2 M-CEX MTF KCR KR L2 PA PDT PGM ProMES PST RET SL SLA UK USA VF

= English Language Arts = Emergency Medicine = Effect size d = Effect size Hedgesâ&#x20AC;&#x2122; g = Effect size r = Feedback = Feedback Intervention = Foreign language = Feedback provider = Frame of reference training = Feedback recipient = Second language = Mini Clinical Evaluation Exercise = Multiple Try Feedback = Correct Response Feedback = Knowledge of Response Feedback = Second Language = Peer assessment = Performance Dimension Training = Post Graduate Medicine = Productivity Measurement and Enhancement system = Performance Standard Training = Rater Error Training = Second language = Second Language Acquisition; = United Kingdom = United States of America = Video feedback

References

203 Appendices

* References marked with an asterisk indicate studies included in the meta-review.

1. MacKinnon DP, Krull JL, Lockwood CM. Equivalence of the mediation, confounding and suppression effect. Prev Sci 2000;1(4):173-181.

2. Bauman AE, Sallis JF, Dzewaltowski DA, Owen N. Toward a better understanding of the influences on physical activity. The role of determinants, correlates, causal variables, mediators, moderators, and confounders. Am J Prev Med 2002;23(2S):5-14.

3. Cohen J. Statistical power analysis for the behavioral sciences. New York: Academic Press, 1969.

4. *Smith DE. Training programs for performance appraisal: A review. Acad Manage Rev 1986;11(1):22-40.

5. *Wanguri DM. A Review, an Integration, and a Critique of Cross-Disciplinary Research on Performance Appraisals, Evaluations, and Feedback, 1980-1990. J Bus Commun 1995;32(3):267-293.

6. O’Reilly CA, Anderson JC. Trust and the communication of performance appraisal information: The effect of feedback on performance and job satisfaction. Hum Commun Res 1980;6(4):290-298.

7. *Lyster R, Saito K. Oral feedback in Classroom SLA. Stud Second Lang Acq 2010;32:265-302.

8. *Cheraghi-Sohi S, Bower P. Can the feedback of patient assessments, brief training, or their combination, improve the interpersonal skills of primary care physicians? A systematic review. BMC Health Serv Res 2008;8:179.

9. *McAfee RB, Winn AR. The use of incentives/feedback to enhance work place safety: A critique of the literature. J Safetey Res 1989;20(1):7-19.

10. *Balzer WK, Doherty ME, O’Connor R. Effects of cognitive feedback on performance. Psychol Bull 1989;106(3):410-433.

11. *Steinman MA, Ranji SR, Shojania KG, Gonzales R. Improving antibiotic selection: a systematic review and quantitative analysis of quality improvement strategies. Med Care 2006;44(7):617-628.

12. *Azevedo R, Bernard RM. A meta-analysis of the effects of feedback in computer based instruction. J Educ Comput Res 1995;13(2):111-127.

13. *Huang C. Magnitude of task-sampling variability in performance assessment: a meta-analysis. Educ Psychol Meas 2009;69(6):887-912.

14. *Conway JM, Huffcutt AI. Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. Hum Perform 1997;10(4):331360.

15. *Murphy KR, Herr BM, Lockhart MC, Maguire E. Evaluating the performance of paper people. J Appl Psychol 1986;71(4):654-661.

16. *Kluger AN, DeNisi A. Effects of feedback intervention on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychol Bull 1996;119(2):254284.

17. *Fukkink NG, Trienekens N, Kramer LJC. Video feedback in education and training: Putting learning in the picture. Educ Psychol Rev 2011;23:45–63.

18. *Clariana RB, Koul R. Multiple-Try Feedback and Higher-Order Learning Outcomes. Int J Instr Media 2005;32(3):239-245.

19. *Jamtvedt G, Young JM, Kristoffersen DT, O’Brien MA, Oxman AD. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2006:19;(2)(2):CD000259.

20. *Viswesvaran C, Ones DS, Schmidt FL. Comparative analysis of the reliability of job performance ratings. J Appl Psychol 1996;81(5):557-574.

Appendices

204

21. *Bangert-Drowns RL, Kulik CLC, Kulik JA, Morgan M. The instructional effect of feedback in test-like events. Rev Educ Res 1991;61(2):213-238.

22. *Kingston N, Nash B. Formative assessment: a meta-analysis and a call for research Educ Meas 2011;30(4)28-37.

23. *Claus L, Briscoe D. Employee performance management across borders: a review of relevant academic literature. Int J Manag Rev 2009;11(2):175-196.

24. *Pritchard RD, Harrell MM, DiazGranados D, Guzman MJ. The productivity measurement and enhancement system: a meta-analysis. J Appl Psych 2008;93(3):540-567.

25. *Li S. The effectiveness of corrective feedback in SLA: A meta-analysis. Lang Learn 2010:60(2):309-365.

26. *Kraiger K, Ford JK. The relation of job knowledge, job performance, and supervisory ratings as a function of ratee race. Hum Perform 1990;3(4):269-279.

27. Alderfer CP. An intergroup perspective on group dynamics. In: Lorsch J, Editor. Handbook of organizational behavior. Englewoord Cliffs, NJ: Prentice-Hall; 1986.

28. *Ford JK, Kraiger K, Schechtman SL. Study of race effects in objective indices and subjective evaluations of performance: A meta-analysis of performance criteria. Psychol Bull 1986;99(3):330-337.

29. *Woehr DJ, Huffcutt AI. Rater training for performance appraisal: A quantitative review. J Occup Organ Psychol 1994;67(3):189-205.

30. *Norton SM. Peer assessments of performance and ability: An exploratory meta-analysis of statistical artifacts and contextual moderators. J Bus Psychol 1992;6(3):387-399.

31. *Jawahar IM, Williams CR. Where all the children are above average: The performance appraisal purpose effect. Pers Psychol 1997;50(4):905-925.

32. *Bommer WH, Johnson J, Rich GA, Podsakoff PM. On the interchangeability of objective and subjective measures of employee performance: A meta-analysis. Pers Psychol 1995;48(3):587-605.

33. *Heneman RL. The relationship between supervisory ratings and results-oriented measures of performance: A meta-analysis. Pers Psychol 1986;39(4):811-26.

34. *Harris MM, Schaubroeck J. A Meta-Analysis of Self-Supervisor, Self-Peer, and Peer-Supervisor Ratings. Pers Psychol 1988;41(1):43-62.

35. *Hamilton ESTC, Coates V, Kelly B, Boore JRP, Cundell JH, Gracey J, Mcfetridge B, McGonigle M, Sinclair M. Performance assessment in health care providers: a critical review of evidence and current practice. J Nurs Manag 2007;15:773-791.

36. *Rout A, Roberts P. Peer review in nursing and midwifery: a literature review. J Clin Nurs 2008;17:427-42.

37. *Miller A, Archer J. Impact of workplace based assessment on doctorâ&#x20AC;&#x2122;s education and performance: a systematic review. BMJ 2010; 342: c5064 doi: 10.1136/bmj.c5064.

38. *Jonsson A, Svingby G. The use of scoring rubrics: Reliability, validity and educational consequences. Educ Res Rev 2007;2:130-144.

39. *Craig, S. Direct observation of clinical practice in emergency medicine education. Acad Emerg Med 2011;18(1):60-67.

40. *Van Gennip NAE, Segers MSR, Tillema HH. Peer assessment for learning form a social perspective: The influence of interpersonal variables and structural features. Educ Res Rev 2009;4:41-54.

41. *Sasanguie D, Elen J, Clarebout G, Van den Noortgate W, Vandenabeele J, De Fraine B. Disentangling instructional roles: the case of teaching and summative assessment. J High Educ 2011;36(8):897-910.

42. *Veloski J, Boex JR, Grasberger MJ, Evans A, Wolfson DB. Systematic review of the literature on assessment, feedback and physiciansâ&#x20AC;&#x2122; clinical performance: BEME Guide No. 7. Med Teach 2006;28(2):117-128.

205 Appendices

43. *Hepplestone S, Holden G, Irwin B, Parkin HJ, Thorpe L. Using technology to encourage student engagement with feedback: a literature review. Res Learn Tech 2011;19(2):117-127.

44. *Rowe AK, de Savigny D, Lanata CF, Victora CG. How can we achieve and maintain highquality performance of health workers in low-resource settings? Lancet 2005; 366(9490):10261035.

45. *Tang SH, Hall VC. The overjustification effect: A meta-analysis. Appl Cognitive Psych 1995;9(5):365-404.

46. *Shute VJ. Focus on Formative Feedback. Rev Educ Res 2008;78(1);153-189.

47. *Jaehnig W, Miller ML. Feedback types in programmed instruction: a systematic review.

Psychol Rec 2007;57: 219-232.

48. *Karelaia N, Hogarth RM. Determinants of linear judgment: A meta-analysis of lens model studies. Psychol Bull 2008;134(3):404-426.

49. *Kulik JA, Kulik ClC. Timing of feedback and verbal learning. Rev Educ Res 1988;58(1):79-97.

50. *Travlos AK, Pratt J. Temporal locus of knowledge of results: A meta-analytic review. Percept Motor Skills 1995;80(1):3-14.

51. *Soumerai SB, McLaughlin TJ, Avorn J. Improving drug prescribing in primary care: a critical analysis of the experimental literature. Milbank Q 1989;67(2):268-317.

52. Burke, D. Strategies for using feedback students bring to higher education. Assess Eval High Educ 2009;34(1):41-50.

53. Whitelock, D. Editorial: e-assessment: Developing new dialogues for the digital age. Br J Educ Tech 2009;40(2):199-202.

Samenvatting

Appendices

206

Feedback is belangrijk in het medisch onderwijs. Veel mensen hebben moeite met het geven van feedback, omdat ze bang zijn dat het de relatie met mensen om hen heen zal schaden. Echter het geven van feedback is belangrijk: co-assistenten en arts-assistenten kunnen veel leren van goede feedback. Wanneer hun wordt uitgelegd wat ze goed doen, waarom ze het goed doen, of juist wat er verbeterd moet worden, dan kunnen ze het toepassen als ze de handeling nogmaals uitvoeren. Als een lerende door goede feedback een handeling niet herhaaldelijk verkeerd uitvoert op een patiënt, dan komt dat de patiëntenzorg ten goede. Ook feedbackgevers zélf hebben baat bij effectieve feedback. Het voorkomt dat ze vaker dan nodig op verbeterpunten terug hoeven komen. De aldus bespaarde tijd kan aan andere zaken worden besteed. Elkaar aanspreken op goede punten en verbeterpunten leidt tot een aanspreekcultuur in het ziekenhuis; dat komt de veiligheid ten goede.

In de afgelopen tijd is er veel aandacht besteed aan feedbackonderzoek. Het onderzoek heeft een lange traditie die teruggaat tot het einde van de negentiende eeuw. Daarnaast wordt feedback in veel verschillende velden onderzocht, zoals in de arbeids- en organisatiepsychologie, in de communicatiewetenschappen, onderwijskunde, gezondheidspsychologie. In het medisch onderwijs krijgen de praktijk en de theorie van feedback veel aandacht. Toch is er nog relatief weinig bekend over de oorzaak dat feedback de taakuitvoering niet vanzelfsprekend positief beïnvloedt. Daarnaast is het niet duidelijk waarom de impact van feedback vaak gering is. Het doel van dit proefschrift is om a) het concept feedback te verduidelijken, b) onderzoek dat in verschillende velden is uitgevoerd met elkaar te verbinden, c) na te gaan of effecten van feedback in verschillende contexten ook gelden voor de medische context, en d) inzicht te krijgen in de variabelen die het feedbackproces en het feedbackeffect beïnvloeden.

In het eerste hoofdstuk wordt een algemeen historisch overzicht gegeven van het feedbackonderzoek. Daarbij wordt ingegaan op de doorwerking van de paradigma’s Functionalisme, Behaviorisme, Cognitivisme en Constructivisme. Vervolgens wordt de aandacht verlegd naar het feedbackonderzoek binnen de Sociale Wetenschappen en hoe dat wordt vormgegeven in onderzoek naar leren en doceren, in arbeids- en organisatiepsychologie, in de therapeutische context, en in de communicatiewetenschappen.

We sluiten af met een beschrijving van het feedbackonderzoek in het medisch onderwijs in de periode 2006-2013. Hieruit blijkt dat het meeste onderzoek wordt gedaan naar de effectiviteit van feedback, en dat er beduidend minder aandacht is voor het feedbackproces, bijvoorbeeld voor de interactie tussen feedbackgever en -ontvanger, of de diversiteit in feedbackmethoden.

In dit proefschrift wordt een feedbackmodel gebruikt dat is afgeleid van de communicatietheorie. Dit model heeft vier fasen (figuur 1.2). Allereerst is er een taak met de daarbij behorende standaarden die door de feedbackontvanger wordt uitgevoerd (fase A). Op het moment dat de taak wordt uitgevoerd observeert de feedbackgever deze taak, en vergelijkt de waarneming met bepaalde criteria (fase B). Het verschil tussen het geobserveerde en de criteria wordt gecommuniceerd tijdens de derde fase, de feedback communicatie (fase C). Terwijl de feedbackgever de boodschap communiceert, wordt deze ontvangen en verwerkt door de feedbackontvanger (fase D).

Dit kan leiden tot een bepaald effect: het feedbackeffect. Als er een effect is, is er een verschil te zien tussen de taakuitvoering op moment één, en de uitvoering van dezelfde

Het doel van het tweede hoofdstuk is het begrip feedback te definiëren. We hebben onderzocht hoe het begrip feedback in de loop van de jaren op conceptueel niveau wordt weergegeven. Drie concepten komen naar voren. Bij het concept informatie ligt de nadruk op de gegeven of ontvangen boodschap. Bij het concept reactie gaat het om de interactie die de feedbackgever en -ontvanger met de informatie aangaan. Bij het cyclische concept - dat vooral gebruikt werd toen het feedbackbegrip in de Sociale Wetenschappen werd geïntroduceerd - gaat het er om dat de verkregen informatie wordt verwerkt en leidt tot output waardoor gedrag verandert. Zowel in de Sociale Wetenschappen als in de Gezondheidswetenschappen wordt het informatieconcept het meest frequent gebruikt.

Een analyse van feedbackdefinities die in het medisch onderwijs voorkomen laat zien dat negen aspecten vaak in definities zijn verwerkt: 1) het type informatie dat tijdens de feedback wordt gegeven (cognitief, over een standaard, resultaten), 2) het doel (ter motivering, om te verbeteren), 3) wie de feedbackontvanger is, 4) de vorm waarin de feedback wordt gegeven (mondeling, schriftelijk, dialoog), 5) de voorbereiding die nodig is (observatie van de lerende, vooraf gegevens verzamelen), 6) uit welke bron de informatie afkomstig is (uit de taak, van een andere persoon), 7) wie de feedbackgever is, 8) onder welke voorwaarden de feedback gecommuniceerd wordt, en 9) contextuele factoren, zoals bijvoorbeeld de plaats waar de feedback gegeven wordt. Gebaseerd op deze analyse wordt de volgende feedbackdefinitie voorgesteld. Feedback in klinisch onderwijs is: “specifieke informatie over een vergelijking tussen een standaard en een geobserveerde taakuitvoering die gegeven wordt met de intentie om de taakuitvoering van de lerende te verbeteren”. Aan deze definitie ligt het informatieconcept ten grondslag. Om deze definitie in praktijk te brengen - de operationele definitie - is het van belang dat de feedbackgever a) informatie verzamelt door bijvoorbeeld observatie, b) het verschil tussen het geobserveerde gedrag en de standaard tot de inhoud van de boodschap maakt, c) deze boodschap richting de lerende communiceert, en d) dit doet met de intentie om de lerende te verbeteren. Aan de hand van de voorgestelde feedbackdefinitie kan men sterke en zwakke feedback onderscheiden. Kenmerken van zwakke feedback zijn: informatie uit de tweede hand, niet gegeven met het doel om de lerende te verbeteren, en een impliciete standaard gebruiken.

207 Appendices

taak op moment twee. Fasen A, B, C en D worden het feedbackproces genoemd. Dit model wordt gebruikt om de twee centrale onderzoeksvragen in dit proefschrift te beantwoorden. Onderzoeksvraag 1: Welke variabelen beïnvloeden het feedbackproces en het feedbackeffect? Wat is de conceptuele en operationele definitie van feedback (Hoofdstuk 2)? Welke variabelen beïnvloeden het feedbackeffect en het feedbackproces volgens reviews en meta-analyses (Hoofdstuk 3)? Hoe beïnvloeden deze variabelen het feedbackproces en het feedbackeffect (Hoofdstuk 3)? Welke variabelen beïnvloeden het feedbackproces en het feedbackeffect volgens co-assistenten en arts-assistenten (Hoofdstuk 4)? Onderzoeksvraag 2: Hoe beïnvloeden variabelen gerelateerd aan feedbackcommunicatie en aan de feedbackgever de taakuitvoering en de beleving van medische studenten? Hoe beïnvloedt framing van de boodschap bij medische studenten de tevredenheid met de feedback, hun geloof in eigen kunnen ten aanzien van een uit te voeren taak, en de daadwerkelijke taakuitvoering (Hoofdstuk 5)? Hoe beïnvloedt de betrouwbaarheid van de feedbackgever bij medische studenten de tevredenheid met de feedback, hun geloof in eigen kunnen ten aanzien van een uit te voeren taak, en de taakuitvoering (Hoofdstuk 6)?

Appendices

208

De studie in het derde hoofdstuk is uitgevoerd om zicht te krijgen op variabelen die volgens de literatuur het feedbackproces en het feedbackeffect beïnvloeden. Daartoe zijn Engelstalige literatuurreviews en meta-analyses die tussen 1986 en 2012 verschenen, systematisch doorgenomen. Aan de hand van een protocol hebben we 46 studies geanalyseerd. We vonden in totaal 33 variabelen die één of meer fases van het feedbackproces en/of het feedbackeffect beïnvloeden. Voorbeelden van variabelen die de taak en de standaard (fase A) beïnvloeden zijn: de taakcomplexiteit, de cultuur en de context, de culturele achtergrond van de feedbackontvanger. De positie die een feedbackgever heeft, het doel van de observatie, en de vertrouwdheid van de feedbackgever met de taak, zijn drie voorbeelden van de 12 variabelen die de taakobservatie en de interpretatie daarvan (fase B) door de feedbackgever beïnvloeden.

De communicatie van de feedbackboodschap door de feedbackgever (fase C) wordt beïnvloed door onder andere de vorm van de feedback, de frequentie waarin de feedback gegeven wordt en de intensiteit van de feedback. De ontvangst en de interpretatie van de feedbackboodschap door de feedbackontvanger (fase D) worden beïnvloed door de tijd die tussen de feedback en de daaropvolgende taakuitvoering zit, of de feedbackontvanger zichzelf doelen stelt, en de eigenwaarde van de feedbackontvanger.

In totaal vonden we 20 variabelen die ook het feedbackeffect beïnvloeden. Van de meeste variabelen is niet eenduidig aan te geven of het effect in positieve of negatieve zin wordt beïnvloed. Zes variabelen beïnvloeden op een eenduidige manier fase B, de taakobservatie en de interpretatie daarvan: 1) Het beoordelen van een hoog complexe taak leidt tot een lage interbeoordelaarsbetrouwbaarheid; 2) Wanneer de feedbackgevers bekend zijn met de taak leidt dat tot meer overeenstemming in beoordelingen tussen feedbackgevers; 3) Training van feedbackgevers in het gebruik van observatieinstrumenten reduceert het aantal beoordelingsfouten; 4) Het gebruik van ‘rubrics’ leidt tot hogere betrouwbaarheid in de scoring en het faciliteert het geven van feedback; 5) Wanneer de feedbackgever en –ontvanger een zelfde culturele achtergrond hebben, dan geeft dat hogere taakbeoordelingen; 6) Wanneer een feedbackgever meer tijd heeft om een relatie met een feedbackontvanger aan te gaan, dan leidt dat tot hogere correlaties tussen een beoordeling van een feedbackgever en objectieve metingen van de taakuitvoering.

Daarnaast vonden we ook zes variabelen die het feedbackeffect duidelijk beïnvloeden: 1) Het feedbackeffect is groot als een feedbackontvanger de taak in beginsel niet goed uitvoert; 2) De eigenwaarde van de feedbackontvanger beïnvloedt het feedbackeffect; 3) Wanneer een feedbackontvanger doelen stelt, vergroot dat het feedbackeffect; 4) Wanneer feedback een onderdeel is van een interventie waarin op diverse wijzen informatie wordt teruggekoppeld - ‘multifacetted intervention’ - dan neemt het feedbackeffect toe; 5) Het feedbackeffect neemt toe als de feedbackinhoud specifiek, uitgebreid en aanmoedigend is; en ten slotte 6) Frequente feedback vergroot het feedbackeffect. Wat duidelijk uit deze studie blijkt is dat de aandacht niet alleen gericht moet worden op de feedbackgever en de manier waarop deze de feedback geeft (fase C), maar ook op de andere onderdelen uit het feedbackproces. Het feedbackproces als geheel beïnvloedt de effectiviteit van feedback.

Op basis van de literatuur uit het voorgaande hoofdstuk en de verhalen uit de praktijk is een instrument ontwikkeld waarmee de beleving van de feedbackontvangers gemeten wordt ten aanzien van het feedbackproces. Het doel van het vierde hoofdstuk is om inzicht te krijgen wanneer feedbackontvangers feedback in de klinische praktijk als

In hoofdstuk vijf en zes vestigen we de aandacht op twee variabelen uit het communicatieproces, te weten de framing van de feedback (H5) en de credibility van de feedbackgever (H6). Beide studies zijn gecontroleerde gerandomiseerde trials. De studenten voeren de stemvorkproeven van Weber en Rinne uit op simulatiepatiënten. Met de stemvorkproeven van Weber en Rinne wordt nagegaan of de patiënten gehoorsproblemen hebben. Op een instructievideo demonstreert een KNO-arts deze vaardigheid en geeft hij een toelichting. Deze taak is gekozen omdat met behulp van video-opnamen goed observeerbaar is of studenten de stemvorkproeven kunnen uitvoeren.

In beide experimenten doorlopen de studenten de volgende procedure (zie ook tabel 5.1 en 6.1). Nadat de studenten toestemmen in medewerking aan dit onderzoek, is hun vertrouwen in hun eigen kunnen om deze taak uit te voeren, ‘self-efficacy’, gemeten.

Vervolgens krijgen ze de video te zien. Daarna voeren ze de stemvorkproeven uit op een simulatiepatiënt, dit wordt op video opgenomen. Na afloop is hun vertrouwen in eigen kunnen voor de tweede keer gemeten. Daarna ontvangen ze feedback en voeren ze de taak nog een keer uit op dezelfde simulatiepatiënt. Na afloop is hun vertrouwen in eigen kunnen voor de derde keer gemeten. Studenten zijn ook bevraagd over hun tevredenheid met de feedback en het proces. Na twee tot drie weken is het onderzoek herhaald. Voor de vierde maal zijn studenten op hun vertrouwen in hun eigen kunnen bevraagd, en voeren zij voor de derde keer de taak op de patiënt uit. De video-opnamen worden met behulp van een checklist gescoord om zo tot een score voor de taakuitvoering te komen. Het vertrouwen in eigen kunnen is gemeten met behulp van een visueel analoge schaal. Dit is een lijn van 10 cm tussen twee polen: ‘ik heb veel zekerheid’ tot ‘ik heb weinig zekerheid’. Studenten geven op deze lijn aan waar ze staan. De tevredenheid is gemeten door middel van een tevredenheidsschaal van vijf vragen.

Het vijfde hoofdstuk over framing handelt over de wijze waarop een boodschap wordt gepresenteerd: is het glas halfvol of halfleeg? In het geval dat het glas halfvol is worden positieve aspecten benadrukt, en als het glas halfleeg is staan de negatieve aspecten

209 Appendices

leerzaam ervaren.

Aan de hand van acht stappen is een vragenlijst ontwikkeld. Deze vragenlijst is afgenomen bij co-assistenten en arts-assistenten uit Nederland (n= 382) en de Verenigde Staten (n = 292). Principale componenten analyse met een varimax rotatie leidt tot een vragenlijst van 46 items, die verdeeld zijn over zeven schalen. Drie schalen hebben betrekking op het gedrag van de feedbackgever: de mate waarin deze doelbewust en betrouwbaar is, de mate waarin deze betrokkenheid toont, en de mate waarin deze afstandelijkheid vertoont. Twee schalen zijn gerelateerd aan de waardering die de feedbackontvanger aan zijn eigen kwaliteiten geeft ten aanzien van de taakuitvoering: een hoge zelf-evaluatie door de lerende, en de mate waarin de lerende zichzelf laag evalueert. Eén schaal betreft feedbackcommunicatie: duidelijkheid van de feedbackboodschap. Drie schalen hebben betrekking op de context waarin de feedback gegeven wordt: de mate van Privacy, de mate waarin de taak een formatief karakter heeft, en de mate waarin de taakuitvoering kritische gevolgen kan hebben voor bijvoorbeeld de patiënt of bijvoorbeeld de arts zelf. De interne consistentie van de schalen varieerde van α = .61 tot α = .75. We zien dat er significante verschillen op deze schalen zijn tussen mannen en vrouwen, tussen Nederland en de Verenigde Staten, tussen co-assistenten en arts-assistenten, en ook tussen diverse medische specialismen.

Verdere validering van deze vragenlijst is nodig om de praktische bruikbaarheid te vergroten.

Appendices

210

centraal. In deze studie is onderzocht wat het effect van de presentatie van een feedbackboodschap is op de tevredenheid van de feedbackontvanger ten aanzien van de feedback, het vertrouwen in het eigen kunnen om een taak uit te voeren en de uitvoering van de stemvorkproeven van Weber en Rinne. In de positieve presentatie was de feedbackboodschap: Je hebt dit goed gedaan, ik heb nog wat tips… In de negatieve presentatie werd gezegd: Dit is niet goed gedaan; dit moet je veranderen… In beide condities krijgen de studenten verbeterpunten te horen. De inhoud van de feedbackboodschap is dus in beide gevallen gelijk. Echter in de ene conditie zijn ze als tips geformuleerd en in de andere conditie als verplichtende punten. Eerstejaars medische studenten (n=59) nemen deel aan dit onderzoek. Studenten in de positief geformuleerde feedbackconditie zijn meer tevreden met de ontvangen feedbacktraining en de feedback. Ze scoren significant hoger in het vertrouwen in hun eigen kunnen gemeten nadat ze feedback kregen. Ze scoren gedurende de hele studie significant hoger qua taakuitvoering dan de studenten in de conditie waarin de boodschap negatief geformuleerd wordt. Door de tijd heen neemt bij beide groepen zowel de taakuitvoering als het vertrouwen in eigen kunnen af (zie figuur 5.2 en 5.3). Het lijkt erop dat positief geformuleerde feedbackboodschappen de taakuitvoering verbeteren. Aanvullende studies zijn nodig om dit te bevestigen.

In hoofdstuk zes is dezelfde gecontroleerde opzet gehanteerd als in hoofdstuk vijf. De interventie in deze studie is de supervisor die een hoog of laag betrouwbare feedbackgever is, ‘credibility’. In de conditie waarin de feedbackgever hoog betrouwbaar is, betreft het een mannelijke trainingsacteur die zich uitgeeft voor een hoogleraar KNO. Hij is formeel gekleed (stropdas, witte lange jas) en hij draagt een badge waarop zijn functie staat. Terwijl hij feedback geeft verwijst hij naar zijn eigen patiëntenpraktijk. In de laag betrouwbaar conditie is de feedbackgever een 20-jarige vrouw. Zij geeft aan dat ze student-assistent is, en dat ze slechts invalt. Zij is informeel gekleed (shirt en spijkerbroek). De betrouwbaarheid van de feedbackgever beïnvloedt het vertrouwen in eigen kunnen niet significant (zie figuur 6.2). De studenten met de hoog betrouwbare feedbackgever zijn meer tevreden met de feedback en de training. We vinden geen significant verschil in de taakuitvoering tussen de twee condities direct nadat de feedback gegeven is. Dit vinden we echter wel na drie weken. De studenten die feedback krijgen van de hoog betrouwbare feedbackgever, percipiëren de feedback vaker als negatief en ze vinden de feedbackgever niet vriendelijk in vergelijking met de studenten die feedback krijgen van de student-assistent.

Het zevende en laatste hoofdstuk vat de resultaten van de studies samen. In de tabel zijn de variabelen weergegeven die het feedbackproces en/of het feedbackeffect beïnvloeden (tabel 7.1). Deze resultaten worden bediscussieerd in het licht van de communicatietheorieën en de sociale perceptietheorie. Van de meeste variabelen die uit metareview naar voren komen zeggen co-assistenten en arts-assistenten dat ze de leerzaamheid van feedback beïnvloeden. Uit dit proefschrift komen diverse aanbevelingen voor feedbackonderzoekers naar voren. Meer aandacht zou aan de onbewuste processen besteed moeten worden, zoals bijvoorbeeld percepties van leerprocessen, percepties ten aanzien van feedbackgevers, en percepties ten aanzien van de communicatie. Vooral de link tussen perceptie en gedrag, dus hoe onze percepties ons gedrag beïnvloeden, is nog weinig onderzocht.

Daarnaast zijn reviews die verschillende velden en perspectieven verbinden belangrijk om het feedbackonderzoek verder te helpen, alsook onderzoek naar het feedbackpro-

211 Appendices

ces, en niet alleen de focus op het feedbackeffect. Het feedbackproces kan behulpzaam zijn bij het verklaren waarom het effect van feedback gering is. Daarnaast is er weinig onderzoek naar de invloed van feedbackgevers en -ontvangers en de doorwerking op het feedbackeffect. Tenslotte is het goed om meer aandacht aan een universele feedbacktaal te besteden, verduidelijking van concepten komt het onderzoek ten goede.

Voor feedbackgevers en -ontvangers wordt afgesloten met twee tips voor de dagelijkse klinische praktijk. Feedbackontvangers moeten de feedbackdialoog stimuleren: feedbackgevers uitnodigen om feedback te geven, en hen aanmoedigen om feedback te expliciteren. Als feedbackontvangers zĂŠlf de regie nemen in het creĂŤren van een followup, door bijvoorbeeld het stellen van doelen en gevolgd door feedback, kan daarmee het effect van feedback worden vergroot.

Feedbackgevers krijgen inzicht in de percepties van de feedbackontvanger als ze vragen hoe de gegeven feedback is overgekomen. Op deze manier wordt nagegaan of de feedbackboodschap begrepen is, en het geeft ook een ingang om stil te staan bij de manier van communicatie door de feedbackgever. Daarnaast is het belangrijk dat de feedbackgever nagaat welke doelen de feedbackontvanger zichzelf heeft gesteld. Alleen wanneer de doelen bekend zijn, kan de feedback daarop worden afgestemd. Op deze manier werken zowel de feedbackgever als de feedbackontvanger beiden aan het verbeteren van de feedbackcultuur op de werkvloer.

Appendices

212

Short biography

ern University Chicago.

In 2008 she moved to the Albert Schweitzer Hospital in Dordrecht as a senior advisor in medical education. Her area of attention was Undergraduate, Graduate and Continuing Medical Education, and especially change management, implementation of new GME-curricula, faculty development, educational quality management, and establishing a feedback culture in the daily practice in the hospital. She is a passionate teacher and enjoys training, supervising Masterstudents, and collaborating with international colleagues in research. In Fall 2015 she hopes to move to Michigan State University in Grand Rapids, Michigan.

Feedback research in Medical Education is her area of expertise. She is interested in the variables that influence the feedback process and the feedback effect, the history and the development of feedback research, the relationship between feedback perception and behavior. Further she is interested in variables that influence the feedback culture, and the practical question on how to create a feedback culture in the wards, department and the hospital.

In her spare time she is member of Platform Mediawijzer, a foundation that focuses on media, family and education. She likes traveling, long distance walking, listening to classical music, reading, cooking and spending time with family and friends.

213 Appendices

Monica van de Ridder was born in 1973 in The Nederlands. After finishing Teacher Training College, she studied Educational Sciences at Utrecht University (with honors cum laude).

From 1996-1999 she was a teacher of Dutch Languages in vocational education.

From 2001-2010 she worked at the Medical School of the University Medical Center Utrecht. She taught courses on doctor-patient communication skills, didactic courses for senior medical students on supervision skills, and faculty development courses on small group teaching and assessment. Further she coordinated the Utrecht Progress Test.

In 2006 she initiated the Medical Education network for Dutch and Flemish PhD-students.

In 2006 and 2007 she was a visiting scholar at the Feinberg School of Medicine, Northwest-