and during transition to the operating room. Because workload impacts performance, and this tool accurately provides performance information not otherwise available to learners and trainers, it should be incorporated into simulator training surgical skills assessment. • Performance Anxiety and Physiologic Measures of Performance Mental stress is a possible factor for technical errors and inferior performance by surgeons, perhaps more so during minimally invasive than during open procedures. Because stress is difficult to evaluate subjectively, several studies have successfully used physiologic measures (heart rate) to objectively indicate stress. Aviation studies have concluded that heart rate is the most useful psychophysical variable to assess pilot workload and mental strain. A recent study linked incomplete transfer of simulator acquired skills to the operating room with a significant increase in the surgical trainee’s heart rate in the operating room compared with the simulator. Because these metrics may provide additional information on learner performance, they should also be considered during simulator training.
Summary Many factors have been shown to optimize surgical skills curricula and the trainee’s learning so that learner proficiency can be achieved. Available studies regarding the best performanceassessment methods suggest that the incorporation of additional, more sensitive performance metrics may improve skill transfer. Simulator curricula that take into account all the factors discussed here can optimize skill acquisition and learner readiness for success in the operating room. medsim About the Author Dr. Dimitrios Stefanidis is Director, Carolinas Simulation Center, Carolinas HealthCare System, Charlotte, North Carolina. He may be contacted at email Dimitrios.Stefanidis@carolinashealthcare.org. Dr. Stefanidis attended Aristotelian University of Thessaloniki-Greece, completed a General Surgery Residency at the University of Texas, San Antonio and Fellowships at Tulane University in Minimally Invasive Surgery and the Carolinas Laparoscopic and Advanced Surgery Program.
ISSUE 1.2012
assessment relies on subjective ratings, the instruments must have proven reliability and validity before they are used for assessment. Global rating scales are superior to checklists for technical skill evaluation when completed by experts. validated rating scales for technical skill assessment and global operative assessment of laparoscopic skills should be incorporated into simulator training. However, the relationship between this assessment type and other more objective performance metrics is not well studied. • Automaticity Automaticity is the ability to perform motor acts automatically, leaving enough attentional capacity to engage in other activities. Automaticity is a characteristic of an expert performer and has been used in the literature to confirm learning by novices. Novices practicing a new task often operate using maximum attentional capacity and cannot attend to other stimuli in their environment. To accurately measure automaticity and spare attentional capacity, a secondary task must compete for the same attentional resources as the primary task. Measuring the performance on the secondary task reflects how much attention can be spared for the primary task. Several previous studies show that the traditional metrics of time and errors are good performance measures during the early learning stages; however, more sensitive performance metrics such as a secondary-task are needed for complete assessment of performance. • Workload Assessment Learner performance can be influenced by task workload, performance anxiety, and stress. Task workload, which can increase operator fatigue and frustration and compromise attention span, is higher during early learning and decreases with experience. With high workload, the ability to deal with unexpected demands can be impaired and performance errors may increase. The National Aeronautics and Space Administration-Task Load Index (NASATLX) tool, first used in flight simulation, is a validated tool for workload self assessment, measures a task’s mental, physical, and temporal demands and the effort, frustration, and perceived performance of the trainee on a 20-point visual analog scale. Evidence suggests that the NASA-TLX tool provides a reliable measure of workload, task difficulty, and learner comfort during simulator training
15 MEDSIM MAGAZINE
Traditional training paradigms such as time-based curricula (setting a specific training duration) and repetition-based curricula (setting a minimum number of repetitions) do not take into account individual learning differences and use arbitrary training endpoints. Given that learners have different baseline abilities, experiences and motivation, such curricula can lead to inadequate training or overtraining. Empiric evidence supports the superiority of proficiency-based training. The way goals are defined may also be important for skill acquisition. • Performance Assessment Many questions regarding expertise in surgery need to be answered using high quality studies. How should expertise in surgery be defined and measured? How should proficiency levels based on expert performance be established? How should acquired expertise on simulators be detected in learners? Should expert levels be used as simulator-training endpoints or are less difficult performance levels more appropriate? Only with appropriate metrics to identify superior performance can these questions be answered. Traditionally, skills curricula have used the easily obtainable metrics of task duration and errors for performance assessment; however, this provides no insight into the effort invested to achieve the performance goal or whether the learning has been completed and can produce a misleading picture of the trainee’s readiness to transition to the more stressful clinical environment. Several studies have shown that surgical trainees achieved expert-level performance based on time and error metrics on simulators, but their performance fell short of expert skill level in the stressful conditions of the operating room. This could be related to difficulties of assessing when simulator learning is complete. More sensitive performance metrics such as limb kinematics, global rating scales, psychophysiologic measures, and measures of mental workload may provide complementary performance assessment to augment skill acquisition and transfer. • Observer Ratings Surgical performance can be reliably assessed by an experienced observer using global rating scales, visual analog scales, checklists, or a combination of these. These instruments are versatile, and some can be used for similar tasks. However, since the