European journal of obstetrics gynecology and reproductive biology 2017 217 (1) 154

European Journal of Obstetrics & Gynecology and Reproductive Biology 217 (2017) 154–160

Contents lists available at ScienceDirect

European Journal of Obstetrics & Gynecology and Reproductive Biology journal homepage: www.elsevier.com/locate/ejogrb

Full Length Article

Improving non-technical skills (teamwork) in post-partum haemorrhage: A grouped randomised trial Pippa M. Letchwortha,b,* , Shane P. Duffya,b , Dan Phillipsc a

Chelsea and Westminster Hospital Foundation Trust, London UK CW+ New Futures Group, Healthcare Charity associated with Chelsea and Westminster Hospital, UK c Service Design, Sustainable Innovation, Royal College of Art, UK b

A R T I C L E I N F O

A B S T R A C T

Article history: Received 21 April 2017 Received in revised form 14 August 2017 Accepted 22 August 2017

Objective: To determine the effect of a decision support technology on teamwork and associated non-technical (NTS) and technical skills when teams manage post-partum haemorrhage (PPH) in the simulated environment.

Methods: Multidisciplinary (MDT) maternity teams were taught how to manage post partum haemorrhage. They were randomised to the intervention: using a decision support mobile digital platform or a control group. Each team managed a post-partum simulation, which was recorded and reviewed by assessors.

Primary outcome measures to assess teams NTS were the validated Global Assessment of Obstetric Team Performance (GAOTP) and Clinical Teamwork Scale (CTS). Secondary outcome measures were the ‘friends and family test', technical skills, and the System Usability Scale (SUS). Sample size estimation was calculated by using 80% power 5% significance two tailed test (p1 = 85% p2 = 40%) n = 34.

Results: 38 teams from August 2014–February 2016, were recruited, technical issues with failure of recording equipment meant 4 teams were excluded from teamwork analysis (1 intervention 3 control).

Teamwork improved across all domains with the intervention (using a decision support mobile digital platform) p < 0.01. CTS improved between 6.7–16.8% (average 14.2%) and GAOTP between 8.6–17.1% (average 13.5%) for all domains. Using the control group as baseline, the intervention improved teamwork by 25% using CTS and 22% using GAOTP. Fewer technical skills were missed with the intervention (p < 0.01). There was no statistical difference in the time technical skills were achieved.

Assessors were more likely to recommend intervention teams 87.5% (77/88) than control teams 63.6% (56/88) p < 0.01 to their friends or family.

The SUS was ‘Good’ (69) becoming excellently ‘Usable’ (81.6) over the study period.

Conclusion: We report a decision support system, which improved NTS when managing PPH. Lack of teamwork is often cited as the cause of failures in care and we report a usable technology that assists with and improves teamwork during an emergency. © 2017 Elsevier B.V. All rights reserved.

Keywords: Teamwork Non-technical skills Technology Postpartum haemorrhage Maternity Emergency

Introduction Confidential enquiries in Obstetrics and Gynaecology have long sited substandard care as contributory to cases of maternal mortality [1,2]. When further analysis is done regular failures in care are reported including: failure to act, recognise, refer, poor or non-existing teamwork and failures in communication. The

* Corresponding author at: Women’s Services, Chelsea and Westminster Hospital, 369 Fulham Road, London, SW10 9NH, UK. E-mail address: Pippaletchworth@hotmail.com (P.M. Letchworth). http://dx.doi.org/10.1016/j.ejogrb.2017.08.030 0301-2115/© 2017 Elsevier B.V. All rights reserved.

response to these failures has been to produce guidelines focusing on technical management, which are intended to facilitate optimal care. These guidelines have become lengthy electronic tomes and remain as reference tools on computers. For example the full intrapartum NICE guidance [3] of December 2014 was 839 pages, with a list of recommendations 108 pages long. Deviations from the protocols of resuscitation by healthcare professionals in clinical domains are between 16 and 55% [4–6].

A mobile digital platform has been developed to combat these issues with guidelines and failures in teamwork and associated non-technical skills (NTS). In the same way that satellite navigation devices have replaced maps on journeys, MedNav has been

P.M. Letchworth et al. / European Journal of Obstetrics & Gynecology and Reproductive Biology 217 (2017) 154–160

developed to replace paper-based checklists and guidelines. This acts as a navigation device for medical emergencies that integrates the latest guidelines into clinical practice. It has been designed to delegate teams, prompt next steps and record/scribe what happens as the emergency is managed; a paradigm shift from reporting failures in teamwork to working towards solutions that improve it.

MedNav is a decision support tool that can be used on a tablet or mobile phone and has integrated vocal prompts and visual cues (Fig. 1).

The null-hypothesis is that there is no difference in teamwork and associated NTS when the decision support technology is used to manage post-partum haemorrhage in the simulated environment.

Materials and method Teams of maternity health care professionals at the Chelsea and Westminster Hospital attended mandatory departmental training, day one involves post partum haemorrhage drills, the following day their skills are evaluated in a full platform simulation. Individuals were recruited into a randomised control trial at a ratio of 1:1. They were clustered into skill set (junior or senior midwife (<3 or 3 years of experience), junior or senior obstetrician or anaesthetist). To match for skill set the simulation faculty performed simple randomisation using folded pieces of paper and randomly allocated participants to intervention or control. Teams of six, seven or eight (matched for number) personnel underwent the same simulated scenario using full platform simulation simMOM model (Laerdal) to manage a brisk atonic post-partum haemorrhage (see Fig. 2). As the control group were undergoing simulation the intervention group were introduced to the technology and taught how to use it via a 15-min tutorial. Both groups were kept isolated from the simulation and each other until the scenarios had ended.

The scenarios were recorded in real time and then the videos reviewed by a team of assessors in a departmental training room under research conditions. The assessors were eight professionals with experience in education and had the following roles: Consultants: Anaesthetics and Obstetrics, Practice Development Midwifes (PDM) and Obstetric Registrars. The assessors were taught how to use the Non-technical skills (NTS) assessment tools Clinical Teamwork Scale (CTS [7]) and Global Assessment of

155

Obstetric Team Performance (GAOTP [8]). The scoring system for GAOTP is 1–5 (where 1 was poorest and 5 highest) and, for CTS, 1–3 (poor), 4–6 (average), 7–9 (good) and 10 (perfect).

Both CTS and GAOTP are validated [7,8] and separate NTS into certain domains: communication, situational awareness, decision making or task/case management, role responsibility, overall teamwork or environment in the room and patient communication/friendliness.

The primary outcome measures were the teams total CTS and GAOTP score from all assessors as per NTS domain. Secondary outcome measures were technical skills completion, System Usability Score [9–12] and ‘friends and family’ test adapted from NHS England’s patient survey questionnaire.

The System Usability Score (SUS) is a scoring system, validated at over 15 responses, which enables evaluation of technologies [9– 11]. Over a one year period the intervention group were asked to complete the usability questionnaire which was then converted into a System Usability Score (SUS) [9]. A score between 0 and 100 is given (not a percentage). A score over 68 is considered above average [12]. The friends and family test was asked to all assessors for the last 22 scenarios.

During the scenario, a member of faculty recorded, in real time, the time from declaration of emergency to completion of key technical skills. This list was based on previous audit data from the UK Clinical Negligent Scheme for Trusts standards and other research papers. The number of missed skills was noted for each scenario and each missed skill was given a 25-min time penalty.

Data was analysed with and without this time penalty.

Estimated sample size was based on previously unpublished work with the technology which had demonstrated improvement in percentage scores between the ‘case’ and the ‘control’ group by 45%. The sample size was calculated using a two sample Binomial (arcsine approximation) differences in percentage scores between the two groups using with 80% power and 5% significance level using a two tailed test (p1 = 85% p2 = 40%) total number of teams required is 34.

Statistical analysis Using IBM SPSS Statistics versions 23, the data from an Excel file was imported into SPSS. The descriptive statistics, mean, standard deviation, minimum and maximum were calculated for the total scores from all the

Fig. 1. Managing a simulated massive obstetric haemorrhage with the decision support tool.

156

P.M. Letchworth et al. / European Journal of Obstetrics & Gynecology and Reproductive Biology 217 (2017) 154–160

Fig. 2. Consort flow diagram.

assessors for each NTS domain and for the timing for each technical skill.

Analysis of the NTS was performed by separating the data by total score from all assessors for each NTS domain per team, using a

Student’s T-Test for independent samples, equal variances not assumed.

The total scores for each NTS for GAOTP and CTS for all the scenarios were combined and calculated as a percentage of the

Table 1 Descriptive statistics, independent samples t-test comparing equality of means and percentage difference for Global Assessment of Obstetric Team Performance (GAOTP) scores broken down into each nontechnical skill with the intervention and the control groups.

NTS Domains Global Assessment of Obstetric Team Performance (GAOTP)

Simulation Group (n = number of teams)

Total number of Mean and scenarios assessed (Standard deviation)

Range t-test % % Change with % improvement p value Score intervention (control as baseline)

Communication with Patient

Intervention (18) Control (16)

144 128

3.51(1.10) 3.09(1.17)

1–5 1–5

<0.01

70.3 61.7

+8.6

+13.9

Case Management

Intervention (18) Control (16)

144 128

3.79(1.00) 2.94(0.95)

1–5 1–5

<0.01

75.8 58.8

+17.1

+29.0

Teamwork

Intervention (18) Control (16)

144 128

3.78(0.94) 3.06(1.01)

2–5 1–5

<0.01

75.6 61.3

+14.3

+23.4

Situational Awareness

Intervention (18) Control (16)

144 128

3.75(1.01) 3.03(1.02)

1–5 1–5

<0.01

75.0 60.6

+14.4

+23.7

Communication with Team

Intervention (18) Control (16)

144 128

3.76(0.96) 2.98(1.00)

2–5 1–5

<0.01

75.1 59.7

+15.5

+25.9

Environment in room

Intervention (18) Control (16)

144 128

3.83(1.00) 3.26(1.00)

1–5 1–5

<0.01

76.7 65.2

+11.5

+17.7

Total GAOTP Score

Intervention Control

144 128

22.42(5.10) 18.36 5.33

8–30 5–30

na

74.8 61.2

+13.6

+22.1

P.M. Letchworth et al. / European Journal of Obstetrics & Gynecology and Reproductive Biology 217 (2017) 154–160

maximum possible score (i.e. all values equalling 5 or 15). The resulting percentage for each NTS in the control group was subtracted from the intervention; these differences between the percentages were quoted as the percentage change of the GAOTP and CTS scores. The percentage improvement, using the control group as a baseline, has also been reported.

Student’s paired-sample T-Test was used to check for any differences between the total times taken for each skill and for the number of missed skills.

Analysis of the technical skills was performed using the univariate Analysis of Variance (univariate ANOVA) procedure to check for any differences in the variances of the times taken for each of the 11 skills with and without the use of the intervention.

The post hoc test Tukey’s honestly significant difference was used to identify any significant differences between these groups.

Analysis of the ‘friends and family’ test was performed using an unpaired Students t-test, equal variance not assumed, comparing intervention teams or control.

This study was reviewed and approved by the R&D Team at Chelsea and Westminster NHS Foundation Trust R&D Ref: C&W14/ 094. NHS Permission was granted on 13/11/2014. Study considered not research by the NHS IRAS and therefore does not require IRAS approval.

157

Results Between October 2014 and February 2016, 296 individuals were recruited and allocated into 38 teams at Chelsea and Westminster Centre for Clinical Practice. For four simulations technical errors meant the scenario was not recorded and thus no video footage was available for analysis. There were no losses to follow up in the analysis of technical skills as the data was collected in real time as the scenario played out.

Tables 1 and 2 demonstrate the descriptive statistics and independent samples t-test comparing equality of the means (equal variance assumed) and the percentage difference in score for each GAOTP and CTS domain with the control and intervention.

All the values when intervention has been used are higher than with control groups.

The mean GAOTP and CTS scores were higher for each NTS with the intervention group compared to those in the control independent samples t-test was highly significant statistical difference, p < 0.01 for all domains. Levene’s test indicates that there is no significant difference between each pair of variances as all results >0.05 in each domain of the GAOTP. The improvement with the intervention was between 8.6 and 17.1% for all the domains of GAOTP (p < 0.01).

Table 2 Descriptive statistics mean and standard deviation of Clinical Teamwork Scales (CTS) values for Each CTS domain Independent Samples t-test and percentage difference in CTS Score with intervention and control group.

Clinical teamwork domain (CTS)

Simulation Group (n = number of teams)

Assessments

Mean (standard deviation)

Range

t-test p-value

% Score

% Change with intervention

% improvement (control as baseline)

Overall Teamwork

Intervention (18) Control (16)

144 128

7.24(1.72) 5.91(1.84)

2–10 2–10

<0.01

72.4 59.1

+13.3

22.5

Overall Communication

Intervention (18) Control (16)

144 128

7.15(1.72) 5.85(1.89)

2–10 2–9

<0.01

71.5 58.5

+12.9

22.1

Orient new members

Intervention (18) Control (16)

144 128

6.98(1.81) 5.75(1.89)

2–10 1–9

<0.01

69.8 57.5

+12.3

21.4

Transparent thinking

Intervention (18) Control (16)

144 128

7.34(1.77) 5.66(1.97)

3–10 2–9

<0.01

73.4 56.6

+16.8

29.6

Directed communication

Intervention (18) Control (16)

144 128

7.08(1.73) 5.52(2.07)

2–10 1–10

<0.01

70.8 55.2

+15.5

28.1

Closed Loop communication

Intervention (18) Control (16)

144 128

5.87(1.88) 4.20(1.91)

1–10 0 9

<0.01

58.7 42.0

+16.7

39.6

Overall Situational Awareness

Intervention (18) Control (16)

144 128

7.31(1.59) 5.73(2.01)

3–10 2–9

<0.01

73.1 57.3

+15.8

27.5

Resource allocation

Intervention (18) Control (16)

144 128

7.21(1.78) 5.68(1.94)

2–10 2–10

<0.01

72.1 56.8

+15.3

26.9

Overall Decision making

Intervention (18) Control (16)

144 128

7.42(1.59) 6.03(1.85)

3–10 2–9

<0.01

74.2 60.3

+13.9

23.0

Prioritisation

Intervention (18) Control (16)

144 128

7.33(1.68) 5.91(1.85)

2–10 2–9

<0.01

73.3 59.0

+14.2

24.0

Overall Role Responsibility

Intervention (18) Control (16)

144 128

7.34(1.87) 5.79(2.04)

2–10 1–9

<0.01

73.4 57.9

+15.5

26.8

Clarity

Intervention (18) Control (16)

144 128

7.02(2.12) 5.59(2.05)

1–10 2–10

<0.01

70.2 55.9

+14.3

25.5

Performance as leader/helper

Intervention (18) Control (16)

144 128

7.33(1.94) 5.79(2.07)

2–10 1–10

<0.01

73.3 57.9

+15.4

26.6

Patient friendly

Intervention (18) Control (16)

144 128

6.82(1.92) 6.15(2.21)

1–10 1–10

<0.01

68.2 61.5

+6.7

10.9

14.2

25.3

Overall

158

P.M. Letchworth et al. / European Journal of Obstetrics & Gynecology and Reproductive Biology 217 (2017) 154–160

The mean GAOTP score for each NTS domain given by each assessor is demonstrated in Table 3, for each pair (apart from one), the value for the intervention group is higher than for the control.

When the independent samples t-test, equal variances not assumed if Levene’s test sig < 0.05 if >0.05 equality of variance assumed was used to compare the CTS scores for each NTS with the control and intervention group, there was a statistically highly significant difference, p < 0.01, for the topics 1 to 14, Table 2.

Calculating percentage change as for the CTS, gave between 6.7 and 16.8% improvement with the intervention for all domains Table 2 (p < 0.01.) This equates to an increase in CTS of 25% and GAOTP of 22%.

The friends and family test was asked during the analysis of last 22 scenarios (=176 questions) all responded. Of these eligible responses to 87.5% (77/88) would recommend the intervention teams and 63.6% (56/88) would recommend the control teams to manage their friends or family, a t-test with unequal variance assumed gave p < 0.01.

The descriptive statistics for the times taken for each technical skill with and without the use of the intervention are shown in Table 4. Timings started when the candidate in the simulation declared the emergency, thus some actions occurred at time 0. This gives the results including the missed skills times of 25 min.

Results for actual recorded (missed skills not included) times were also analysed.

The results of the univariate ANOVA and Tukey’s homogeneous subsets, show that none of the skills show a significant difference in the times taken with and without the intervention as each pair appears in the same column at least once.

Boxplots showing the five statistics (minimum, first quartile, median, third quartile, and maximum) Fig. 3, including 25-min time penalties for missed skills. The graphs display the distribution of the skill times and also pinpoint the outliers. There are no significant differences between the timings.

The total numbers of missed skills were calculated for the skills in the control and intervention group, Fig. 4. Independent samples T-test p < 0.01, with fewer missed skills when using the intervention vs the control group at 5.2% and 14% respectively.

System usability data was collected from Oct 2014- October 2015, 83 of the eligible individuals that used the decision support tool were asked and responded to the questionnaire. The score was

initially 69 (above good) and this climbed to 81.6 (towards excellent) over the study period.

Comments Non-technical skills improved across all domains in this study comparing teams using the technology (Mednav) to those in control groups. This was a reproducible finding using two different validated teamwork scoring techniques [7,8]. These improvements were also independent of the assessors, who were of multi-disciplinary background. Using CTS, improvements varied between 6.7 and 16.8% (overall 14.2%), using GAOTP they varied from 8.6 and 17.1% (overall 13.5%). The greatest improvements were seen in ‘case management’ and ‘transparent thinking’. These aspects of NTS were found to improve by 17.8% and 16.8% respectively. Comparing these improvements to the teams that actually manage post partum haemorrhage (those in our study in control group) we find the improvement in CTS of 25% and in GAOTP of 22%. NTS have been traditionally analysed using qualitative methods, with this study we are attempting to use quantitative methods. Thus the clinical impact of this improvement is unknown. However combining this information with our secondary outcomes (less missed technical skills and friends and family test) further strengthens the weight of this improvement. This was first use of the technology for all the clinicians; with increased familiarity of the technology these improvements will inevitably increase.

In view of the ethics of introducing new point of care technology in the clinical domain, simulated scenarios were used, in awareness that this is the simulation environment. We found our team of assessors more likely to recommend the teams using the technology to manage their friends or family. The technology was found to be a ‘good’ to ‘excellently’ useable piece of technology, implying that it is an intuitive piece of technology.

Teams were less likely to miss key steps in the management of post-partum haemorrhage when the decision support tool was used compared to control (p < 0.01) with the percentage of missed events being 5.2% vs 14% respectively.

Patient safety is improved by the support of checklists, as seen with introduction of WHO surgical checklist [13]. Our study supports this, three (of nineteen) teams in the control group did not insert a urinary catheter when managing an atonic

Table 3 Mean Global Assessment of Obstetric Team Performance (GAOTP) Score for each non-technical skill as evaluated by each Assessor for teams using intervention or control groups.

Non-Technical Skill

Group

Number of GAOTP assessments made

Assessor 1

Assessor 2

Assessor 3

Assessor 4

Assessor 5

Assessor 6

Assessor 7

Assessor 8

Communication with Patient

Intervention Control

18 16

2.83 2.94

4.11 3.25

3.61 3.25

3.11 3.00

4.00 3.25

3.50 3.06

3.28 2.50

3.67 3.44

Case Management

Intervention Control

18 16

3.67 3.25

4.17 3.00

3.78 3.06

4.22 2.69

3.89 3.31

3.56 2.75

3.50 2.56

3.56 2.88

Teamwork

Intervention Control

18 16

3.61 3.44

4.17 3.00

3.83 3.31

3.94 2.75

3.72 3.38

3.61 2.94

3.72 2.88

3.61 2.81

Situational Awareness

Intervention Control

18 16

3.67 3.06

4.17 2.94

3.72 3.06

4.00 2.88

3.89 3.69

3.56 2.94

3.44 2.94

3.56 2.75

Communications with Team

Intervention Control

18 16

3.78 3.44

4.28 2.94

3.67 3.19

3.94 2.81

3.89 3.25

3.39 2.88

3.44 2.56

3.67 2.81

Environment in Room

Intervention Control

18 16

3.50 3.19

4.00 3.31

3.78 3.63

4.06 3.44

4.22 3.75

3.61 3.00

3.83 2.88

3.67 2.88

Total Score

Intervention Control

18 16

21.06 19.31

24.89 18.44

22.39 19.50

23.28 17.56

23.61 20.63

21.22 17.56

21.22 16.31

21.72 17.56

P.M. Letchworth et al. / European Journal of Obstetrics & Gynecology and Reproductive Biology 217 (2017) 154â&#x20AC;&#x201C;160

159

Table 4 Descriptive Statistics for times taken for Each Technical Skill achieved including Missed Skills with intervention or control.

Technical skill

Intervention

N = number of teams

Mean time from emergency being declared (secs) and Standard deviation

Minimum time (secs)

Maximum time (secs)

Massive Obstetric Haemorrhage call

Intervention Control

19 19

153 135 221 331

0 0

394 1500

Explore perineum for trauma

Intervention Control

19 19

147 149 341 440

0 0

510 1500

Palpate Uterus

Intervention Control

19 19

45 99 43 114

0 0

384 500

Examine Placenta

Intervention Control

19 19

293 374 415 500

0 19

1500 1500

Intravenous Fluids Started

Intervention Control

19 19

463 278 426 113

224 260

1500 635

Uterotonics Started

Intervention Control

19 19

337 307 238 116

62 0

1500 494

Bloods taken and sent

Intervention Control

19 19

447 293 455 291

140 152

1500 1500

Urinary Catheter Inserted

Intervention Control

19 19

434 309 574 510

125 75

1500 1500

Bimanual Compression Started

Intervention Control

19 19

218 174 220 163

5 0

730 523

Blood Transfusion

Intervention Control

19 19

719 305 1028 472

373 370

1500 1500

Decision to Move to Theatre

Intervention Control

19 19

768 380 1020 530

285 340

1500 1500

post-partum haemorrhage, this error occurred once with the intervention group (due to oversight by the faculty.)

There was an inability to blind the assessors to the intervention, which may produce bias, however we chose independent

assessors, two different assessment tools, and more assessors than the tools were validated for and found reproducibility of the outcome. Sample size calculation was done incorporating ďŹ ndings from previously unpublished work with the technology. We

Fig. 3. Box plot diagram showing the times taken to complete each skill including missed skills intervention (mednav) vs control groups.

160

P.M. Letchworth et al. / European Journal of Obstetrics & Gynecology and Reproductive Biology 217 (2017) 154–160

Fig. 4. Box Plot diagram showing the number of missed technical skills when managing post-partum haemorrhage intervention (mednav) vs control.

acknowledge that this data should have been published prior to this study commencing.

Potential bias could occur by having teams with differing clusters of skill set. To mitigate this bias we clustered the individuals into skill set then randomly allocated them to the control or intervention. Four teams were lost to follow up, three from the control group and one from the intervention group, this was related to problems with recording technology and was completely random.

The use of scales as markers of good teamwork is an evolving area of medical education with frameworks derived to provide assistance in developing these scales [14]. Different disciplines [15,16] have developed there own scores that have become a part of training and reflection [15–18] rather than objective testing.

As decision support technology advances, tools such as MedNav will inevitably be key to aiding the clinician in providing safer medical care. The introduction of technology into the clinical domain requires rigorous testing in terms of outcome and usability, as the authors have demonstrated. Technologies have the opportunity to improve care, however, more research is needed to aid the development and assessment of these tools in the clinical environment.

We report the first example of a decision support tool for the management of post-partum haemorrhage. Confidential enquiries continue to report failures in teamwork as contributory to poor outcomes, the addition of technology into our clinical domain could be the paradigm shift that is required to act against these failures rather than reporting them, improving safety for pregnant women.

Funding MedNav is funded by CW+ and their enterprising health care partnership; this is a partnership between the charity CW+ and the Chelsea and Westminster Healthcare NHS Foundation Trust, London.

References [1] Knight M, Kenyon S, Brocklehurst P, Neilson J, Shakespeare J, Kurinczuk JJ, on behalf of MBRRACE-UK, editors. Saving Lives, Improving Mothers’ Care – Lessons learned to inform future maternity care from the UK and Ireland Confidential Enquiries into Maternal Deaths and Morbidity 2009-12. Oxford: National Perinatal Epidemiology Unit, University of Oxford; 2014.

[2] Centre for Maternal and Child Enquiries (CMACE). Saving mothers’ lives: reviewing maternal deaths to make motherhood safer: 2006–08. the eighth report on confidential enquiries into maternal deaths in the United Kingdom.

BJOG 2011;118(Suppl. 1):1–203.

[3] Intrapartum care | Guidance and guidelines | NICE [Internet]. Nice.org.uk. 2015 [cited 19 April 2016]. Available at: https://www.nice.org.uk/guidance/qs105.

[4] Chitkara R, Rajani A, Oehlert J, Lee H, Epi M, Halamek L. The accuracy of human senses in the detection of neonatal heart rate during standardized simulated resuscitation: implications for delivery of care, training and technology design.

Resuscitation 2013;84(3):369–72.

[5] Mitchell A, Niday P, Boulton J, Chance G, Dulberg C. A prospective clinical audit of neonatal resuscitation practices in Canada. Adv Neonatal Care 2002;2 (6):316–26.

[6] Thomas E, Sexton J, Lasky R, Helmreich R, Crandell D, Tyson J. Teamwork and quality during neonatal care in the delivery room. J Perinatol 2006;26(3): 163–9.

[7] Guise J, Deering S, Kanki B, Osterweil P, Li H, Stanford S, et al. STORC OB safety initiative: validating a tool to evaluate team performance. American Journal of Obstetrics and Gynecology. 2006;195(6):S216.

[8] Morgan P, Tregunno D, Pittini R, Tarshis J, Regehr G, Desousa S, et al.

Determination of the psychometric properties of a behavioural marking system for obstetrical team training using high-fidelity simulation. BMJ Quality & Safety. 2011;21(1):78–82.

[9] Brooke J. SUS: a quick and dirty usability scale. 1st ed. In: McClelland L, Jordan W, Thomas B, Weerdmeester A, editors. Usability Evaluation in Industry, vol. 9.

Jordan P. London: Taylor & Francis; 1996. p. 189–94.

[10] Brooke J. SUS: a retrospective. Journal of Usability Studies 2013;8(2)29–40.

[Internet]. Usabilityprofessionals.org. Available at: Retrieved 19 April 2016 http://www.uxpajournal.org/sus-a-retrospective/.

[11] Sauro J. A practical guide to the system usability scale. Denver, CO: CreateSpace; 2011.

[12] Aaron Bangor C, Philip Kortum P, James Miller P. Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale JUS. 2016. [Internet].

Uxpajournal.org. 2016 [cited 19 April 2016]. Available from: http://www. uxpajournal.org/determining-what-individual-sus-scores-mean-adding-anadjective-rating-scale/.

[13] Haynes A, Weiser T, Berry W, Lipsitz S, Breizat A, Dellinger E, et al. A surgical safety checklist to reduce morbidity and mortality in a global population.

NEJM 2009;360:491–9.

[14] Grand J, Pearce M, Rench T, Chao G, Fernandez R, Kozlowski S. Going DEEP: guidelines for building simulation-based team assessments. BMJ Q Saf 2013;22(5):436–48.

[15] Royal college of surgeons of edinburgh. The Non-Technical Skills for Surgeons (NOTSS) Handbook v1.2. 1st ed. Aberdeen: University of Aberdeen Press; 2006.

[16] Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R. Rating non-technical skills: developing a behavioural marker system for use in anaesthesia.

Cognition, Technology & Work 2004 6(3)..

[17] [Internet]. 2016 [cited 5 May 2016]. Available from: http://www.rcog.org.uk/ globalassets/documents/careers-and-training/assessment-and-progressionthrough-training/notss-labour-ward-guidance.pdf.

[18] Jackson S, Brackley K, Landau A, Hayes K. Assessing non-technical skills on the delivery suite: a pilot study. Clin. Teacher 2014;11(5):375–80.