How do we ensure that the NJR holds high quality data?

Chris Boulton, Derek Pegg, Tim Wilton and Mark Wilkinson, on behalf of the NJR

Chris Boulton is the Deputy Director of Operations of the National Joint Registry.

Derek Pegg is a Consultant Trauma and Orthopaedic Surgeon in Mid Cheshire Hospitals Foundation Trust, Leighton Hospital, Crewe. He is Chair of both NJR Regional Clinical Coordinators Committee and NJR Data Quality Committee.

Tim Wilton is a Consultant Orthopaedic Surgeon at Department of Orthopaedics, Royal Derby Hospital, Derby, with specialist interests in knee and hip replacement surgery. He is Past President of the BOA and BASK and is Medical Director of the NJR.

Mark Wilkinson is Professor of Orthopaedics at the University of Sheffield and an Honorary Consultant Orthopaedic Surgeon at Sheffield Teaching Hospitals. He is Chair of the Research Committee at the NJR.

Data of high quality is critical to meaningful interpretation of registry information. The National Joint Registry (NJR) currently has a compliance accuracy of 97.9% and continues auditing year-on-year to maintain and improve this degree of correctness. The NJR was established in 2002 in response to an unexpectedly high failure rate of a cemented total hip replacement.

In this short article we describe the establishment of a formal data quality audit process within the NJR that enables bi-directional validation of every NJR entry and retrospective correction of any identified errors. This process has led to an increase in overall baseline compliance from 92.6% in 2014/2015 to 97.9% in 2018/19 period, with 91.5% of units ultimately achieving >95% compliance as of October 2022. We also briefly outline the further automation of the process that was initiated in 2018, to reduce hospital administrative burden and to integrate the data quality process into routine workflows. Our quality improvement results demonstrate that the process may be implemented successfully at national level, whilst minimising the burden on hospitals.

Why is good quality data important?

Good clinical audit requires good quality data. In the setting of joint replacement registries, key quality metrics include: compliance –the proportion of procedures performed that are entered onto the registry and measured against independently-collected administrative data; consent – the proportion of patients undergoing those procedures who have consented for their personal identifiers to be used by the registry; and linkability to outcomes events – the presence of a valid set of identifiers that can be used to match the record to revision and mortality events [1]. Here, we briefly describe the effect of the implementation of the NJR’s formal data quality audit (DQA) programme on data compliance.

Where did we start on the data quality pathway?

In the early years of the NJR our funding model was based on a levy system in which orthopaedic implant manufacturers collected a fee for each construct they sold and this was passed on to the NJR. Comparison of sales numbers with the corresponding records in the NJR gave a rough estimate of the registry completeness, but was unable to distinguish within year rates of compliance nor differentiate between primary and revision procedures. An alternative comparator was therefore needed. The Hospital Episode Statistics (HES) dataset, maintained by NHS Digital, has been used for this purpose for English NHS organisations since 2006. This comparison of data entry between the NJR and HES data gave a clear indication of incomplete data, but did not establish a mechanism for missed cases to be retrieved. It was also unable to provide the data for the examination of independently funded procedures. For this reason, a formal audit cycle capable of reconciling the two sources of data and allowing their correction was set up. Here, the NJR dataset is compared directly with each NHS organisation’s Patient Administration System (PAS) and each independent sector organisation’s business administration system, and vice versa.

How did we put the strategy into practice?

A Data Quality Committee (DQC) was established to determine and oversee the strategy, with the support of NHS England, patient representatives, and other key stakeholders. All organisations contributing data to the NJR were sent an annual statement showing cases found on the registry but missing from that unit’s PAS data, along with a similar list

of those cases found in the PAS data but not in the NJR data. Data entry managers at each organisation were asked to check the individual unit records and upload the correct data to the NJR. Each unit worked with the NJR to share data and identify areas for improvement. An overview of the process is shown in Figure 1.

Figure 1: Overview of NJR data quality strategy.

Process and procedure documents were agreed for each step of the audit process undertaken by the NJR team and the unit-nominated Data Quality (DQ) lead. The audit tool was finalised as a means to 1) semi-automate the process of validating returned audit data, and 2) provide a mechanism to track progress metrics against individual unit records. This enabled clear audit status reporting back to the NJR DQC. A compliance audit report was developed and sent to the Chief Executive Officer (CEO) and Clinical Lead for each organisation containing the key findings, recommendations and additional learning points from the audit process. Thus, the report provided each organisation with their own key learning points to act upon.

Has the DQA process improved baseline data completeness?

The baseline level of compliance does not represent the final compliance achieved in each year since the audit cycle was introduced. Our final compliance accuracy for the 2018/19 period is 97.9%. The baseline completeness in 2014 prior to any missed cases being entered demonstrated an average organisation-level compliance of 92.6%, but there remained over 7,000 cases where a record was present in the provider data without a corresponding record in the NJR. There was a higher percentage of revision cases that were missed, with 9.7% of revision hips missing versus 5.6% of primaries and 9.8% revision knees versus 5.1% of primaries. From 2015, the audit was conducted at individual unit level and included the independent sector. This demonstrated an average baseline unit level compliance rate of 91.8% in 2015/16, increasing to 92.6% in 2016/17 and to 94.1% in 2017/18. Baseline performance of data completeness shows a year-on-year improvement across all units. For the 2018/19 audit year, the first year of automation, 61.5% of units achieved 95% or higher compliance upon first run of the audit (Figure 2). This increased to 76.4% of units for the 2019/20 period. As of October 2022, 91.5% of units have now achieved >95% compliance for 2018/19 and 88.5% of units for 2019/20. These audits remain open and units continue to work to improve these figures towards 100%.

Figure 2: Effect of the data quality audit on completeness for 2018/19 and 2019/20.

Can we incorporate the DQA into ‘business-as-usual’ workflows for hospitals?

The NJR oversaw a national roll out of a semi-automated audit process in 2020/21. An automated secure data quality platform was developed to allow the upload of PAS data direct to the NJR data entry system (Figure 3). This enables users to upload PAS data at their convenience and to produce real-time reports of compliance against NJR submissions. Following processing, an updated compliance percentage is displayed, providing motivation for users to correct all missing or incorrect entries. Automation greatly reduces the number of potential missing cases that have to be checked each time the audit is run and also allows units to qualify for and receive their NJR Quality Data Provider certificates closer to real time. A new reporting suite also supports the programme by providing comprehensive information on the status of each unit in the audit cycle and exception reports flag any areas of concern for the NJR compliance team. This system also allows units to take responsibility locally to ensure target compliance is achieved and maintained.

Figure 3: Process flow diagram of secure file transfer protocol (SFTP) for automated data quality audit.

What still needs fixing?

The NJR’s Annual Report now reports on a ‘whole construct’ basis, meaning that an incomplete set of components entered for a case would be classed as ‘unconfirmed’, and excluded from some analyses. Work to examine these unconfirmed components has commenced across key areas, including elbow surgery, reverse shoulder replacement, dual mobility hip replacement and multiple-unicompartmental knee replacement.

This ongoing work is led by the NJR DQC and the relevant specialist societies, and involves both validation of component classification with industry and examination of individual cases by units. For elbow surgery, a nationwide DQA led by the British Elbow and Shoulder Society (BESS) with support from the BOA and RCS, has recruited BOTA surgical trainees to examine the operative notes of cases that are either absent from the registry (but present in administrative data) or with unconfirmed constructs. As a result of these efforts by the BESS team and the trainees, completeness of the elbow dataset has improved from 61% to 84% and accuracy from 93% to 98% (2022 NJR Annual Report https://reports.njrcentre.org.uk/downloads).

Take-home messages

Implementation of the DQA programme has had a substantial effect on ensuring the high quality of data in the NJR. Completeness is now routinely above 95% at baseline, increasing to 97% nationally post-audit, with many units capturing 100% of cases within each cycle. For NHS organisations, this has also resulted in gains in best practice tariff through increased recorded procedure volumes.

Establishment of the data quality audit as ‘business-as-usual’ has meant that the NJR has been able to increasingly focus on a more targeted examination of areas of data quality concern relating to implants.

Other registries looking to implement similar data audit processes would be encouraged to consider the following points:

• A data quality committee should be established to set the strategic aims and oversee and monitor the process.

• Clear and regular communication should be targeted to named data quality leads at each unit involved with the registry.

• Registry staff should be appropriately resourced to support units with engaging with the process. Particularly in the early years of implementation.

• Where possible, technological solutions should be implemented to reduce the burden on units and help establish these processes as part of routine work stream registry participation.

Explanatory note

For full details of the DQA audit development and implementation, please see our companion piece in BJO [2].

References

1. Burgess R. New Principles of Best Practice in Clinical Audit. 1st ed: CRC Press; 2011 20 Jan 2011.

2. Boulton C, Harrison C, Wilton T, Armstrong R, Young E, Pegg D, et al. Implementing large-scale data quality validation in a national arthroplasty registry to improve compliance: the National Joint Registry data quality audit programme. Bone Jt Open. 2022;3(9):716-25.