External Validation of clinical prediction models - Statswork

EXTERNAL VALIDATION OF CLINICAL PREDICTION MODELS

An Academic presentation by Dr. Nancy Agnes, Head, Technical Operations, Statswork Group

www.statswork.com

Email: info@statswork.com

TODAY'S DISCUSSION Outline External Validation Factors influences affect external validation data Future Scope

Validation, particularly external validation, is a crucial part of developing a predictive model.

External validation is needed to ensure that a prediction model is generalizable to patients other than those in the derivative cohort.

External validation can be done by testing the model's output in data that isn't the same as the data used to create the model.

As a consequence, it is carried out after the creation of a prediction model.

EXTERNAL VALID ATION External validation can take many forms, including validation in the field such as temporal, geographical and independent validation.

For external validation studies, the sample size calculation estimates based on statistical power considerations have not been extensively investigated.

However, in order to achieve adequate model output in the validation set, a large sample size is needed to validate the prediction model.

FACTORS INFLUENCES AFFECT EXTERNAL VALID ATION DATA

The sample size for external validation data for the implementation of the prediction model is affected by the number of events and predictors.

External validation of the prediction model requires a minimum of 100 events and/or non-events, according to simulation studies, and a systematic analysis found that small external validation studies are ineffective and inaccurate.

Example: Radiology imaging is often treated as effective predictive parameters and researchers often validate the findings using clinical prediction model.

Contd...

Every prediction model is based on the regression analysis.

The most common predictive model or the regression model used for the clinical prediction model are linear regression if the dependent variable is continuous in nature, logistic regression model if the dependent variable is binary, and Cox-proportional model if the dependent variable is time-to-event in nature.

Al-Ameri et al (2020) presented a detailed review on clinical prediction models for liver transplantation study.

Further, Ratna et al (2020) discussed the quality of clinical prediction model in vitro fertilisation and human reproduction.

Validation of model has been carried out using re-sampling technique and measured the accuracy using AUC, calibration plot as shown in figure 1, c-index, and HosmerLemeshow test statistic.

Contd...

Contd...

Figure 1: Slope of Calibration plot (Source: Stevens and Poppe (2020)) In addition, Stevens and Poppe (2020) suggested the Cox- calibration slope using logistic regression model instead of using simply the calibration slope for the predictive model.

This suggestion has been made after the scrutiny of around 33 research articles and found that most of the validation are external validation and identified the validity using the calibration slope and the sample articles are presented in the following table.

Contd...

Contd...

Contd...

Table1.Stated Interpretation of the “Calibration Slope” Source: Stevens and Poppe (2020) Arjun et al (2020) considered the pandemic mortality study of COVID19 and discussed the development and validation of clinical prediction Contd... model.

FUTUR E SCOPE

Though many literature suggests several validation techniques for the predictive model, there is no such proper technique which can be suitable for all the clinical datasets.

Further, proper adjustment has to be made for the calibration index to validate the prediction model suitable for all clinical datasets.

Contd...

Contact Us UNITED KINGDOM +44-1143520021

INDIA +91-4448137070

EMAIL info@statswork.com