External Validation of clinical prediction models
Dr. Nancy Agnes, Head, Technical Operations, Statswork info@statswork.com
Validation, particularly external validation,
II. FACTORS INFLUENCES AFFECT
is a crucial part of developing a predictive
EXTERNAL VALIDATION DATA
model. External validation is needed to ensure
that
a
prediction
model
is
generalizable to patients other than those in
the
derivative
cohort.
External
The sample size for external validation data for the implementation of the prediction model is affected by the number of events and predictors.
validation can be done by testing the model's output in data that isn't the same as
External validation of the prediction model
the data used to create the model. As a
requires a minimum of 100 events and/or
consequence, it is carried out after the
non-events,
creation of a prediction model.
studies, and a systematic analysis found
according
to
simulation
that small external validation studies are I. EXTERNAL VALIDATION
ineffective
and
inaccurate.Example:
External validation can take many forms,
Radiology imaging is often treated as
including validation in the field such as
effective
temporal, geographical and independent
researchers often validate the findings
validation. For external validation studies,
using clinical prediction model. Every
the sample size calculation estimates
prediction
based on statistical power considerations
regression analysis. The most common
have not been extensively investigated.
predictive model or the regression model
However, in order to achieve adequate
used for the clinical prediction model are
model output in the validation set, a large
linear regression if the dependent variable
sample size is needed to validate the
is continuous in nature, logistic regression
prediction model.
model if the dependent variable is binary, and
predictive
model
is
Cox-proportional
parameters
based
model
on
if
and
the
the
dependent variable is time-to-event in
Copyright © 2021 Statswork. All rights
1
nature. Al-Ameri et al (2020) presented a
identified the validity using the calibration
detailed review on clinical prediction
slope
models for liver transplantation study.
presented
and
the in
sample the
articles
following
are table
Further, Ratna et al (2020) discussed the quality of clinical prediction model in vitro fertilisation
and
human
reproduction.
Validation of model has been carried out using re-sampling technique and measured the accuracy using AUC, calibration plot as shown in figure 1, c-index, and HosmerLemeshow test statistic.
Figure 1: Slope of Calibration plot (Source: Stevens and Poppe (2020)) In addition, Stevens and Poppe (2020) suggested the Cox- calibration slope using logistic regression model instead of using simply the calibration slope for the predictive model. This suggestion has been made after the scrutiny of around 33
Table1.Stated Interpretation of the “Calibration Slope” Source: Stevens and Poppe (2020) Arjun et al (2020) considered the pandemic mortality study of COVID19 and discussed the development and validation of clinical prediction model.
research articles and found that most of the validation are external validation and
Copyright © 2021 Statswork. All rights
2
II. FUTURE SCOPE Though many literature suggests several validation techniques for the predictive model, there is no such proper technique which can be suitable for all the clinical datasets. Further, proper adjustment has to be made for the calibration index to validate the prediction model suitable for all clinical datasets. References: 1.
2.
3.
4.
5.
Stevens, R. J. and Poppe, K. K. (2020). Validation of Clinical Prediction Models: What does the "Calibration Slope" Really Measure?. Journal of clinical epidemiology, 118, pp. 93– 99. Adibi, A., Sadatsafavi, M., Ioannidis, J. P. A. (2020). Validation and Utility Testing of Clinical Prediction Models: Time to Change the Approach. JAMA. 2020; 324(3):235–236. Ratna, M. B., Bhattacharya, S., Abdulrahim, B. and McLernon, D. L. (2020). A Systematic Review of the Quality of Clinical Prediction Models in Vitro Fertilisation, Human Reproduction, 35(1), pp. 100–116 Arjun S Yadaw., Yan-chak Li., Sonali Bose., Ravi Iyengar., Supinda Bunyavanich., Gaurav Pandey. (2020). Clinical Features of COVID19 Mortality: Development and Validation of a Clinical Prediction Model, The Lancet Digital Health, 2(10), pp. 516-525. Al‐Ameri, A.A.M., Wei, X., Wen, X., Wei, Q., Guo, H., Zheng, S. and Xu, X. (2020), Systematic review: risk prediction models for recurrence of hepatocellular carcinoma after liver transplantation. Transpl Int, 33, pp. 697712.
Copyright © 2021 Statswork. All rights
3