Clinical Research Insider Summit No. 11

Device validation and diagnostic tests

Validation is a process that determines the suitability of a test that has been properly developed, optimized and standardized for a specific purpose. One of the main reasons why the validation of a diagnostic device or test is sought is to assert that such an instrument will allow us to discriminate between patients suffering from a disease, syndrome or pathological process in particular from those who do not (healthy subjects). Then, the result of the device or test, together with the signs and symptoms for which the patient is entering, are the data that will help the doctor establish a diagnosis as accurate as possible to indicate the therapy that he believes is appropriate.

The validation process comprises a series of steps where more than one particular question will have to be answered, which will help establish the performance characteristics and technical specifications with which it will operate. As a first step, it is necessary to define the design and conditions under which the device or diagnostic test works, that is, what analyte does it identify?, what type of biological sample does it require (blood, saliva, urine, etc.)?, does the sample require a pretreatment, or how it will have to be prepared / diluted?, how many samples can I read simultaneously?, does it require a calibration curve or not?, What is the Gold Standard to use as a reference?

Subsequently, it will be necessary to define the target population, that is, the condition, syndrome or pathological process that the device or test can diagnose, in addition to the number of necessary individuals that will have to be tested through a sample calculation, establishing target values of sensitivity and specificity, with respect to the values operated by the gold standard, that is, If the Gold Standard has an 80% sensitivity, it would be desirable for the device or diagnostic test to have a sensitivity ≥ 80%, in order to operate as well or better than the reference test.

Some of the important aspects to consider for the study population that will have to be included are: 1) include patients with and without the disease; 2) those sick patients who are admitted must be patients to whom in common practice the test would be applied, that is, if they are considered for the diagnosis of tuberculosis, to be applied in subjects of whom tuberculosis is suspected and not of some other disease; and 3) that patients have a similar level of disease, to avoid spectrum bias. This will allow us to answer the following question: does the test discriminate between patients and healthy people where it is reasonable for the disease to exist? and Are patients with certain values in the test result more likely to make a diagnosis than others with other values?

Once the necessary patients have been included according to the sample size and established design, which can suggestively be: cross-sectional (from a single group made up of subjects to whom the test would be applied in usual practice), case-controls (defined by the gold standard) or cohorts (defined by the device or diagnostic test), we proceed to the evaluation of all participants in the test, which should be blinded (without knowing what the result of the GS was or if it is sick/healthy) to avoid over/underestimation bias.

Finally, we proceed to the statistical analysis of the results, for this conventionally a ROC curve (ReceiverOperating-Characteristic) is constructed, a statistical tool that is used to evaluate the discriminative capacity of a dichotomous diagnostic test (presence/ absence) and a cut-off point (threshold) is established from the observed results, which will help us define the final result of the test. Finally, the calculation of the performance characteristics is made: sensitivity, specificity, predictive values, probability rates, etc., by means of contingency tables (2x2 tables).

Although the statistical methods mentioned are not the only ones by which the validation of a device or diagnostic test can be reached, they are the most used for dichotomous and scale tests.

As an additional step, it is advisable to determine reproducibility, for this it will be necessary to follow a prospective study with cohort design in undiagnosed subjects, in order to evaluate the predictive diagnostic capacity of the device or test. This prospective nature and the fact of applying the test before making the diagnosis, allows the control of the main biases, being able to ensure both the independent and blind evaluation of the test to be validated and the diagnostic test with which it is compared.

As additional points of reproducibility, it is advisable to do tests with different operators and in different laboratories, to make sure of the measure of precision with which the device or test operates, regardless of who handles it or where it is used, as long as the methodology of use is followed.

Wendy Lopez Romero PhD

Research and Development Analyst at Drox Health Science. PhD in Innovation in Medical and Pharmaceutical Biotechnology. Master in Molecular Biology.

References

Hajian-Tilaki, K. (2013). Receiver operating characteristic (ROC) curve analysis for medical diagnostic test evaluation. Caspian journal of internal medicine, 4(2), 627.

Donis, J. H. (2012).Evaluation of the validity and reliability of a diagnostic test. Avances en biomedicina, 1(2), 73-81.

Medina, M. C. (2011). Generalities of diagnostic tests, and their usefulness in medical decision making. Revista Colombiana de Psiquiatría, 40(4), 787-797.

Márquez-Fernández, A. J., Ramos-Ligonio, A., & López-Monteon, A. (2015). What do you know about the validation of diagnostic tests?. Revista Mexicana de Ciencias Farmacéuticas, 46(3), 86-90.

Cerda, J., & Cifuentes, L. (2012). Use of ROC curves in clinical research: Theoretical and practical aspects. Revista chilena de infectología, 29(2), 138-141.