Diagnosis Vs Screening

A diagnostic test is done on sick people

Patient presents with symptoms

Pre-test probability of disease is high (i.e. disease prevalence is high)

A screening test is usually done on asymptomatic, apparently healthy people

Healthy people are encouraged to get screened

Pre-test probability of disease is low (i.e. disease prevalence is low)

Evaluating a diagnostic test

Define gold standard

Recruit consecutive patients in whom the test is indicated (in whom the disease is suspected)

Perform gold standard and separate diseased and disease free groups

Perform test on all and classify them as test positives or negatives

Set up 2 x 2 table and compute: Sensitivity Specificity Predictive values Likelihood ratios



Evaluating a diagnostic test

Diagnostic 2 X 2 table*:

*When test results are not dichotomous, and then can use ROC curves [see later]

Disease + Disease - Test + True Positive False Positive Test - False Nagative True Nagative

Sensitivity: The proportion of patients with disease who test positive, this is known as true positive rate also.

This is the formula we can calculate the true positive rate of sensitivity bases on following:

Sensitivity = TP / (TP+FN)

, where TP = True positive, FN = False Negative

Specificity: The proportion of patients without disease who test negative, this known as true negative rate also.

The formula of the specificity:

= TN / (TN + FP)

,where TP = True positive, FN = False Negative

Predictive value of a positive test: Proportion of patients with positive tests who have disease

PPT = TP / (TP+FP)

Predictive value of a negative test: Proportion of patients with negative tests who do not have disease

= TN / (TN+FN)

Diagnostic Odds Ratio (DOR): Odds of positive test result in persons with the target condition compared to those without the target condition

DOR = (a/c) / (b/d)

DOR = ad / bc

Example of all measurement calculation through formula using the serological test for TB

Disease Yes Disease No Total Serological Test Positive(+) 14 3 17 Negative ( - ) 54 28 82 Total 68 31 99

Sensitivity = TP / (TP+FN) = 14/ (14+54) = 21%

Specificity = TN / (TN + FP) = 28 / (3+28) = 90%

PPT = TP / (TP+FP) = 14/ (14+3) = 82%

PNT = TN / (TN+FN) = 28/ (28+54) = 34%

DOR = ad/bc = 14*28/3*54 = 392/162 = 2.41

ROC Curve

A receiver operating characteristic curve, i.e. ROC curve, is a graphical plot that illustrates the diagnostic ability of a binary classifier system as its discrimination threshold is varied.

The diagnostic performance of a test or the accuracy of a test to discriminate diseased cases from normal cases is evaluated using Receiver Operating Characteristic (ROC) curve analysis

A Receiver Operating Characteristic (ROC) Curve is a way to compare diagnostic tests. It is a plot of the true positive rate against the false positive rate.

Interpretation of ROC curves

a. A ROC curve of a random classifier

A classifier with the random performance level always shows a straight line from the origin (0.0, 0.0) to the top right corner (1.0, 1.0).

Two areas separated by this ROC curve indicate a simple estimation of the performance level. ROC curves in the area with the top left corner (0.0, 1.0) indicate good performance levels, whereas ROC curves in the other area with the bottom right corner (1.0, 0.0) indicate poor performance levels

A ROC curve represents a classifier with the random performance level. The curve separates the space into two areas for good and poor performance levels.

ROC curve of a perfect classifier

A classifier with the perfect performance level shows a combination of two straight lines – from the origin (0.0, 0.0) to the top left corner (0.0, 1.0) and further to the top right corner (1.0, 1.0).

It is important to notice that classifiers with meaningful performance levels usually lie in the area between the random ROC curve (baseline) and the perfect ROC curve.

ROC curves for multiple models

Comparison of multiple classifiers is usually straight-forward especially when no curves cross each other. Curves close to the perfect ROC curve have a better performance level than the ones closes to the baseline.

Two ROC curves represent the performance levels of two classifiers A and B. Classifier A clearly outperforms classifier B in this example.

AUC (Area under the ROC curve) score

Another advantage of using the ROC plot is a single measure called the AUC (area under the ROC curve) score. As the name indicates, it is an area under the curve calculated in the ROC space.

One of the easy ways to calculate the AUC score is using the trapezoidal rule, which is adding up all trapezoids under the curve.

The AUC score can be calculated by the trapezoidal rule, which is adding up all trapezoids under the curve. The areas of the three trapezoids 1, 2, 3 are 0.0625, 0.15625, and 0.4375. The AUC score is then 0.65625.

Although the theoretical range of AUC score is between 0 and 1, the actual scores of meaningful classifiers are greater than 0.5, which is the AUC score of a random classifier.

It shows four AUC scores. The score is 1.0 for the classifier with the perfect performance level (P) and 0.5 for the classifier with the random performance level (R). ROC curves clearly shows classifier A outperforms classifier B, which is also supported by their AUC scores (0.88 and 0.72).

Tests with continuous results:

ROC curve analysis:

Blood sugar level (2-hour after

food) in mg/100 ml Sensitivity (%) Specificity (100%) 70

80

90

100

110

120

130

140

150

160

170

180

190

200 98.6

97.1

94.3

88.6

85.7

71.4

64.3

57.1

50.0

47.1

42.9

38.6

34.3

27.1 8.8

25.5

47.6

69.8

84.1

92.5

96.9

99.4

99.6

99.8

100

100

100

100

Note - For each cut-off, we can generate a 2 x 2 table with estimates of sensitivity and specificity

Area under the curve (AUC) can range from 0.5 (random chance, or no predictive ability; refers to the 45 degree line in the ROC plot) to 1 (perfect discrimination/accuracy.

The closer the curve follows the left-hand border and then the top-border of the ROC space, the more accurate the test. The closer the curve comes to the 45-degree diagonal of the ROC space, the less accurate the test.

Studies internet links