Task 1: Understanding Diagnostic Accuracy of Binary

Canadian Association of Pathologists

Statistics for Pathologists Primer: Focus on Diagnostic Accuracy Statistics Workshop Task1

Nickolas Myles, MD, MSc(Oxon), PhD, FRCPC Anatomical Pathologist, St.Paul’s Hospital Clinical Associate Professor, Department of Pathology and Laboratory Medicine University of British Columbia

QC July 7th 2018

1

Task 1: Understanding Diagnostic Accuracy of Binary Test Step 1: please complete the Task 1 intuitively or using only pencil and paper calculations. https://www.surveymonkey.com/r/GS3ZPRN or, if you have no access, select your answer:

2

Task 1 Workings Hint: construct 2X2 table (“Target disorder”= Reference ER test/lab result; “Test result” = your lab result) Task 1 Step 1 Fill up your lab test optimization (or QC) phase results: Reference + Your + lab 50 50 total 100 Results: Prevalence, % _____ Sn,%______ Sp ,% _____ PPV,% _____NPV, %_____ Error of tested negative (False negatives), %__ (Hint 100 - (% true negatives of all tested negative, i.e. NPV).

3

Task 1 Solution Task 1 Step 1 Fill up your lab test optimization (or QC) phase results: Reference + Your + lab -

45 5

5 45 50

50

total 100

Results: Prevalence, %

50% (50/100=50, or 50%)

Sn,% 45/50 or 90%__ Sp ,% 45/50 or 90% PPV,% 45/50 or 90% _NPV, % 45/50 or 90% Error of tested negative (False negatives), % 10 (Hint 100 - (% true negatives of all tested negative, i.e. NPV). 100-90=10

4

Task 1 Workings Task 1 Step 2 Fill up your lab EXPECTED re-test results: Hint: construct 2X2 table (“Target disorder”= Reference ER test/lab result; “Test result” = your lab result) using real life prevalence of ER positive breast cancers and total 1000 cases (for simplicity of calculations).

Reference + Your + lab -

Total 1000 Remember, that you already know the following information: Prevalence: 80% ER positives Your sensitivity: 90% Your specificity: 90%

5

Task 1 Solution Task 1 Step 2 Solution How you can fill up 2x2 table using this information? Prevalence: 80% ER positives Your sensitivity: 90% Your specificity: 90% -

For a sake of easy calculations, let’s assume your total number of ER tests over the years is 1000 (imaging your lab is BIG if not HUGE)

-

Prevalence of 80% is indeed the total ER+ by reference lab, i.e. 80% of 1000= (1000/100)*80 = 800 cases (expected ER positive)

-

Derive the EXPECTED number of “ER –“ by reference lab, i.e. 1000-800 = 200 cases

Reference -

+ Your lab

+ -

800 Prev:

80%

200

Total 1000

100-80%=20%

Next, using the KNOWN data on your lab, i.e. sensitivity (i.e. 90%) and specificity (90%) calculate the expected numbers of true positives and true negatives. Reference + Your lab

+ -

? ? 800

200

Total 1000

6

+ Your lab

+ -

Reference -

720 180 800

200

Sens= 90% out of 800 = 800/100 *90 = 720 Spec= 90% out of 200 = 200/100 *90 = 180 The rest of the table fills easily by simple subtraction:

+ Your lab

+ -

720

Reference 20

80

180 800

200

You are now ready to get the answer to the question, i.e. what’s the expected error of your negative ER cases retested by the impeccable reference lab (= % reference negatives of all ER-tested negative by your lab, i.e. NPV) Reference + Your lab

+ -

720

20

80

180 800

200

NPV = % of true negatives of all tested negative = 180/ (80+180)= 0.69 (69%) The EXPECTED error of ER negative test is thus 100% -69% = staggering 31% !!!

Task 1 Workings

7

Task 1 Step 3. Hint: use 2X2 table, and the same approach as in the previous part the exercise, but revise the sensitivity, as your new antibody is now 95% sensitive. Use the same data of 90% specificity and 80% prevalence.

Reference + Your + lab Total 1000 800

200

Prevalence: 80% ER positives Your sensitivity: 95% Your specificity: 90% *Hint Populate the right column with the data from the previous exercise, since the specificity did not change.

8

Task 1 Step 3 Solution Task 1 Step 3 Solution

+ Your lab

Reference -

+ -

20 180 8000

2000

Now, calculate the EXPECTED values of the left column cells. “++” cell, i.e TP = 95% of 8000= (8000/100)*95= 7600, Derive the rest: 8000-7600=400 Reference + Your lab

+ -

760

20

40

180 800

200

You are now ready to get the answer to the question: What’s the expected negative test error rate with your new antibody? NPV= 180/(180+40)= 0.81 (81%), therefore the EXPECTED error is 100-81=19%. Your test improved by 12% (i.e. 31-19 = 12%.) Conclusion: Be careful with your intuitive assumptions! Predictive values of the test are not fixed test parameters and CHANGE with prevalence. If the goal is to monitor the error, sensitivity and specificity are the parameters you use in QC, not predictive values. The judgement of presumably “poor” laboratory performance should not be based on PPV and NPV, since it’s a function of prevalence, and not only due to a laboratory performance. The mathematical reason for this paradox is based on Bayesian theorem. The relationships between the test sensitivity, specificity and positive and negative predictive values are non-linear (they are, indeed, logarithmic).

9

Task 1 Step 4 Now complete the same task 1 (question 1 and question 2) using the online calculator (provided by the University of Oxford, Center for Evidence-based medicine) http://www.cebm.net/catmaker-ebm-calculators/

Save your screen shots or write down the results A. Use the test optimization data: Reference + Your + lab -

45 5

5 45 50

50

total 100

SN______Sp______ LR+_____LR-_____ PPV_____NPV_____ Error of tested negative, %__ B. Use the data from ER-negative re-test exercise

+ Your + lab -

Reference -

720

20

80 800

180 200

SN______Sp______ LR+_____LR-_____ PPV_____NPV_____

10

Error of tested negative, %__ (% true negatives of all tested, i.e NPV) C. Use the data from the new antibody exercise Reference + Your + lab -

760 40 800

20 180

200

SN______Sp______ LR+_____LR-_____ PPV_____NPV_____ Error of tested negative, %__ (% true negatives of all tested, i.e NPV) Compare you results with the other group members. What’s happening with SN, SP, LR+, LR-, PPN and NPV ? See the solution screens on the next page.

11

Task 1 Step 4 Solutions A.

B.

C.

12

Task 1 Step 5 (optional): now, instead of 2X2 table, use the Nomogram and likelihood ratios (calculated in the last exercise C.). http://www.cebm.net/catmaker-ebm-calculators/ Select Nomogram option (Next to your working screen, right bottom button NOMOGRAM). A. Set prevalence at 50% (similar to recommended test optimization phase) at the left hand scale of the nomogram (prevalence=pre-test probability) - set LR+ (value 9.5) on the middle scale of the nomogram) - get post-test probability (% of TP of all tested), write down your result (right hand scale arrow), this is positive predictive value. - repeat the same steps for LR- and write down negative predictive value of the test (i.e.% of true negatives of all tested). - The % error of the negative test will be =100 – NPV :______________. B. - Now set prevalence at 80% (similar to ER-positivity in breast cancer) at the left hand scale of the nomogram, and repeat the same steps: - set LR+ on the middle scale of the nomogram) - get post-test probability (% of TP of all tested), write down your result (right hand scale point) - repeat the same steps for LR- and write down negative predictive value of the test (i.e.% of true negatives of all tested). - The error of the negative test will be 100 - NPV. c. – Now, with the same prevalence of 80%, get the LR- (for sensitivity of 95% after your improvements and same specificity of 90%). Explore how NPV is changing. See the sample solutions on the next page. These are the two ways of getting the same result (by either filling up 2X2 table with the revised prevalence or by using the nomogram (both are the alternative solutions of Bayesian theorem). The online advantage of online calculators over manual 2x2 table is that all parameters come along with their degree of precision (95%CI).

13

4a.

4b.

4c.

14

Task 1 General Comments: Steps 1 and 2 Sensitivity, Specificity, LR+ and LR- are the same, but post-test probabilities (predictive values) change due to a different prevalence (pre-test-probability) changed from 50% to 80%). This is not due to a failure of the technical or interpretative aspects of the test. Step 3 Now, you improved your sensitivity, which is now an enviable 95%: - at the same natural prevalence of 80% (i.e. natural occurrence of ER+ breast cancer), out of total 1000 tests 800 are biologically ER positive, and 200 are biologically ER-negative. - Sensitivity of 95% out of 800 i.e. 800/100*95=760 cases (true positive cell); - specificity of 90% out of 200, i.e. (200/100)*90=180 cases (true negative cell). - The other two cells can be derived by simple subtraction: 800-720=40 (false negative cell) 200-180=20 (false positive cell) - Negative predictive value is now 82% (95%CI 79-84%) - False negative error: 100-82%= 18% (95%CI 16-1%) These are the two ways of solving the Task 1 (they give the same result) 1. by either filling up 2X2 table with the revised prevalence or 2. by using the nomogram and likelihood ratios of the test. Advanced users may find Bayesian theorem mathematical formula (not covered by this tutorial) gives similar result (in fact all calculation methods are based on the same Bayesian theorem, which has multiple solutions) I prefer the online calculators based on 2x2 table as they generate all parameters I need (SN, SP, LR+, LR-) along with their degree of precision (i.e.95%CI). Some people prefer nomograms as they are more interactive and easy to use. Conclusion Task 1 : We explored the phenomenon called “prevalence effect” in diagnostic testing, which is not due to a failure of the performance of the test. This effect is well known to clinical epidemiologists and public health physicians. The pathologists should be aware of it and cannot use it as a stable indicator of poor test performance. Rather, it predicts how the test with fixed Sn and Sp works in different prevalence settings. References: Thompson M, Van den Bruel A. Diagnostic testing Toolkit, BMJ Library 2012. Diagnostic Testing Workbook, 2016 (Cochrane Collaboration), under development http://methods.cochrane.org/sdt/handbook-dta-reviews Makretsov N. Why We Need Evidence-Based Breast Biomarker testing. Editorial, Ann Clin Oncol 2014, 2(3);1027.

15