The Reverend Thomas Bayes was an 18

Consider the case of medical diagnostic testing. We start with a patient who we are concerned might have a particular disease. Our estimate of how likely it is that they have the disease is the “prior probability”. Once we carry out the diagnostic test, we can use Bayes theorem to work out how to use the results of the testing to “update” the probability that the patient has the disease. This revised estimate is the “posterior” or “post-test probability”.

The prior probability can be hard data e.g. it has been demonstrated by extensive study that in this population the prevalence of tuberculosis is 35%. Alternatively, it can just be an educated estimate based on the clinician’s experience e.g. I have now seen a lot of patients like this in this town with fever, fatigue and dry cough and most have had COVID-19. If they have a known exposure to an infected patient, I would estimate the probability at 95%. If there is no known prior exposure, I would say the probability is 40%.

We then do the test e.g. for tuberculosis or for COVID-19. To accurately interpret a test, it is essential that we have some knowledge of the test performance – how accurate it is. In this context, pathology test performance is measured in terms of

In an ideal world all our diagnostic tests would have 100% sensitivity and specificity. Unfortunately, we don’t live in an ideal world and real diagnostic tests have less than ideal performance and as we will see, this has consequences.

So far almost all the testing that has been done around the world is aimed at detecting viral RNA in swabs taken from the nose and throat of patients. Most of this is RT-PCR nucleic acid testing but other related tests are being introduced. Due to the rapid pace of change in this pandemic we are in the very unusual situation in that none of these tests have received the exhaustive evaluation that new diagnostic tests usually undergo. They have received emergency authorization for use by bodies such as the FDA in the US and the TGA in Australia. Thus, we don’t know the exact sensitivity and specificity of these tests and how they compare to each other in performance. We are not completely in the dark however, as all of the tests have had preliminary evaluations performed and we have a pretty good idea of their sensitivity and specificity and we can use these numbers in our Bayesian estimates.

It has become apparent that for several reasons, but mainly because of difficulties getting good samples of respiratory secretions, that the sensitivity of COVID-19 nucleic acid tests may be as low as 70%. That is, 30% of people who do have the infection may return a negative nucleic acid test result. Specificity of COVID-19 nucleic acid tests is much higher; some tests are estimated to have a specificity of 98%.

Now we can look at the consequences of these test performance figures in two different situations.

Situation 1.

A person returns from New York by plane and two days later develops symptoms consistent with COVID-19. The doctor estimates the probability of COVID-19 in this person as being 90%

However, to better understand how these calculations are done we can walk through the process. The best way to do this is to assume that we have 1000 patients just like this. We can then draw up a table to demonstrate the outcomes for all of these patients (see below). If we assume a pre-test probability of 90% then of our 1000 patients, 900 will be infected and 100 not. These numbers go down as the bottom line of the table. We then use the test sensitivity and specificity to fill in the rest of the table. Given a sensitivity of 70%, of the 900 infected patients 630 should test positive (0.7 x 900 = 630). These are the

90% PRE-TEST PROB | INFECTED | NOT INFECTED | TOTALS |

TEST POSITIVE | 630 TRUE POS | 2 FALSE POS | 632 TOTAL POS |

TEST NEGATIVE | 270 FALSE NEG | 98 TRUE NEG | 368 TOTAL NEG |

TOTALS | 900 | 100 | 1000 |

We can use these figures to see how we should update our belief in whether the patient has coronavirus or not based on the results of the test. Firstly, let’s consider the case that the test came back positive. How likely is it,

What about if the test came back negative? How likely is it,

Situation 2.

Another person shared a maxi-taxi with the person above for a short ride the next day. They sat in the front with the window open, the infected person right at the back. They have not developed any symptoms yet. Contact tracers ask them to be tested. The contact tracers estimate the probability of this person being infected with COVD-19 at 5%.

5% PRE-TEST PROB | INFECTED | NOT INFECTED | TOTALS |

TEST POSITIVE | 35 TRUE POS | 19 FALSE POS | 54 TOTAL POS |

TEST NEGATIVE | 15 FALSE NEG | 931 TRUE NEG | 947 TOTAL NEG |

TOTALS | 50 | 950 | 1000 |

In this case the PPV = 35/54 = 65%

And NPV = 931/947 = 98.4%

The most likely outcome is a negative result and this will be reassuring, we can be quite confident the person is not infected. A positive result however, is not definitive, a positive result means that they have a 2/3 chance of being truly infected. We should be very suspicious and keep this person isolated for the next couple of weeks and consider repeating the test after 2 or 3 days.

COVID-19 serology testing is not yet widely available but it will be soon. Depending on how they are used, we will have different requirements of serology tests. Nucleic acid tests have a problem with false-negative test results. Serology tests have the opposite problem, they are subject to false-positive test results. This is because our blood teems with thousands of different antibodies to myriad different antigens that we have been exposed to in our lives. Some of those antigens are closely-related coronaviruses with quite similar structure to SARS-CoV-2. We need to design serological assays that have capture antigens that are unique to SARS-CoV-2 and will not react with antibodies against other coronaviruses and other fortuitously similar antigens. This is not a simple task and assays need extensive evaluation against ideally many hundreds or thousands of SARS-CoV-2 negative samples to prove they have high specificity.

If we are testing someone who has been ill, had a positive nucleic acid test and then recovered, the pre-test probability is 100%. The only reason we would do the test is to assess how strong their immune response was. The only caveat is that we have to be patient. Standard procedure is to take two blood samples for testing, one as early as possible in the first week of the illness and then another several weeks later and to look for a significant rise in the amount of antibodies present (the “titre”). False-positive test results are not a problem in this situation as we don’t expect to see any.

The second very important role for COVID-19 serology tests is to assess exposure in the general population. Most, but not all, authorities expect that people who have seroconverted to SARS-CoV-2 will have some degree of immunity against reinfection, at least in the medium term. Knowing the degree of immunity in the population will be vital information for modellers and help them plan a way out of our current social restrictions. However, evidence to date suggest that although asymptomatic infection with SARS-CoV-2 is not uncommon, it is likely that the prevalence of seroconversion against the virus in the general population will be quite low, especially in Australia where we have succeeded in bringing the infection rate down quite quickly compared with many other countries. This fact compounds the problem of false-positive test results with COVID-19 serology.

Let’s consider a scenario with three different tests.

Scenario 1.

Consider an Australian city or a group of mine workers where 5% of people have had COVID-19 but this is unknown to us. This is the “prior” or “pre-test probability” we use in our calculation. The serological testing is performed with a simple fingerprick blood sample point-of-care (POC) test that can be done at a doctor’s office or a mining site. Its sensitivity and specificity are both 95%. Some POC tests may have better test performance than this, some will have worse. We don’t yet have full independent evaluation data on the POC tests that have received emergency TGA approval in Australia.

5% PRE-TEST PROB | SEROCONVERTED | NOT SEROCONVERTED | TOTALS |

TEST POSITIVE | 48 TRUE POS | 47 FALSE POS | 95 TOTAL POS |

TEST NEGATIVE | 2 FALSE NEG | 903 TRUE NEG | 905 TOTAL NEG |

TOTALS | 50 | 950 | 1000 |

The first thing to notice is that almost half of our positive test results are false-positive. The

Scenario 2.

Here we have what seems to be a better serology test. The sensitivity of this test is 98% and the specificity is also 98%.

5% PRE-TEST PROB | SEROCONVERTED | NOT SEROCONVERTED | TOTALS |

TEST POSITIVE | 49 TRUE POS | 19 FALSE POS | 68 TOTAL POS |

TEST NEGATIVE | 1 FALSE NEG | 931 TRUE NEG | 932 TOTAL NEG |

TOTALS | 50 | 950 | 1000 |

For the modellers this testing might or might not provide useful information depending on how much their calculations are affected by error in the estimated seroprevalence parameter. The positive test rate of 68 positive results out of 1000 people tested (6.8%) is reasonably close to the true underlying prevalence (5%). This information can be used to estimate the likely transmission dynamics in the population if restrictions were loosened. However, for any individual who tested positive there is only a 49/68 = 72% chance that they have in fact had COVID-19. This also means that 19 of the 68 people who tested positive have not had the disease and are therefore susceptible to getting it if they believe themselves immune and expose themselves to infection again. When the prevalence of the condition is low (5% in this case), we need a more specific test to have confidence in the results.

Scenario 3.

Here we have an improved serology test. The sensitivity is the same at 98% but the specificity is better at 99.8%, i.e. we only expect one false positive test result for every 500 people who definitively don’t have the disease.

5% PRE-TEST PROB | SEROCONVERTED | NOT SEROCONVERTED | TOTALS |

TEST POSITIVE | 49 TRUE POS | 2 FALSE POS | 51 TOTAL POS |

TEST NEGATIVE | 1 FALSE NEG | 948 TRUE NEG | 949 TOTAL NEG |

TOTALS | 50 | 950 | 1000 |

This new test estimates the prevalence of seroconversion in the population as 51 positive tests out of 1000 tested or 5.1% which is almost exactly the true prevalence. Additionally, the problematic false positives have now been reduced to 2 out of 1000 tested but this still means that 2 out of 51 positive tests are false-positives. This increase in the specificity (increasing from 98% to 99.8%) has had a large impact on the usefulness of the test. Since we don’t know the prevalence of seroconversion in the population before we start, it is crucial that the test to be used has adequate sensitivity and very high specificity in particular when we expect the true seroprevalence to be low.

Whoever is using the test has to decide on the utility of the results. The information provided will be very useful for disease modellers. However, we have to decide if it is reasonable that 2 out of every 1000 people tested will be falsely reassured that they have had COVID-19 already.

The current COVID-19 crisis highlights how important it is to correctly interpret diagnostic tests and it is clear that Bayesian thinking is central to this. The Reverend Bayes could never have imagined that his theorem would one day be used to help guide government decisions in the setting of a global pandemic.

*Last Review Date: September 2, 2020*