print   Print full article
Interpreting coronavirus testing- some things we need to consider.

Readers please note – this is a bit complex and we are going to use some very simple mathematics in our explanation. If you are not comfortable with this then you can read a simple explanation of Bayesian thinking in relation to testing without any maths (How accurate are COVID-19 tests?).

The Reverend Thomas Bayes was an 18th-century Presbyterian minister and mathematician. He developed an important theorem late in his life but it was not published until 1763, two years after his death. It is now known as Bayes theorem. Bayesian probability theory specifies how we should update our belief in the likelihood of something given new evidence or data.  Bayesian probability is not intuitive and is not well understood by many people including some doctors.

Consider the case of medical diagnostic testing.  We start with a patient who we are concerned might have a particular disease.  Our estimate of how likely it is that they have the disease is the “prior probability”. Once we carry out the diagnostic test, we can use Bayes theorem to work out how to use the results of the testing to “update” the probability that the patient has the disease.  This revised estimate is the “posterior” or “post-test probability”.

The prior probability can be hard data e.g. it has been demonstrated by extensive study that in this population the prevalence of tuberculosis is 35%. Alternatively, it can just be an educated estimate based on the clinician’s experience e.g. I have now seen a lot of patients like this in this town with fever, fatigue and dry cough and most have had COVID-19. If they have a known exposure to an infected patient, I would estimate the probability at 95%. If there is no known prior exposure, I would say the probability is 40%.

We then do the test e.g. for tuberculosis or for COVID-19. To accurately interpret a test, it is essential that we have some knowledge of the test performance – how accurate it is. In this context, pathology test performance is measured in terms of sensitivity and specificity.

Sensitivity is the proportion of patients who do have the disease that return a positive test result. It is expressed as a number e.g. 0.95 or as a percentage e.g. 95%.

Specificity is the proportion of patients without the disease who return a negative test result. Again, it is expressed as a number or percentage.

In an ideal world all our diagnostic tests would have 100% sensitivity and specificity. Unfortunately, we don’t live in an ideal world and real diagnostic tests have less than ideal performance and as we will see, this has consequences.

COVID-19 Nucleic Acid Testing
So far almost all the testing that has been done around the world is aimed at detecting viral RNA in swabs taken from the nose and throat of patients. Most of this is RT-PCR nucleic acid testing but other related tests are being introduced. Due to the rapid pace of change in this pandemic we are in the very unusual situation in that none of these tests have received the exhaustive evaluation that new diagnostic tests usually undergo. They have received emergency authorization for use by bodies such as the FDA in the US and the TGA in Australia. Thus, we don’t know the exact sensitivity and specificity of these tests and how they compare to each other in performance. We are not completely in the dark however, as all of the tests have had preliminary evaluations performed and we have a pretty good idea of their sensitivity and specificity and we can use these numbers in our Bayesian estimates.

It has become apparent that for several reasons, but mainly because of difficulties getting good samples of respiratory secretions, that the sensitivity of COVID-19 nucleic acid tests may be as low as 70%. That is, 30% of people who do have the infection may return a negative nucleic acid test result. Specificity of COVID-19 nucleic acid tests is much higher; some tests are estimated to have a specificity of 98%.

Now we can look at the consequences of these test performance figures in two different situations.

Situation 1.

A person returns from New York by plane and two days later develops symptoms consistent with COVID-19. The doctor estimates the probability of COVID-19 in this person as being 90% before the test is done.  This is the “prior” or “pre-test probability”. The test is performed. We can now do a simple calculation to estimate the new probability of the patient having COVID-19 after we get test results. There are online calculators to do these calculations for example this one at MedCalc.

However, to better understand how these calculations are done we can walk through the process.  The best way to do this is to assume that we have 1000 patients just like this.  We can then draw up a table to demonstrate the outcomes for all of these patients (see below).  If we assume a pre-test probability of 90% then of our 1000 patients, 900 will be infected and 100 not. These numbers go down as the bottom line of the table. We then use the test sensitivity and specificity to fill in the rest of the table. Given a sensitivity of 70%, of the 900 infected patients 630 should test positive (0.7 x 900 = 630).  These are the true positives. Given a specificity of 98% of the 100 who are not infected 98 should test negative (0.98 x 100 = 98).  These are the true negatives.  Given these values we can simply calculate the false negative and the false positives by subtracting the true results from the totals at the bottom.  For example, if there are 900 infected and only 630 have a positive test results, then the remaining 900 - 630 = 270 must be false negatives.
 
90% PRE-TEST PROB          INFECTED     NOT INFECTED          TOTALS
TEST POSITIVE 630           TRUE POS 2               FALSE POS 632          TOTAL POS
TEST NEGATIVE 270           FALSE NEG 98             TRUE NEG 368          TOTAL NEG
        TOTALS 900 100 1000
 
We can use these figures to see how we should update our belief in whether the patient has coronavirus or not based on the results of the test.   Firstly, let’s consider the case that the test came back positive.  How likely is it, in this group, that a person with a positive test has coronavirus?  The best way to do this is to calculate what percentage of the total number of positive tests are in fact true positives (true positives/total positive tests).  In this case it is 630/632 = 99.7%.  This is the positive predictive value (PPV).  This means that we can be virtually certain that if a person from this group has a positive result then they have coronavirus (99.7% certain in fact!). 

What about if the test came back negative?  How likely is it, in this group, that a person with a negative test truly doesn’t have coronavirus?  To work this out we need to calculate the negative predictive value (NPV).  This is the percentage of negative tests that are true negatives.  In this case it is 98/368 = 26.6%.  This means that a negative test will only be correct about a quarter of the time!  The result should be disregarded and the patient still considered to be infective.  In these cases, a different type of diagnostic test such as imaging may be useful.

Situation 2.

Another person shared a maxi-taxi with the person above for a short ride the next day. They sat in the front with the window open, the infected person right at the back. They have not developed any symptoms yet. Contact tracers ask them to be tested. The contact tracers estimate the probability of this person being infected with COVD-19 at 5%.  
 
5% PRE-TEST PROB          INFECTED     NOT INFECTED          TOTALS
TEST POSITIVE 35               TRUE POS 19               FALSE POS 54            TOTAL POS
TEST NEGATIVE 15               FALSE NEG 931             TRUE NEG 947          TOTAL NEG
        TOTALS 50 950 1000
 
In this case the PPV = 35/54 = 65%
And NPV = 931/947 = 98.4%

The most likely outcome is a negative result and this will be reassuring, we can be quite confident the person is not infected. A positive result however, is not definitive, a positive result means that they have a 2/3 chance of being truly infected. We should be very suspicious and keep this person isolated for the next couple of weeks and consider repeating the test after 2 or 3 days.

COVID-19 Serology Testing
COVID-19 serology testing is not yet widely available but it will be soon. Depending on how they are used, we will have different requirements of serology tests. Nucleic acid tests have a problem with false-negative test results. Serology tests have the opposite problem, they are subject to false-positive test results. This is because our blood teems with thousands of different antibodies to myriad different antigens that we have been exposed to in our lives. Some of those antigens are closely-related coronaviruses with quite similar structure to SARS-CoV-2. We need to design serological assays that have capture antigens that are unique to SARS-CoV-2 and will not react with antibodies against other coronaviruses and other fortuitously similar antigens. This is not a simple task and assays need extensive evaluation against ideally many hundreds or thousands of SARS-CoV-2 negative samples to prove they have high specificity.

If we are testing someone who has been ill, had a positive nucleic acid test and then recovered, the pre-test probability is 100%. The only reason we would do the test is to assess how strong their immune response was. The only caveat is that we have to be patient. Standard procedure is to take two blood samples for testing, one as early as possible in the first week of the illness and then another several weeks later and to look for a significant rise in the amount of antibodies present (the “titre”). False-positive test results are not a problem in this situation as we don’t expect to see any.

The second very important role for COVID-19 serology tests is to assess exposure in the general population. Most, but not all, authorities expect that people who have seroconverted to SARS-CoV-2 will have some degree of immunity against reinfection, at least in the medium term. Knowing the degree of immunity in the population will be vital information for modellers and help them plan a way out of our current social restrictions. However, evidence to date suggest that although asymptomatic infection with SARS-CoV-2 is not uncommon, it is likely that the prevalence of seroconversion against the virus in the general population will be quite low, especially in Australia where we have succeeded in bringing the infection rate down quite quickly compared with many other countries. This fact compounds the problem of false-positive test results with COVID-19 serology.
Let’s consider a scenario with three different tests.

Scenario 1.
Consider an Australian city or a group of mine workers where 5% of people have had COVID-19 but this is unknown to us. This is the “prior” or “pre-test probability” we use in our calculation. The serological testing is performed with a simple fingerprick blood sample point-of-care (POC) test that can be done at a doctor’s office or a mining site. Its sensitivity and specificity are both 95%. Some POC tests may have better test performance than this, some will have worse. We don’t yet have full independent evaluation data on the POC tests that have received emergency TGA approval in Australia.
 
5% PRE-TEST PROB     SEROCONVERTED NOT SEROCONVERTED          TOTALS
TEST POSITIVE 48               TRUE POS 47               FALSE POS 95              TOTAL POS
TEST NEGATIVE 2                 FALSE NEG 903             TRUE NEG 905            TOTAL NEG
        TOTALS 50 950 1000
 
The first thing to notice is that almost half of our positive test results are false-positive. The positive predictive value or PPV is given by true positives/total positives = 48/95 or 50.5%. A negative result is more helpful, the negative predictive value or NPV is given by true negatives/total negatives = 903/905 or 99.8%. The estimated prevalence of seropositivity in the population is total positives/1000 = 95/1000 or 9.5%. This is almost twice the true prevalence. This test is essentially useless for the modellers and also useless for individuals who receive positive test results.
 
Scenario 2.

Here we have what seems to be a better serology test. The sensitivity of this test is 98% and the specificity is also 98%.
5% PRE-TEST PROB     SEROCONVERTED NOT SEROCONVERTED          TOTALS
TEST POSITIVE 49               TRUE POS 19               FALSE POS 68              TOTAL POS
TEST NEGATIVE 1                 FALSE NEG 931             TRUE NEG 932            TOTAL NEG
        TOTALS 50 950 1000
 
For the modellers this testing might or might not provide useful information depending on how much their calculations are affected by error in the estimated seroprevalence parameter. The positive test rate of 68 positive results out of 1000 people tested (6.8%) is reasonably close to the true underlying prevalence (5%).  This information can be used to estimate the likely transmission dynamics in the population if restrictions were loosened.  However, for any individual who tested positive there is only a 49/68 = 72% chance that they have in fact had COVID-19.  This also means that 19 of the 68 people who tested positive have not had the disease and are therefore susceptible to getting it if they believe themselves immune and expose themselves to infection again. When the prevalence of the condition is low (5% in this case), we need a more specific test to have confidence in the results.

Scenario 3.

Here we have an improved serology test. The sensitivity is the same at 98% but the specificity is better at 99.8%, i.e. we only expect one false positive test result for every 500 people who definitively don’t have the disease. 
5% PRE-TEST PROB     SEROCONVERTED NOT SEROCONVERTED          TOTALS
TEST POSITIVE 49               TRUE POS 2                 FALSE POS 51              TOTAL POS
TEST NEGATIVE 1                 FALSE NEG 948             TRUE NEG 949            TOTAL NEG
        TOTALS 50 950 1000
 
This new test estimates the prevalence of seroconversion in the population as 51 positive tests out of 1000 tested or 5.1% which is almost exactly the true prevalence. Additionally, the problematic false positives have now been reduced to 2 out of 1000 tested but this still means that 2 out of 51 positive tests are false-positives. This increase in the specificity (increasing from 98% to 99.8%) has had a large impact on the usefulness of the test. Since we don’t know the prevalence of seroconversion in the population before we start, it is crucial that the test to be used has adequate sensitivity and very high specificity in particular when we expect the true seroprevalence to be low.

Whoever is using the test has to decide on the utility of the results. The information provided will be very useful for disease modellers. However, we have to decide if it is reasonable that 2 out of every 1000 people tested will be falsely reassured that they have had COVID-19 already.

The current COVID-19 crisis highlights how important it is to correctly interpret diagnostic tests and it is clear that Bayesian thinking is central to this. The Reverend Bayes could never have imagined that his theorem would one day be used to help guide government decisions in the setting of a global pandemic.

External links
TGA list of approved tests
Nature Biotechnology article on types of tests
Annals of Internal Medicine review
 
 

Last Review Date: September 2, 2020