One of the most frequent questions I’ve been getting recently is how accurate I think the covid tests are, and in particular the PCR tests. As it happens, a systematic review has recently been published in Evidence Based Medicine that looks at the covid tests (both PCR and antibody), so I thought it would be interesting to look in to the evidence together. This article gets a bit technical and math-heavy in places, so please bear with me. I think the payoff is worth it.
First, let’s make sense of what the two types of test are and how they work. The PCR (Polymerase Chain Reaction) test is designed to detect a specific sequence of nucleotides, and when it comes to detecting SARS-CoV-2, the sample is usually taken from the back of the throat. Nucleotides are the building blocks of genomes, and the idea is that if you can detect a string of nucleotides that is specific for a certain organism, then that proves the organism is present at the sample site. Since PCR is designed to detect bits of viral genome that are currently present in your respiratory tract, its purpose is to detect a currently active infection (as opposed to a past infection).
PCR works by repeating a series of chemical reactions over and over. If the sequence of nucleotides that is sought is present in the sample, then each time the reaction is repeated, the number of copies of the sequence will double, so that more and more copies accrue.
So, if you start of with one copy of the nucleotide sequence you are looking for, then after one cycle you will have two copies. After two cycles you will have four copies. After three cycles, you will have eight copies. After four cycles, you will have 16 copies. And so on. As you can see, the fact that each cycle doubles the number of copies means that the numbers quickly build to massive levels. The covid PCR tests frequently keep going up to 40 (or sometimes even 45) times.
If you start off with just one copy of the viral nucleotide sequence in the sample, then after 40 doublings, you will have over 1,000,000,000,000 copies (that’s one thousand billion copies). The reason you do this repeated cycle of doubling, is that once you get enough copies of the sequence you’re looking for, then you can use other technologies to detect it. For example, you can add molecules to the sample that visibly light up if enough copies of the sequence are present. So after enough copies are present in the sample, then they can be detected, and you get a positive result.
The number of times you choose to cycle through the steps of PCR before you decide that there was no virus in the sample after all is known as the cycle threshold. The number of cycles used to get a positve result is actually a pretty important number, because it tells you how much virus is in the sample. The lower the number of cycles required, the more virus is in the sample. The higher the number of cycles, the more likely that the result is a false positive, caused perhaps by having a tiny amount of inactive virus in the respiratory tract, or by contamination of the sample in the lab. Like I said, after 40 cycles, even a single copy of the viral sequence has become over one thousand billion copies.
One thing that’s important to understand at this point is that PCR is only detecting sequences of the viral genome, it is not able to detect whole viral particles, so it is not able to tell you whether what you are finding is live virus, or just non-infectious fragments of viral genome. If you get a positive PCR test and you want to be sure that what you’re finding is a true positive, then you have to perform a viral culture. What this means is that you take the sample, add it to respiratory cells in a petri dish, and see if you can get those cells to start producing new virus particles. If they do, then you know you have a true positive result. For this reason, viral culture is considered the “gold standard” method for diagnosis of viral infections. However, this method is rarely used in clinical practice, which means that in reality, a diagnosis is often made based entirely on the PCR test. A systematic review looking at the ability to culture live virus after a positive PCR test found that the probability of a false positive result increased hugely with each additional cycle after 24 cycles. After 35 cycles, none of the studies included in that review was able to culture any live virus.
In most clinical settings (including the one I work in), all the doctor is provided with is a positive or negative result. No mention is made of the number of cycles used to produce the positive result. This is a problem, since it’s clear that a positive result after 40 cycles is almost certainly a false positive, while a positive result after 20 cycles is most likely a true positive. Without information about the number of cycles, you have to assume that the patient sitting in front of you has covid and is infectious, with all the downstream consequences that entails.
Anyway, enough about the PCR test for now. The other main type of test is the antibody test. Here, the sample is usually taken from the blood stream. There are five different types of antibodies, but most antibody tests only look for one type of antibody, IgG, which is the most common type. Generally it takes a week or two after a person has been infected before they start to produce IgG, and with covid, you’re generally only infectious for about a week after you start to have symptoms, so antibody tests are not designed to find active infections. Instead the purpose is to see if you have had an infection in the past.
One common method that is used for antibody tests is ELISA (enzyme linked immunosorbent assay). In this method, you have a plate on which you’ve fixed antigen that the antibody you are looking for can bind to (antibodies bind to antigens – antigen is short for “antibody generator”, and it’s basically the molecular structure that a certain antibody is specifically designed to bind to).
You then add the blood sample that you want to study to the plate, at which point the antibodies in the sample will bind to the antigens (assuming the antibodies you want to find are actually present in the sample). After that you wash the plate, so that any other antibodies in the sample that you’re not actively looking for are washed off (since there’s no antigen for them to bind to).
Next you add a signaling molecule that can bind to antibodies, and which has the ability to change color when exposed to a certain enzyme. You then wash the plate again. If there are no antibodies stuck to the plate for this molecule to bind to, it will wash off. If the antibodies you are looking for were present in the blood sample, they will have stuck to the antigen on the plate, and this new molecule will in turn have stuck to them.
Finally you add an enzyme that changes the color of the signaling molecule. If the signaling molecule hasn’t been washed off in the previous step, then you will see the plate change color, and the antibody test is positive.
Apart from understanding how the tests work, we also need to understand two important terms before we get in to the details of the recent systematic review. Those terms are sensitivity and specificity, and they are critical for all diagnostic tests used in medicine, because they tell you how good a test is.
Sensitivity is the probability that a disease will be detected if the person actually has the disease. So, for example, a test for breast cancer with a sensitivity of 90% will detect breast cancer 90% of the time. Nine out of ten patients with breast cancer will correctly be told that they have the disease. One out of ten will incorrectly be told that they don’t have the disease, even though they do.
Specificity is the opposite of sensitivity. It is the probability that a person who doesn’t have the disease will be told that they don’t have the disease. So, a specificity of 90% for our imaginary breast cancer test means that nine out of ten people who don’t have breast cancer will be correctly told that they don’t have it. One out of ten people who don’t have breast cancer will incorrectly be told that they do have it.
To put it another way, sensitivity is the ability of a test to detect true positives. Specificity is the ability of a test to avoid producing false positives. A perfect test will have a sensitivity and specificity of 100%, which would mean that it catches everyone who has the disease, and doesn’t tell anyone they have the disease if they don’t. No such test exists. In general, sensitivity and specificity are in conflict with each other – if you push one up, the other will go down.
If I just told everyone I meet that they have breast cancer, my sensitivity for detecting breast cancer would be 100%, because I wouldn’t miss a single case, but my specificity would be 0%, because every single person who doesn’t have breast cancer would be told that they do. So, when designing a test, you have to decide if you’re going to maximize sensitivity or specificity. If you design a covid PCR test with a cycle threshold of 40, then you are going for maximal sensitivity – the probability of missing a case is minimized, but you’re going to get a lot more false positives than if you set the threshold at 30.
Ok, now that we know what a PCR test is and what an antibody test is, and understand sensitivity and specificity, we can move on to the recent systematic review. The review included 38 studies of PCR tests (and LAMP tests, an alternative technique that is similar to PCR). The overall sensitivity for PCR/LAMP was between 75% and 100% in the different studies, while the overall specificity was between 88% and 100% . 16 studies, with a total of 3,818 patients, were able to be pooled together to get a more accurate estimate of sensitivity. In the pooled analysis, sensitivity was determined to be 88% . It wasn’t possible to determine a pooled specificity value, since the studies included in the pooled analysis were all of people who were already known with complete certainty to be infected with covid.
The review included 25 studies of antibody tests, but only ten of these (with a total of 757 patients) provided enough data to allow sensitivity to be calculated. The sensitivity of the antibody tests varied from 18% to 96%. 12 studies provided enough information for specificity to be determined, and in these it varied from 89% to 96% .
Ok, it might be hard to understand what these numbers mean in practical terms, so we’re going to play around with them a bit in order to clarify this, and I’m going to focus on the PCR test in this final discussion, since that is what’s generating much of the hysteria around covid. As mentioned, the sensitivity of the PCR test seems to be around 88% . A good value for the specificity is harder to determine, but it’s somewhere between 88% and 100%, so if we assume a specificity of 94% (halfway between the two values) we’re probably not far off.
Let’s say the disease is spreading rampantly through the population, and one in ten people are infected at the same time. If we test 1,000 people at random, that will mean 100 of those people actually have covid, while 900 don’t. Of the 100 who have covid, the test will successfully pick up 88. Of the 900 who don’t have covid, the test will correctly tell 846 that they don’t have it, but it will also tell 54 healthy people that they do have covid. So, in total 142 people out of 1,000 are told that they have covid. Of those 142 people, 62% actually have the disease, and 38% don’t.
That’s not great. Four in ten people getting a positive test result don’t actually have covid, even in a situation where the disease is so common that 10% of people being tested really do have the disease.
Unfortunately, it gets worse. let’s assume the disease is starting to wane, and now only one in a hundred people being tested actually has covid. If we test 1,000 people, that will mean ten will really have covid, while 990 won’t. Of the ten who have covid, nine will be correctly told that they have it. Of the 990 who don’t have it, 931 will be correctly told that they don’t have it, while 59 will be incorrectly told that they do have the disease. So, in total, 68 people will be told that they have covid. But only 9 out of 68 will actually have the disease. To put it another way, in a situation where only 1% of the population being tested has the disease, 87% of positive results will be false positives.
There is another thing about this that I think is worth paying attention to. When one in ten people being tested has the disease, you get 142 positive results per 1000 people tested. But when one in a hundred has the disease, you get 68 positive results. So, even though the actual prevalence of the disease has decreased by a factor of ten, the prevalence of PCR positive results has only decreased by half. So if you’re only looking at PCR results, and consider that to be an accurate reflection of how prevalent the disease is in the population, then you will be fooled, because the disease will seem to be much more prevalent than it is.
Let’s do one final thought experiment to illustrate this. Say the disease is now very rare, and only one in a thousand tested people actually has covid. If you test 1,000 people, you will get back 61 positive results. Of those, one will be a true positive, and 60 will be false positives. So, even though the prevalence of true disease has again decreased by a factor of ten, the number of positive results has only decreased slightly, from 68 to 61 (of which 60 are false positives!). So by looking just at positive PCR tests, you can easily be convinced that the disease is continuing to be roughly as prevalent in the population, even as it goes from being present in one in a hundred people to only being present in one in a thousand. The rarer the disease becomes in reality, the less likely you are to notice any difference in the number of tests returning positive results.
I want to restate this again, in a slightly different way, to make sure the message sinks in. As the disease drops enormously, by a factor of 100, from affecting one in ten to one in a thousand tested people, there is little more than a halving in PCR positive results, from 142 to 61. So a huge reduction in real infections only causes a small reduction in PCR confirmed “cases”. In fact, the disease could vanish from the face of the Earth, and you would still be getting 60 positive results for every 1,000 tests carried out!
The same trend is seen even if the PCR test were to have a much better specificity than we are estimating here, of say 99% . Here’s a quick illustration, since I don’t want to tire you with too many more numbers. If one in ten has the disease and you test 1,000 people, you will get back 97 positive results, of which 88 will be true positives and 9 will be false positives. If one in 100 has the disease, you will get back 19 positive results, of which 9 will be true positives and ten will be false positives. If one in 1,000 has the disease, you will get back 12 positive results, of which 11 will be false positives.
So, even if the test has a very high specificity of 99%, when the virus stops being present at pandemic levels in the population and starts to decrease to more endemic levels, you quickly get to a point where most positive results are false positives, and where the disease seems to be much more prevalent than it really is.
As you can see, the less prevalent the disease is in reality, the more likely the test is to generate a false positive result, and the less useful the test is as a method for figuring out who actually has covid. And the less prevalent the disease is, the more prevalent it will seem to be in relation to reality. If decisions about covid continue to be made largely based on what PCR tests show, we might never be able to call off the pandemic!
And that, ladies and gentlemen, is why PCR positive cases are a very poor indicator of how prevalent covid is in the population, and why we should instead be basing decisions on the rates of hospitalization, ICU admission, and death. If we just look at the PCR tests, we will continue to believe that the disease is widespread in the population indefinitely, even as it becomes less and less common in reality. And that is assuming the rate of testing doesn’t increase. If we combine this built-in problem with accuracy, with a massive increase in testing (as has happened in most countries over the course of the pandemic), then we can create the impression of a disease that is continuing to spread wildly through a population, even when it isn’t.
You might also be interested in my article about how deadly covid actually is, or, if you want to dig further in to the problems created by testing, you might be interested in my article about breast cancer screening.