One of the common themes of SBM is that medical evidence can be very complex. In order to come to any robust conclusion about what the science says on any question you need to do a careful and thorough evaluation of all the relevant evidence, including plausibility as well as observational and experimental clinical evidence. Essentially we want to look at the question from every angle and make the best inference possible to the most likely answer.
What you cannot do is look at a single study, in the absence of any other context, and come to a firm conclusion about what the study says. You can sort-of do this with large rigorous clinical trials, but that is only because those trials sit on top of a mound of other research which tell us how to interpret that study, and told us how to design that study in the first place.
Despite this complex reality, the media and pundits will happily tout any individual study that makes good click-bait or appears to support their ideological stance or their partisan team. This means that SBM and other responsible science communicators have the endless task of trying to put those individual studies into a deeper context. It’s definitely a finger-in-the-dike situation, and worse, it takes much more effort explain all the complexity of a study than to just tout its apparent bottom-line finding. So we also try to give people the tools to at least understand how misleading individual studies can be.
With that background, there is a study going around with claims that the COVID vaccines increase the risk of contracting COVID. To be clear, the authors are not claiming this, and the study does not show this, but there are some unexpected and quirky results that superficially can be presented this way. Here is the quote from the results that people are focusing on: “The risk of COVID-19 also increased with time since the most recent prior COVID-19 episode and with the number of vaccine doses previously received.”
This is not a misquote or taken out-of-context – this is an accurate reflection of the data. The problem is with the interpretation of this result. This was an observational study, and as I have to endlessly point out, observational studies are not controlled and are therefore subject to confounding factors. It is extremely common for unusual confounding factors to rear their head in observational studies, resulting in counterintuitive results. The medical literature is rife with such studies – they don’t mean anything until the result is replicated and diced-and-sliced every way possible. Even then, it may take decades in some cases to hunt down and identify the confounding factor giving the quirky result.
What the study actually showed is that being vaccinated correlated with a lower risk of being diagnosed with COVID. They found: “The estimated vaccine effectiveness was 29% (95% confidence interval, 21%–37%), 20% (6%–31%), and 4% (−12% to 18%), during the BA.4/5-, BQ-, and XBB-dominant phases, respectively.” What they concluded was that the effectiveness at this time is modest and depends heavily on the strains of the virus that are dominant at the time. That was the point of the study – as the virus mutates, are the vaccines maintaining their effectiveness. As every expert in the world predicted, as the virus mutates the vaccine will become less well targeted. This is why boosters are necessary, as with the flu vaccine, to track the current dominant strain. What this study means is that perhaps we need to update the vaccine more frequently and better track this virus as it mutates.
So – if the vaccine reduces the chance of being infected, how is it possible that increased number of doses results in a greater chance of being infected? This makes no mechanistic sense. This result screams confounding factor. At this point we can only speculate what factors might be at work, and then perhaps do follow up studies to test those hypotheses. The authors speculate: “It is possible that the association of number of prior vaccine doses with increased risk of infection may have been confounded by time since last prior exposure to SARS-CoV-2.”
But there are many other possible confounding factors. People could take tests at home without ever being registered as infected. There could be many behavioral difference between those choosing to get vaccinated and those who do not. Perhaps, for example, being at high risk for infection causes people to take every booster shot. Also we need to keep in mind the overall infection rate was low, 8.7%. This means that small confounding effects could affect the outcome.
One major limiting factor of this study, in my opinion, is that it looked at infection but not markers of severity of infection. This is tricky because there are many things that affect whether or not someone who is infected gets diagnosed and documented. The door for confounding factors is wide open. It is far better, for an observational study, to use hard outcomes that are easier to track, such as ER visits, hospitalizations, and death. These are not free from confounding factors, but should be much less than mere infection status.
For example, here is a study from this month in the NEJM, which found:
“At 6 months of follow-up, the estimated vaccine effectiveness was 29.3% (95% confidence interval [CI], 19.1 to 39.2) against Covid-19–associated emergency department visits (risk difference per 10,000 persons, 18.3; 95% CI, 10.8 to 27.6), 39.2% (95% CI, 21.6 to 54.5) against Covid-19–associated hospitalizations (risk difference per 10,000 persons, 7.5; 95% CI, 3.4 to 13.0), and 64.0% (95% CI, 23.0 to 85.8) against Covid-19–associated deaths (risk difference per 10,000 persons, 2.2; 95% CI, 0.5 to 6.9).”
That’s a 29.3% reduction in ER visits, 39.2% reduction in hospitalizations, and a 64% reduction in COVID related deaths. These outcomes are easier to quantify, and are much more clinically relevant. It has also become clear over time that the vaccines are much better at preventing infections from being severe than preventing infection at all.
But of course, this is also a single study, so let’s look at a recent systematic review of 284 articles. They found that: “All the approved vaccines were found safe and efficacious but mRNA-based vaccines were found to be more efficacious against SARS-CoV-2 than other platforms.”
We can say with confidence, based on hundreds of studies, including the initial placebo-controlled trials, that the approved COVID vaccines are effective against COVID. They are better with strains that they track more closely. They are also better at reducing severe illness than completely preventing infection. But in every way – they are safe and effective.
