Suppose that a drug company is trying to get approval for a new pain medication that might have some serious negative side effects. How can the company keep regulators from finding any?

Even if you have no training in statistics it is easy to understand that the sure-fire strategy is to focus attention on side effects that are rare. The smaller the number of events, the easier it is to dismiss the few that arise as chance outcomes.

This same strategy for avoiding a discovery is being used now by people who want to keep us from finding that the protection from vaccines diminishes over time.

Negative Side Effects of a New Pain Medication

Imagine that some researchers look for side effects by observing one group of ||100|| people who are taking the new pain medication and ||100|| matched individuals who are not. The research plan is to count the number of people in each group who have a heart attack.

During the year of the study, ||3|| people in the group taking the drug have a heart attack versus only ||1|| heart attack in the other group. These numbers – ||3|| and ||1|| – are so small that the difference really could be due to chance fluctuations. Using their jargon, the researchers report that “there is no statistically significant difference in the number of heart attacks between the two groups.”

But the researchers also measured the weight of the study participants at the beginning and end of the year. They notice that ||80|| people taking the new pain medication gain more than ||20|| pounds over the course of the year, but only ||5|| in the group that did not take the drug gain this much. With numbers this large – ||80|| versus ||5|| – the researchers are confident that the difference between the two groups is not due to chance. “There is a statistically significant increase in weight among those taking the pain medication.”

The advocates for the company keep repeating that “weight gain is an unimportant endpoint. Moreover, it is not what our study was designed to assess.” You don’t care. You look at all the evidence and pay attention when the numbers are big enough to reveal something important.

You understand that “no statistically significant effect on the incidence of heart attack” is an opaque way of saying that the researchers learnt nothing by counting heart attacks. The true effect of the pain medication could be to triple the incidence of heart attacks, to have no effect, or to cut it in half. On the basis of the heart attacks alone, you have no clue.

But because you have some common sense, you pay attention to the fact that the data on weight reveals statistically significant evidence that the pain medication does indeed have worrisome side effects.

You wouldn’t let the drug company get away with a burden of proof game in which they can exclude evidence that could be statistically significant and allow in only evidence that can’t. You wouldn’t approve the new medication.

Rare Events and Small Subgroups

When people want to encourage an official decision of “no effect,” they recommend measures that yield low numbers. In a study with realistic numbers of participants, the way to get low numbers is to count only events that are rare.

This strategy has worked well those who do not want an official finding that the protection offered by vaccines has been decreasing over time. All they had to do was to insist on measures of rare events such as “breakthrough infections that require hospitalization.” In the analogy with the pain medication, hospitalizations are like heart attacks; breakthrough infections are like weight gain. There are enough of these infections that in a study with a realistic number of participants has a realistic chance of finding statistically significant evidence of decreasing protection against infections if it is truly decreasing.

I’m not sure why so many researchers are opposed to approving boosters. Perhaps they think that drug companies are “bad guys”. But whatever their motivation turns out to be, they are pushing for a conclusion of “no change in protection” by putting the focus on only the number of hospitalizations. Relative to breakthrough infections, this reduces the number of events by an order of magnitude.

But what if there is a large study with so many subjects that there are enough hospitalizations to find a statistically significant fall in protection? Easy. Just reduce the numbers once again by insisting on an analysis of a smaller sub-group of the entire study population. “But is the protection falling among people between the ages of ||16|| and ||40||?”

No matter how big the original study was, there is always an event that is rare enough and a subgroup that is small enough to make sure that the conclusion will be “no statistically significant reduction in protection.”

Illustrative Numbers For Breakthrough Infections

To show how this plays out with some realistic numbers, consider the Mayo clinic study of vaccine effectiveness in Minnesota. It relied on data for about ||60,000|| people – a group of ||20,000|| who were not vaccinated, a matched group of about ||20,000|| who were vaccinated with the Pfizer vaccine, and a matched group of about ||20,000|| vaccinated with the Moderna vaccine. You’d think that with this many subjects, we could learn something.

Here is a new version of the table from a previous post that shows the actual number of breakthrough infections and hospitalizations for the Pfizer an unvaccinated groups:

If we restrict attention to only the hospitalization data, there is no way to say anything about what the trend is in the Pfizer group. (To be precise, what the researchers were interested in was the trend in the Pfizer group relative to the trend in the unvaccinated group, but the logic is the same.) The number of events is too low.

But look at the numbers for the more frequent event – breakthrough infections – for the Pfizer group. These are big enough to reveal an unambiguous, statistically significant upward trend that exceeds the trend in the unvaccinated group. This means a statistically significant fall in the protection against infection offered by the Pfizer vaccine.

In this figure, the shaded area shows the ||95\%|| confidence interval for effective protection in each month. A value of ||100\%|| would mean no fall in protection. By July, the effective protection in the Pfizer group had fallen to about ||50\%||, which is statistically significantly different from both ||95\%|| for Pfizer in March and from the ||75\%|| protection in the Moderna group in July.

So What?

“But if breakthrough infections are rare, can’t we can just ignore them?”

To see, we have to scale things up to the size of the US population. To keep the calculation simple, assume that the US soon reaches the point where ||60\%|| of the entire population is fully vaccinated. This would mean roughly ||200|| million people who are at risk each month for a breakthrough infection.

Also, looking forward, assume that the prevalence of infection in the whole US is roughly the same as the prevalence of infection in April in Minnesota, ||30|| people per day per ||100,000|| residents. As I write, in Sept. of 2021, the US is running at the rate of 45 new infections per day per ||100,000|| people, so ||30|| relies on the optimistic assumption that the prevalence falls."

If we scale up the numbers from the Mayo study from ||20,000|| people to ||200|| million people, we should scale up the study finding of ||11|| breakthrough infections per month to ||110,000|| of them per month. If about one tenth of them end up being hospitalized, we should expect ||11,000|| new hospitalizations in the US each month among people who are double vaccinated.

||10,000|| new hospitalizations per month among people who are double vaccinated? This doesn’t seem like something we can just sweep under the rug.

How Much Difference Could Boosters Make

The Mayo study found a statistically significant fall in the effectiveness of protection against infection for the Pfizer vaccine from an initial level of about ||90\%|| to a value a few months later of about ||50\%||. If the booster gets people back to ||90\%|| protection against infection, this would mean that instead of having ||100,000|| new breakthrough infections each month, we would have only ||20,000|| of them.

What can we say about the effect on hospitalizations? From the data, if we do as we are encouraged to do and ignore the evidence on breakthrough infections, we cannot say anything on the basis of the trend in hospitalizations alone.

If we can reduce the number of new breakthrough infections that occur every month from ||100,000|| to ||20,000||, is it realistic to assume that the number of hospitalizations that result from these infections will remain constant at ||10,000|| per month?

If the number of breakthrough infections goes down by a factor of ||5||, the best estimate is that the number of hospitalizations will also go down by a factor of ||5||.

If everybody who is eligible gets a booster, boosters could reduce hospitalizations among people who are double vaccinated from ||10,000|| new admissions per month to ||2,000|| admissions per month.

Will Everyone Who is Eligible Get Boosters?

If boosters are allowed, some people who could get them won’t get them. So a reduction from ||10,000|| to ||2,000|| is not a prediction about the aggregate numbers. Instead, what it suggests is that in the population as a whole, getting a booster will reduce the probability of getting an infection that requires hospitalization by a factor of ||5||. Among people like me who are over the age of ||60||, the reduction will be much higher.

So boosters sound good to me. And to the many people I know who are already breaking the law to get a third dose.

So if you are one of those experts who are trying to prevent the FDA from approving boosters, explain to me why, at a time when we are desperate to get more people to accept more doses of vaccines, you want to make it illegal for someone like me who will actually accept an additional dose to get one?