Hypothesis Tests for Vaccines
Hypothesis Testing is commonly used when trials for vaccine testing are performed. With all the recent vaccine news for Covid-19, I wanted to shed light on the general hypothesis testing process for vaccines. In this article, I will go over the following topics:
· Why is Hypothesis Testing Even Needed for Vaccines?
· Overview of Hypothesis Testing for Vaccines
· Trade Off between Type I vs Type II Errors
· Common Problems with Hypothesis Testing in Vaccines
Why is Hypothesis Testing Even Needed for Vaccines?
It is imperative to test a vaccine before it is accessible because researchers want to ensure that the vaccine is effective in stopping people from getting a disease and the vaccine does not have any other negative side effects. To reduce these risks, hypothesis testing is leveraged. Researchers try to minimize the disparity between anticipated and true outcomes. The tinier the disparity, the more confident researchers are with the potential outcome of the vaccine. Hypothesis testing’s main objective is to evaluate the evidence from the data and provide a metric which helps to quantify if the null hypothesis is likely given the data from the study being researched.
Overview of Hypothesis Testing for Vaccines
In this section, I will review basic hypothesis testing by providing an overview and important terms and concepts with a straightforward example. This will help to clarify hypothesis testing for vaccines before we discuss prioritization of type I and type II errors and common problems with hypothesis testing in vaccines.
Hypothesis tests are a formal way of carrying out the scientific method. It aims to show that a notion ought to be rejected if the evidence from the data is very different from what the notion forecasts.
Hypothesis testing has two main hypotheses, the null hypothesis (H0) and the alternative hypothesis (HA). The null hypothesis normally states that there is no difference between two processes or samples. A researcher’s null hypothesis for a vaccine for example is:” The Covid-19 vaccine makes no difference in whether or not the people in the testing sample contract Covid-19.” The researcher will want to prove that the null hypothesis is not true in their study by rejecting the null hypothesis. They prove the alternative hypothesis, that the Covid-19 vaccine made a difference in whether or not the people in the testing sample contract Covid-19.
The researcher uses a test static (T) to calculate the amount of similarity between the test data and that the null hypothesis is true. The p-value helps the researcher quantify the significance of the Covid-19 vaccine by showing a difference in whether or not the people in the testing sample contract Covid-19. The p-value is the minimum significance level (or alpha) in which the result is rejecting the null hypothesis; it shows that the results are probably not by chance but the probability that the results show the vaccine works are valid. On the other hand, the critical region is a set of values in a region of values that result in the rejection of the null hypothesis at a predetermined probability.
If the p-value is less than .05(alpha) we reject the null hypothesis and accept the alternative hypothesis.
The researcher assigns a specific significance level which is the probability of rejecting a true null hypothesis. An alpha/significance level of 5% results in a 5% risk of a researcher rejecting the null hypothesis when it is in fact true. This scenario is referred to as a Type 1 error or a false positive, when we think that the Covid-19 vaccine made a difference in whether or not the people in the testing sample contract Covid-19 but there is no actual difference. On the other hand, if the researcher fails to reject the null hypothesis when the vaccine for Covid-19 actually made a difference in whether or not the people in the testing sample contract Covid-19, then it is a Type II error (false negative).
Trade off Type I vs Type II Errors
Although one would hope the risk associated with receiving a vaccine is none, that is unfortunately not the case. To minimize risk, large sample sizes and small significance levels are leveraged when carrying out the testing. Some risks though, can’t be mitigated, Type I and Type II errors share a see-saw relationship, where some type of trade-off between the two is needed to adjust the risk. A decision must be made based on the type of hypothesis test being run to prioritize either the Type I error or the Type II error.
Let’s use an example to help illustrate:
H0: Old Flu Vaccine and New Flu Vaccine are the same
HA: New Vaccine is superior to Old Flu Vaccine
Type I Error: New vaccine and old vaccine are the actually the same, but we believe the new vaccine is superior to the old flu vaccine
Type I Error Outcome: Company manufactures and releases a new flu vaccine, which results in more revenue for the company but also society wastes their efforts and time on the new vaccine
Type II Error: The new vaccine is actually superior to the old flu vaccine, but we believe the old flu vaccine and the new flu vaccine are the same
Type II Error Outcome: The company does not manufacture or release a new vaccine. Flu severity could have been minimized from the new vaccine but since it was not, society was negatively impacted. Lives could have been saved and the disease could have spread less.
In this example, the type II error would be considered more of an issue. Thus, the type II error would try to be minimized. This is not always the case. Depending on the hypothesis test and the outcomes of the type I and II errors, certain steps are taken. If a false positive is considered more of an issue than a false negative, the type I error will be minimized so a small alpha/significance level is chosen. The researcher/company will be apprehensive to reject the null, and so strong proof from the data will be required to support the alternative hypothesis. If instead the false negative is considered more of an issue than the false positive, the researcher chooses to minimize a type two error, a higher alpha/significance level is prioritized so strong results will be needed to fail to reject the null hypothesis when the null hypothesis is in fact actually false.
Common Mistakes/Overlooks of Hypothesis Tests for Vaccine Studies
The goal of a vaccine is either to avoid people from getting sick or to reduce a disease’s symptoms. Vaccine studies can either be observational or experimental studies. Hypothesis testing for vaccines requires the researchers to know statistical principles, the science behind the research, and how to integrate them. If any of these three qualities are missing, the hypothesis test is in danger of not being properly designed or carried out appropriately.
Really understanding the metrics
Hypothesis Testing is so ubiquitous when studying the effect of vaccines but sometimes researchers do not fully understand the concept of p-values. Setting alpha to a standard .05 can be meaningless and misunderstood. The threshold used should consider the risk appropriately (such as the type I and type II errors we discussed above). It is important to have a large enough sample size because small sized samples have a lot more variability in their results which directly effects the p-value. P-values are not an easily reproducible metric so it is important that when there are mistakes in the model development or computer calculations, that the researcher understands the metrics to better be able to pick up on issues that may arise.
Developing the right hypothesis
Suitable hypotheses for testing vaccines use known facts, are simple to comprehend, address a problem and proposed resolution, and are able to be tested. It is important to understand both the statistical theory and science background when establishing the hypothesis. Depending on the vaccine study and what is being looked at, developing the null hypotheses can be more difficult at times. The null hypothesis can vastly change the test being run, what is being looked at, and even not picking the appropriate type of test static. By not using the right test statistic, then there is an increased risk of the correct answers are not being answered.
Capturing all results in testing
Vaccines studies address a specific disease, but hypothesis tests are also carried out to look out for possible side effects of the vaccine. Most vaccine trials hope to represent a population similar to either the general population or the population who will be receiving the vaccination. Since the sample size is not the actual population slight differences may arise, especially when the sample population is not very large. Because of this, severe but uncommon side effects can be missed during the trial period. Additionally, long terms effects may not appear from the time of the vaccine trial up to when the public is using the vaccine. By missing severe or long-term side effects, data/evidence is missed from the test. It is ideal to set up your test to capture as much data as possible. The potential result from missing data during the vaccine hypothesis testing, is that the lack of data failed to support the null hypothesis.
Conclusion
We can clearly see that hypothesis testing is an integral part of vaccine testing. To ensure vaccine hypothesis tests are being run successfully, it is imperative to minimize risk and ensure capturing as much evidence as possible. Vaccine studies should try to ensure large diverse population sizes with trial periods that include long term effects. If they are unable to do so, they should at least ensure that they are heavily monitoring results after the vaccine is being used by the general population. One way to continue monitoring those who have received a specific vaccine is to monitor hospital or insurance claims data. The effectiveness of a vaccine hypothesis test is clearly largely determined by having as much data and evidence as possible.
Bibliography
Hu, Zonghui, and Michael Proschan. “Two-Part Test of Vaccine Effect.” Statistics in Medicine, vol. 34, no. 11, 20 May 2015, pp. 1904–1911, www.ncbi.nlm.nih.gov/pmc/articles/PMC4393783/, 10.1002/sim.6412. Accessed 21 Nov. 2020.
Kim, Hae-Young. “Statistical Notes for Clinical Researchers: Type I and Type II Errors in Statistical Decision.” Restorative Dentistry & Endodontics, vol. 40, no. 3, 2015, p. 249, 10.5395/rde.2015.40.3.249.
Kulldorff, Martin, et al. “A Maximized Sequential Probability Ratio Test for Drug and Vaccine Safety Surveillance.” Sequential Analysis, vol. 30, no. 1, 21 Jan. 2011, pp. 58–78, 10.1080/07474946.2011.539924. Accessed 21 Nov. 2020.
Ramzai, Juhi. “Holy Grail for P-Values and How They Help Us in Hypothesis Testing.” Medium, 2 May 2020, towardsdatascience.com/holy-grail-for-p-values-and-how-they-help-us-in-hypothesis-testing-bce3d0759604.