Reanalysis of the efficacy of COVID-19 vaccines

A new study, posted to the medRxiv* preprint server, investigates the impact of a trial study design and other parameters on the estimates of vaccine efficacy. This study reanalyzes the data from the Janssen and Pfizer coronavirus disease 2019 (COVID-19) trials under a uniform protocol.

The study also suggests assigning a causal interpretation to per-protocol analyses which are typically performed in vaccine trials. Furthermore, this study also provides alternative methods to measure vaccine efficacy in settings with delayed immune responses.

Background

During this COVID-19 pandemic, several vaccines have been developed. The U.S. food and drug administration (FDA) issued emergency use authorizations to the Pfizer and Moderna mRNA vaccines in December 2020 and to the Janssen adenoviral vaccine in February 2021. This decision was based on the data obtained from phase 3 randomized double-blind, placebo-controlled trials to assess the safety and efficacy of these vaccines. This data is essential for policymakers to decide about the authorization and potential uses of vaccines.

Vaccine efficacy is determined by the relative reduction of the risk of developing the disease for vaccinated individuals compared to unvaccinated individuals or controls. The FDA and world health organization (WHO) have the threshold of 50% reduction of symptomatic disease and the lower limit for the confidence interval is above 30%. However, the efficacy estimates of the COVID-19 vaccines from the trials mentioned above were much higher than the FDA and WHO threshold.

In all the different trials, different factors have been considered to calculate vaccine efficacy. Therefore, the efficacy estimates cannot be compared across the trials.

Usually, a clinical trial adheres to the intention-to-treat analysis, but vaccine efficacy trials use per-protocol analysis where the population is restricted to eligible participants receiving all doses according to the schedule and only compliant individuals are included. There is a delay between vaccination and the development of immunity called the vaccine ramp-up time. Per-protocol analyses start after the vaccine ramp-up time. They estimate the intrinsic vaccine efficacy but are vulnerable to post-randomization biases.

The primary efficacy endpoint in phase 3 is determined as a clinical disease with laboratory confirmation. Calculations consider the relative risk of disease in vaccinated individuals compared to placebo individuals. Typically, cumulative incidence, incidence, or hazard rate is used as a risk measure to interpret vaccine efficacy estimates. The estimates may differ for each of these risk measures.

Per-protocol analyses are vulnerable to selection bias because of the comparison of subgroups that are selected after randomization. In the Janssen trial, the cumulative incidence was similar in the vaccinated and placebo-administered individuals until around day 14. After this, more cases accumulated in the placebo group. Approximately 18% of the placebo cases and 39% of the vaccine cases were observed during the first 14 days. Removing these will not introduce selection bias if the vaccine has no effect on infections and there are no side effects during the ramp-up period.

In the Pfizer trial, the cumulative incidence was again similar in the vaccinated and placebo-administered individuals until approximately 14 days after randomization. This trial specified a ramp-up period of 28 days during which there were approximately 34% of the placebo cases and 82% of the vaccine cases. In the AstraZeneca and Moderna trials, it seems that the vaccine already affected infection rates before the end of the specified ramp-up period. In the Moderna trial, vaccine efficacy was estimated using a hazard ratio. The individuals infected before day 14 after the second dose were censored. This may also lead to possible selection bias. None of the trials provided a rationale for the choice of the length of the ramp-up period in the protocol.

Findings and implications

This study argues that the intention-to-treat (ITT) estimates should be used because they provide information about the effectiveness of a public health strategy of using the vaccine, and they reflect the speed at which the vaccine offers protection. ITT estimates may be more meaningful than per-protocol estimates to compare vaccines with different dosing regimens or ramp-up periods. Recommendations include reporting both vaccine efficacy estimates.

This study focuses on COVID-19 vaccine trials. However, the authors comment that the methods are applicable to any vaccine trial. The AstraZeneca, Janssen, Moderna, and Pfizer vaccine trials had variable dosing regimens, endpoints, risk measures, ramp-up periods, and follow-up times. Also, the study length was different in these trials. These trials were conducted in different populations and considered different vaccine efficacy estimators. The intercurrent events were handled differently in these trials. The primary estimates were compared based on the per-protocol principle to evaluate efficacy and safety in preventing COVID-19.

When cumulative incidence is used as a risk measure, the corresponding vaccine efficacy represents the number of cases that can be avoided by vaccination. Interpreting the hazard and incidence rate is not straightforward but show more stability and result in approximately the same vaccine efficacy estimates in most settings with rare diseases.

There were differences up to 6 percentage points, depending on the chosen risk measure, in the data analyses of the Janssen and Pfizer COVID19 trials. For the Pfizer and Janssen COVID-19 vaccine trials, the ITT estimates were approximately 10% smaller than the per-protocol estimates.

This study proposes two hypothetical estimands that provide insight into the intrinsic effect of the vaccine where there is a delayed immune response. First, the vaccine efficacy that would have been observed if cases during the ramp-up could be prevented. Second, the vaccine efficacy if the ramp-up period can be eliminated. The authors have developed a novel estimator for this estimand using a Structural Distribution Model. This proposal includes modeling assumptions by comparing the modeled vaccine survival curve to the observed curve. This estimand targets an effect for the entire trial population.

Conclusion

In conclusion, the vaccine efficacy estimates from the different COVID-19 trials are quite comparable. However, there were differences in the study design. The per-protocol vaccine efficacy estimates can be interpreted as the number of cases that are vaccine-avoidable if the vaccine would provide immediate protection.

In other settings, the use of naive per-protocol effects may be problematic. The proposed estimand is recommended in those settings. This study proposes estimands applicable to the COVID-19 trials and other vaccine efficacy trials too.

*Important notice

medRxiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.

Written by

Dr. Shital Sarah Ahaley

Dr. Shital Sarah Ahaley is a medical writer. She completed her Bachelor's and Master's degree in Microbiology at the University of Pune. She then completed her Ph.D. at the Indian Institute of Science, Bengaluru where she studied muscle development and muscle diseases. After her Ph.D., she worked at the Indian Institute of Science, Education, and Research, Pune as a post-doctoral fellow. She then acquired and executed an independent grant from the DBT-Wellcome Trust India Alliance as an Early Career Fellow. Her work focused on RNA binding proteins and Hedgehog signaling.