One Thursday in October, the Food and Drug Administration’s vaccine advisory committee gathered to discuss booster shots for Moderna and Johnson & Johnson’s COVID-19 vaccines. Yet one of the first presentations wasn’t given by Americans — it featured scientists from Israel’s Ministry of Health and Weizmann Institute. And the presentation wasn’t about Moderna or Johnson & Johnson’s vaccines — the Israelis shared their findings from the country’s Pfizer booster shot campaign.
This was less than ideal. Not only did the presentation focus on a different vaccine from the two up for discussion, it also centered around a population that’s much smaller and more homogenous than America’s. Plus, Israel’s vaccine rollout happened earlier and faster than the U.S.’s — meaning that the population had more vaccine-induced immunity but was potentially more vulnerable to the impacts of waning protection.
Israeli presenters provided slide after slide showing the power of booster shots. But not all of the FDA advisors were convinced. “What they’re seeing in Israel is not necessarily what we’re seeing here in the U.S.,” said Dr. Archana Chatterjee, dean of Chicago Medical School and member of the advisory committee, during the meeting. In an interview with FiveThirtyEight, she explained that Israel’s data is “interesting and very compelling,” but that Israel differs from the U.S. on key characteristics: Namely, a higher share of the Israeli population is inoculated, and a larger proportion of breakthrough cases in Israel led to hospitalization prior to the booster shot rollout. As a result, she said, Israel had a clear need for additional shots to bump up immunity. In the U.S., meanwhile, the vaccines were still highly protective against severe COVID-19 disease and death.
Chatterjee said that her eventual votes — in favor of booster shots — were not based on data from Israel. Still, the Israeli scientists’ very presence at the meeting demonstrated the shortcomings of the U.S. health system. If the U.S. doesn’t comprehensively track its own data, it has to rely on other countries to tell it how to keep Americans safe. Meanwhile, without clear evidence that they can refer to in making their own COVID-19 decisions, many Americans have been confused about whether they are eligible for — or even need — a booster shot.
Israel has a universal health care system for all citizens and permanent residents. So does the U.K., another country that the U.S. looks to for COVID-19 data. Beyond the health care benefits that such policies provide to residents, universal health care has a clear advantage for data scientists seeking to answer medical questions. When every person in the country is plugged into the same health care system, it’s very easy to standardize your data.
“During the pandemic, very rapidly, detailed datasets were created that link primary care records, secondary care records, national testing data, and vaccination data,” said Jonathan Sterne, a statistician at the University of Bristol who studies vaccine effectiveness in the U.K.
The standardized COVID-19 data “enabled extraordinary research based on the whole population,” Sterne said. Once a researcher gained National Health Service authorization, they could essentially download anonymized health records for the entire country, with information ranging from a patient’s most recent COVID-19 test to their body mass index.
In the U.S., vaccine research is far more complicated. Rather than one singular, standardized system housing health care data, 50 different states have their own systems, along with hundreds of local health departments and thousands of hospitals. “In the U.S., everything is incredibly fragmented,” said Zoë McLaren, a health economist at the University of Maryland Baltimore County. “And so you get a very fragmented view of what’s going on in the country.”
Related: How Are Kids Handling The Pandemic? We Asked Them. Read more. »
For example, a database on who’s tested positive in a particular city might not be connected to a database that would reveal which of those patients was vaccinated. And that database, in turn, is probably not connected to health records showing which patients have a history of diabetes, heart disease or other conditions that make people more vulnerable to COVID-19.
Each database has its own data fields and definitions, making it difficult for researchers to integrate records from different sources. Even basic demographics such as age, sex, race and ethnicity may be logged differently from one database to the next, or they may simply be missing. The Centers for Disease Control and Prevention, for instance, is missing race and ethnicity information for 35 percent of COVID-19 cases as of Nov. 7.
The complications extend to outside researchers. In one recent analysis, researchers at the New York State Department of Health linked the state’s COVID-19 testing, hospitalization and vaccination datasets to examine how well the vaccines worked during the state’s surge of the delta variant of the coronavirus. The study revealed that the vaccines became less effective at preventing COVID-19 cases among New Yorkers while remaining highly protective against hospitalization. But the researchers were not able to look at the medical conditions, job occupations or other risk factors for those who got breakthrough cases.
Eli Rosenberg, an epidemiologist at the New York Department of Health who led the study, said that combining datasets can provide some useful statistics to inform vaccine policy. Yet he acknowledged that this type of analysis is limited because researchers can’t look at patients’ entire medical records. Meanwhile, in the U.K., Sterne and other researchers have easily compared COVID-19 metrics with pre-pandemic information. For instance, “people who had a flu jab in 2019 are much more likely to have had a COVID-19 vaccine in 2021,” Sterne said.
Without a unified dataset allowing U.S. researchers to analyze how well the vaccines are working, policymakers are left with limited information to make crucial decisions, such as determining who should be first in line for a booster shot.
“We talk about evidence-based medicine, and we also care about evidence-based policy,” McLaren said. “When the quality of data is poor, the quality of our policy is going to be worse.”
With poor data, we’re essentially “flying blind” into poor policy, McLaren said. Leaders can make poor decisions, or they can decide on a type of policy she calls “one size fits all.” U.S. booster shot policy is one example. All vaccinated Americans who are over age 65, have a medical condition that increases risk of severe COVID-19 or live or work in an environment that’s susceptible to COVID-19 outbreaks are eligible for a booster. About 89 percent of American adults will qualify after enough time has passed from when they got their original shots, according to an analysis by the Computational Epidemiology Lab at Boston Children’s Hospital — even though some vaccine experts say there’s only clear evidence that seniors need the shots. This type of policy “undermines confidence” in public health leaders, McLaren said.
At the same time, many Americans lack the information to decide whether they need — or can get — a booster shot. About four in 10 fully vaccinated adults said they were unsure whether they were eligible for a booster, according to an October poll from the Kaiser Family Foundation.
Misinformation and news overload also contribute to the confusion, said Dr. Lee Harrison, an epidemiologist at the University of Pittsburgh. “For a lot of laypeople, it’s very difficult to know, ‘What source should I be using?’ And it’s even more difficult when you have all this misinformation trying to intentionally misguide people,” he said. State and local officials who undermine national policies — for example, by prohibiting vaccine mandates — don’t help, either.
While breakthrough cases and booster shots have dominated the COVID-19 news cycle in recent months, vaccine effectiveness isn’t the only metric which the U.S. has struggled to record. The country is also failing to comprehensively track case demographics, long COVID-19 and coronavirus variants, to name a few metrics.
“There isn’t a simple fix,” McLaren said. “All of these health systems have their own electronic health records, and integrating them is really costly and hard to do.” Such integration also comes with privacy concerns, McLaren added, as data scientists need to protect patients’ anonymity.
But to start pulling those disparate data systems together, the CDC could provide state and local data collectors with standard reporting requirements — in other words, a list of data points that agencies need to provide, with consistent definitions and protocols. Federal officials have successfully done this before: In summer 2020, the Department of Health and Human Services built a brand-new system to collect COVID-19 data from every hospital in the country, every day. After initial hiccups, the resulting national dataset became highly reliable.
Meanwhile, health and science agencies could better communicate how they use data to make policy decisions. Harrison called the FDA and CDC’s decision-making process “extremely transparent,” and indeed, all advisory committee meetings are livestreamed for the public. But for someone without a medical degree, the meetings are hard to follow — full of packed slides, jargon and members asking each other to go off mute.
There is one other way to improve the U.S.’s public health data collection: follow Israel and the U.K. in implementing universal health care. Whatever you might think of the merits of universal health care, such a system would likely make it easier for public health leaders to standardize how Americans’ health records are kept. “Medicare for All would solve so many problems,” McLaren said.
CLARIFICATION (Nov. 9, 2021, 4:35 p.m.): An earlier version of this article stated that Zoë McLaren was a health economist at the University of Maryland. She is a health economist at the University of Maryland Baltimore County.