The speed at which the virus we now call SARS-CoV-2 spread throughout the world in the spring of 2020 proved more than our institutions could handle: Within a matter of weeks, more than 100 countries were reporting cases of COVID-19, the complex – and sometimes deadly – disease caused by the virus. So much was unknown in those first weeks: Why were there such a variety of symptoms – or sometimes no symptoms at all – among those who’d been infected? Why were there such vast differences in patient outcomes? Some people seemed to recover quickly, while others – some of whom were otherwise healthy, with no other diagnosed health conditions – succumbed to the disease.

As clinicians and researchers worked to gather information and devise strategies for detection, prevention, and treatment, a noisy and often contradictory data set emerged. Some early pieces of the puzzle didn’t seem to fit with others. It seemed clear that older people, and people with underlying health conditions such as diabetes, were more likely to have severe cases of the disease – but plenty of questions remained.

In these uncertain months, the capacity for testing was limited; the effectiveness of treatments was unknown; and a good protective vaccine was estimated to be anywhere from 12 months to 10 years away. Clinicians needed a way to understand their patients’ risks for the worst outcomes, independent of these unknowns.

Amy Justice, MD, PhD, a clinical epidemiologist, is the C.N.H. Long Professor of Medicine and Public Health at the Yale School of Medicine and a staff physician at the VA Connecticut Healthcare System. She has spent much of her career developing large national research cohorts based on data in the VA’s electronic health record (EHR), other national databases, patient surveys, and tissue repositories, and she knew a close look at VA patients could offer some insights into individual risks for COVID-19 mortality. Justice began to assemble a team of researchers to look at VA patient data and devise an index that could accurately predict whether a person, once infected with SARS-CoV-2, would die within 30 days.

“We wanted to come up with some things that could be used to look at people before they get COVID, and be able to say: This person is really at risk for a bad outcome,” said Justice. “Because early on, we were talking about having to decide who should get vaccines, or even who should get tested, because even testing was limited then. So we wanted to be able to target the folks who were at the greatest risk – and we wanted to use information that was generally available.”

Joseph King, MD, MSCE, chief of neurosurgery at the Connecticut VA and an associate professor at the Yale School of Medicine, became a valued team member soon after the effort was launched. He earned a master’s degree in clinical epidemiology and biostatistics while training as a neurosurgeon, and since the 1990s has been performing epidemiological research on big data sets; today he directs Yale’s neurosurgical outcomes research program.

STEP ONE: FINDING THE RIGHT DATA

The starting point for developing the index was a review of every VA patient who tested positive for SARS-CoV-2 between Feb. 8 and Aug. 18, 2020: a total of 13,323 individual records. The knowledge and expertise needed to analyze and make sense of a dataset this large is considerable – but at the VA, it was already available: a team of clinician researchers such as Justice and King, data scientists, programmers, and other analysis experts. The team needed to do more than simply pull the data out, said Justice. “You have to really know what’s behind that data; you need to understand what missing data means,” she said. “You need to understand how to clean the laboratory data – which is not trivial. You need to understand how to calculate what medications people are on. But we’ve been doing that for 25 years.”

The VA team split the sample into three temporal cohorts and looked for commonalities among the 946 people who died within 30 days of testing positive. As one might imagine, 946 people had a lot in common among them. Cutting through the noise and focusing on meaningful data, said King, required clinical experience: “We have experience treating patients and treating diseases, so we don’t just look at every single variable that’s available in the dataset. There may be spurious connections, or there may be things that just don’t make sense clinically.”

The team’s analysis focused exclusively on data available before the positive test: demographics such as age, race, and gender, as well as comorbid conditions. Some associations were clear: being older, or male, clearly increased one’s chance of dying of COVID-19.

Other correlations weren’t as immediately apparent. Looking at comorbidities individually – heart disease, for example, or diabetes – led the team to think a bigger-picture view of a patient’s disease burden would offer a better prediction of mortality. This work, to an extent, had already been done for them: In 1987, Mary Charlson, MD, an internist and now-professor of medicine at Weill Cornell Medical College, developed a weighted index of 17 specific conditions to predict a hospitalized patient’s risk of death within one year. The Charlson Comorbidity Index has since been used in literally thousands of medical research publications.

“She found that hypertension by itself – which is incredibly common; after the age of 65, more people have hypertension than don’t, right? – isn’t actually a risk factor, when you account for the other conditions,” Justice said. “Once people are 65, they tend to have more than one condition. They tend to have two or three or four conditions, and one of them might be hypertension – but hypertension isn’t what’s driving your risk for bad outcomes with COVID. So that was one of the things we thought was the most important thing to tease apart: To what extent was it particular diagnoses – everyone was talking about diabetes; you kept reading about the COVID risk of diabetes in the newspapers – and to what extent was that just burden of disease?”

The Charlson Index score turned out to be a better predictor of COVID mortality than a look at nearly all the individual comorbidities – except for heart attack or peripheral vascular disease, each of which sent a strong enough signal that they were scored separately by the VA team.

“So basically,” said King, “our index uses age, gender, and Charlson score, and then whether or not you have had a heart attack or peripheral vascular disease.” The team’s result – the Veterans Health Administration COVID-19 Index for COVID-19 Mortality, widely known as the VACO Index – was validated among the VA patient population and published in the journal PLOS One in November 2020.

A RACE-FREE INDEX

Anyone who has followed news media coverage of the pandemic may be surprised by at least two factors that are excluded from the VACO Index: race and body mass index (BMI). From the beginning of the pandemic, the public has been told, in a variety of ways, that a person’s risk of dying of COVID-19 is greater if they are non-white and/or if they are overweight. The VACO Index team’s work reveals that while this isn’t exactly wrong, it’s not exactly right in the way a clinician caring for an individual needs it to be right.

An explanation begins with clarifying what the index is for: It’s a screening tool that estimates an individual patient’s risk of all-cause mortality within 30 days after a COVID-19 infection. “All-cause” means simply that the patient died of something after being infected by the virus; whether COVID-19 could be specified as the cause of death, or even foremost among multiple contributing causes, clouds the purpose of the VACO Index: simply to determine a patient’s risk of dying (of anything) after a COVID-19 infection.

When peer-reviewed articles are titled for medical journals, their language is more precise than the headlines of news articles reporting on it. For example, a National Institutes of Health (NIH) study on racial disparities in COVID-19 outcomes was published in the journal Annals of Internal Medicine in October 2021, titled: “Racial and Ethnic Disparities in Excess Deaths During the COVID-19 Pandemic, March to December 2020.” The team’s findings – that there were “profound racial/ethnic disparities in excess deaths in the United States in 2020 during the COVID-19 pandemic” – were summarized in an article on the Healthline website titled: “Why Black, Native American, and Latino Communities Experience Higher COVID-19 Death Rates.”

The word “rates” is unintentionally misleading in this context; studies such as these have indicated that overall, non-white groups are at greater risk for dying of COVID. But importantly, the estimates are typically based on everyone in a specified racial or ethnic group, whether they’ve been infected with the virus or not.

It’s an important distinction, said Justice: “So many of the papers that have talked about race being an incredible risk factor were asking: OK, if I am Black, what’s the chance that I will die of COVID? Not: If I am Black, and I have COVID, what’s the chance that I’ll die with COVID? Most of those publications did not specify who had tested positive.” The VACO Index group, and other VA investigators, have made that distinction, and they’ve found, “if you account for who tests positive, the risk for mortality – given that you’re positive – is no different by race,” said Justice. “But the risk of testing positive was dramatically higher for people of minority status, especially Black and Hispanic patients. So if we are to intervene on that disparity, we need to keep people from getting the infection. It’s not that we need to manage them differently once they get the infection.”

The VACO team also found that obesity – a BMI of 30 or higher – wasn’t a strong risk factor for COVID-related mortality after accounting for other risk factors; it wasn’t until BMI reached about 35 that it became modestly significant.

VALIDATION AND ROLLOUT

One of the pressing questions, after the VACO Index had been developed, was whether it would be validated by other samples. VA patients are a diverse group overall, but they are not without some distinctions. For example, said King: “Within the VA, minority groups have better access to health care than they do in the general population … so the quality of care they receive, and their access to care, is better within the VA than it is for many people who receive care outside the VA.”

The big question, then, after the VACO Index was developed, was whether it would accurately predict short-term COVID-related mortality in other patient populations. The team ran data from two additional sources: more than 1,300 COVID-positive patients from Yale New Haven Hospital, and more than 425,000 Medicare patients. There were some obvious differences among these patients: The VA sample included both outpatients (some of whom took a drive-through test and never got very sick) and inpatients; the Yale hospital patients were all inpatients – in other words, sick enough to require hospital admission. Medicare patients are, by definition, at least 65 years of age, so their risk skews higher than a population with a wider age range, such as the VA patient population.

According to Justice, testing the index with these different patient populations was key to validating its effectiveness. “I’ve spent a lot of my life working on how you decide when a predictive index generalizes to the next patient who walks through your door – because you can only base it on what you’ve seen in other patients,” she said. “We looked at these three groups that were intentionally different from each other, and the index worked reasonably well in all three.”

After validation, the team looked for ways to help clinicians use the VACO Index for three specific purposes: to prioritize patients for primary vaccination (and now booster vaccinations); to motivate high-risk people and their contacts to practice social distancing until vaccinated; and to identify people testing positive at drive-up or other off-campus sites who should undergo clinical examination and possibly laboratory evaluation.

The VACO Index has been widely adopted by clinicians in its first year, after the team presented it to the Centers for Disease Control and Prevention (CDC), the Department of Defense, the Food and Drug Administration (FDA); the Centers for Medicare and Medicaid Services (CMS); Dr. Anthony Fauci, director of the National Institute of Allergy and Infectious Diseases and the chief medical adviser to the president; and several HMOs. With MDCalc, the online medical reference that provides web-based and mobile decision-support tools for health care professionals, Justice – who has worked with the company to develop other decision-support tools – was able to get a VACO Index calculator available to web and mobile users. More than twothirds of U.S. attending physicians regularly use MDCalc. “That made it available to VA and non-VA docs alike,” said Justice. MDCalc reported more than 31,000 users of the index worldwide, from Omaha to Ho Chi Minh City, by October 2021.

The rollout of the index has revealed another use for the VACO Index: It can inform larger-scale risk models for health insurers. CareJourney, an analytics company founded in 2014 to provide accountable care organizations (ACOs) with insights from health record data, used the index to both look back at examples – individual cases – that might have been managed differently, and look forward to managed care. “The output from this model,” said Aneesh Chopra, president of CareJourney, “has been incredibly useful for health plans looking to improve their risk models.”

The VACO Index will no doubt be refined over time, but because its sample was split into three validation cohorts (the first as the model was being developed; the second and third as testing became more widely available, treatments improved, and the Delta variant of SARS-CoV-2 emerged as a more transmissible and possibly more pathogenic virus), King and Justice expect it to hold up. “The disease is kind of a moving target,” said Justice, and the unfolding of new developments such as more testing and better treatments, “set the bar higher for us to show our index was, in fact, a solid index that could give good value – and it still did, even as we used this later group of people to validate it.”

This story originally appears in Veterans Affairs & Military Medicine OUTLOOK, Winter 2021 edition.