Margaret C Fang, MD, MPH

Within the nationwide effort to combat the opioid epidemic and reduce opioid prescribing, researchers have described different prescribing patterns for non-White racial and ethnic groups, including Black and LatinX populations. This remains a largely unexplored area within hospital medicine. Earlier studies of racial disparities demonstrate how some patients are assessed less often for pain and prescribed fewer opioids from the emergency department, surgical settings, and outpatient primary care practices. Researchers also have documented racial and ethnic disparities in analgesia for cancer pain and chronic noncancer pain.^1-11 Studies have demonstrated that White patients are more likely to receive opioid prescriptions compared with Black patients. Even with similar documented pain scores, there is evidence that Black patients receive fewer analgesics compared with White patients. For example, a recent study found that Black and Hispanic patients presenting to the emergency room for renal colic received less opioid medication compared with White patients.³ A study across 22 sites in Northern California found that racial minorities with long-bone fractures received fewer opioids at discharge than White patients.¹

It is unknown whether differential prescribing patterns by race exist among patients hospitalized on general medicine services. The objective of our study was to assess whether race and ethnicity were associated with the likelihood of opioids being prescribed and the duration of opioids prescribed when these patients are discharged from the hospital. Quantifying and seeking to understand these differences are the first steps toward ensuring racial and ethnic health equity in patient care.

Study Population and Data Sources

We identified all adults (age ≥18 years) discharged from the acute care inpatient general medicine services between June 1, 2012, and November 30, 2018, at the University of California, San Francisco (UCSF) Helen Diller Medical Center at Parnassus Heights, a 785-bed urban academic teaching hospital. All data were obtained from the hospital’s Epic-based electronic medical record (Epic Systems Corporation). Data elements were extracted from Clarity, the relationship database that stores Epic inpatient data. Patients discharged from the inpatient cardiology or bone marrow transplant services were not included. We excluded patients who did not receive opioids in the last 24 hours of their hospitalization. Patients with cancer-related pain diagnoses or sickle cell disease pain crises and patients who were discharged to hospice or followed by palliative care were excluded from the study based on International Classification of Diseases, Tenth Revision (ICD-10) codes (available on request) or service codes, when available, or admitting provider electronic health record documentation (Appendix Figure 1). Palliative care and hospice patients have significantly different pain needs, with management often directed by specialists. Patients with sickle cell disease are disproportionately Black and have distinct opioid prescribing patterns.^12,13 We also excluded discharge opioid prescriptions that were a resumption of the patient’s opioid prescription before admission based on medication documentation. Only new prescriptions signed by the discharging hospitalist, including different doses and formulations, were included in this study.

We performed a subgroup analysis of patients who were not prescribed opioids before their admission based on medication reconciliation but were started on opioids while hospitalized.

Primary Outcomes

We examined two primary outcomes: whether a patient received an opioid prescription at discharge, and, for patients prescribed opioids, the number of days prescribed. Days of opioids at discharge were calculated as total morphine milligram equivalents (MMEs) prescribed divided by MMEs administered during the final 24 hours of hospitalization. This metric was used as a patient-specific approach to calculating how long an opioid prescription will last after discharge, standardized according to the actual opioid requirements from hospitalization.¹⁴ If a patient was discharged with prescriptions for several opioids, the longest single prescription duration was used.

Primary Predictors

The primary predictor was the patient’s primary self-reported race/ethnicity, categorized as White, Black, LatinX, Asian, Native Hawaiian or other Pacific Islander, American Indian or Alaska Native, and other/unknown. Other/unknown included patients who were listed as other, declined, or who were otherwise unspecified. Self-reported race/ethnicity is obtained through reporting to the registrar. These race/ethnicity groupings were done in concordance with US Census Bureau definitions. Researchers classified patients as LatinX if they had Hispanic documented as their ethnicity, no matter their racial identification. These categorizations were chosen to be consistent with the existing literature, recognizing the role of a combined race/ethnicity definition for Hispanic or LatinX populations.¹⁵ These definitions of race/ethnicity are self-reported and reflect socially—not genetically defined—groupings.¹⁶ This variable serves as a surrogate for the structural factors that contribute to racism, the determining factor for racially disparate outcomes.¹⁷

Covariate Data Collection

Additional data were obtained regarding patient demographics, hospitalization factors, and medical diagnoses. Demographic factors included age, sex, and limited English proficiency (LEP) status. LEP was defined as having a primary language other than English and requiring an interpreter. Hospitalization factors included length of stay, whether they required intensive care unit (ICU) management, average daily MMEs administered during their entire hospitalization, MMEs administered during the final 24 hours of their hospitalization, whether the patient was on a teaching service or direct-care hospitalist service, their disposition on discharge, and year. Medical diagnosis variables included the adjusted Elixhauser Comorbidity Index based on ICD-10 codes; whether the patient was taking opioids at admission; and specific diagnoses of cancer, posttraumatic stress disorder (PTSD), and mood, anxiety, or psychotic disorder based on ICD-10 documentation.¹⁸

Statistical Analysis

All statistical analyses were performed using Stata software version 16 (StataCorp LP). Baseline demographic variables, hospitalization factors, and medical diagnosis variables were stratified by race/ethnicity. Within group comparisons were performed using chi-square or analysis of varianace (ANOVA) testing. For regression analyses, we fit two models. First, we fit a multivariable logistic regression model on all patients who received opioids during the last 24 hours of their hospitalization to examine the association between patient race/ethnicity and whether a patient received opioids at discharge, adjusting for additional patient, hospitalization, and medical covariates. Then we fit a negative binomial regression model on patients who were prescribed opioids at discharge to examine the association between patient race/ethnicity and the amount of opioids prescribed at discharge, adjusting for covariates. We used a negative binomial model because of the overdispersed distribution of discharge opioid prescriptions and only examined patients with an opioid prescription at discharge. We included the listed variables in our model because they were all found a priori to be associated with discharge opioid prescriptions.¹⁹ Instead of using days of opioids based on the last 24 hours, we performed a secondary analysis using the actual days of opioids supplied as the outcome. For example, a prescription of 12 tablets with every 6 hours dosing would be 3 days’ duration.

For both models, subgroup analyses were performed using the adjusted models restricted to patients newly prescribed opioids during their hospitalization and who were not previously taking opioids based on admission medication reconciliation. After testing for effect modification, this subgroup analysis was performed to reduce selection bias associated with earlier opioid use.

For all models, we reported predicted population opioid prescribing rates from the average marginal effects (AME).²⁰ Marginal effects were used because ours was a population level study and the outcome of interest was relatively common, limiting the effective interpretation of odds ratios.²¹ Marginal effects allow us to observe the instantaneous effect a given independent variable has on a dependent variable, while holding all other variables constant. It is implemented using the margins command in Stata. Marginal effects enable us to present our results as differences in probabilities, which is a more accurate way to describe the differences found among patient groups. Further, marginal effects are less sensitive to changes in model specifications.²²The UCSF Institutional Review Board for Human Subjects Research approved this study with a waiver of informed consent.

Unadjusted Results

We identified 10,953 patients who received opioids during the last 24 hours of hospitalization (see Appendix Figure 1 for study consort diagram). The patient population was 52.2% White, 18.4% Black, 11.5% Latinx, 10.1% Asian, 6.2% other/unknown, 0.9% Native Hawaiian/Other Pacific Islander, and 0.8% American Indian/Alaska Native (Table 1, Appendix Table 1). Black patients had fewer cancer diagnoses and the highest rate of prescribed opioids on admission. Asian patients were older and more likely to be female, and had higher rates of cancer, the highest median comorbidity index, and the smallest median daily MME during both the last 24 hours and total duration of hospitalization. Representative of general medicine patients, the most common principal discharge diagnoses in our dataset were pneumonia, cellulitis, altered mental status, sepsis, and abdominal pain.

Overall, 5541 (50.6%) patients who received opioids in the last 24 hours of their hospitalization received an opioid prescription at discharge. There were significant differences among racial/ethnic groups receiving an opioid prescription at discharge. Black patients were less likely to be discharged with an opioid compared with White patients (47.7% vs 50.3%; P < .001) (Table 2). The median discharge prescription duration for all patients was 9.3 days (interquartile range [IQR], 3.8-20.0). Black patients received the fewest median days of opioids at 7.5 days (IQR, 3.2-16.7) compared with White patients at 8.8 days (IQR, 3.7-20.0; P < .001) (Table 2).

Adjusted Regression Results

Demographic, clinical, and diagnosis specific factors were significantly associated with opioid prescriptions, including previous opioid use, sex, and a concurrent cancer diagnosis. There were fewer opioid prescriptions over time (Figure).

Following multivariable logistic regression for the association between race/ethnicity and opioid on discharge and controlling for covariates, we found that Black patients were less likely to receive an opioid prescription on discharge compared with White patients (predicted population rate, 47.6% vs 50.7%; AME −3.1%; 95% CI, −5.5% to −0.8%). Asian patients were more likely to receive a prescription on discharge compared with White patients (predicted population rate, 55.6% vs 50.7%; AME +4.9; 95% CI, 1.5%-8.3%).

Following multivariable negative binomial regression for the association between race/ethnicity and the number of opioid days on discharge, we found that Black patients received a shorter duration of opioid days compared with White patients (predicted days of opioids on discharge, 15.7 days vs 17.8 days; AME −2.1 days; 95% CI, −3.3 to −0.87) (Table 3). There were no significant differences among patients and the other racial/ethnic groups.

Our secondary analysis from the negative binomial regression with the days of opioids supplied metric yielded similar results to our primary analysis showing that Black patients received statically significantly fewer days of opioid therapy compared with White patients (Appendix Table 2).

Subgroup Regression Results

After testing for effect modification, which was negative, we examined the relationships for patients started on opioids during their hospitalization (Appendix Table 3 and Appendix Table 4). There were 5101 patients with newly prescribed opioids during their hospitalization. Adjusting for covariates, we found that Black patients were less likely to receive opioids at discharge compared with White patients (predicted population rate, 34.9% vs 40.4%; AME −5.5%; 95% CI, −9.2% to −1.9%). American Indian or Alaska Native patients were more likely to receive opioids on discharge (predicted population rate, 58.3% vs 40.4%; AME +17.9%; 95% CI, 1.0%-34.8%). We also found that Asian patients received more days of opioids on discharge (predicted days of opioid on discharge, 16.7 vs 13.7 days; AME +3.0 days; 95% CI, 0.6-5.3 days) (Appendix Table 4, Appendix Figure 2).

We found that Black patients discharged from the general medicine service were less likely to receive opioids and received shorter courses on discharge compared with White patients, adjusting for demographic, hospitalization, and medical diagnosis variables. Asian patients were more likely to receive an opioid prescription at discharge—a finding not reported in the literature on opioid prescribing disparities in most other practice settings.¹

Previous studies have shown racial disparities in pain management in emergency and surgical settings, but these relationships have not been characterized in an inpatient medicine population. Medicine patients comprise the majority of admitted patients in the United States and reflect a wide diversity of medical conditions, many requiring opioids for pain management. Determining the etiology of these differential prescribing patterns was not within the scope of our study, but earlier studies demonstrate a number of reasons why these patterns exist across racial and ethnic groups in other practice settings.^23,24 These reports give us insight into potential mechanisms for our study population.

Differences in pain management likely represent the multiple structural mechanisms by which racism operates.¹⁷ Drawing from the existing literature and the socioecological model, we hypothesize the ways that individual, interpersonal relationships, organizations, communities, and public policy impact opioid prescribing.^25,26 Using this model and considering the framework of Critical Race Theory (CRT), we can work towards understanding how race and ethnicity stand in as surrogates for racism and how this manifests in different outcomes and identify areas for intervention. CRT draws attention to race consciousness, contemporary orientation, centering in the margins, and praxis. In the context of this analysis, we recognize race consciousness and the interactions among factors such as race/ethnicity, language, and diagnoses such as PTSD.²⁷ This approach is necessary because racism is a multilevel construct influenced by macrolevel factors.²⁸

Individually and interpersonally, there is clinician-driven bias in pain assessment, which is activated under times of stress and diagnostic uncertainty and is amplified by a lack of clear guidelines for pain management prescriptions.^23,29-32 Institutional and organizational culture contribute to disparities through ingrained culture, practice patterns, and resource allocation.^29,33 Last, public policy and the larger sociopolitical environment worsen disparities through nondiverse workforces, state and federal guidelines, criminal justice policy, supply chain regulation, and access to care.

As individual clinicians, departments, and health systems leaders, we must identify areas for intervention. At the individual and interpersonal levels, there is evidence that taking implicit association tests could help clinicians become more aware of their negative associations, and empathy-inducing, perspective-taking interventions can reduce pain treatment bias.^31,34 At the institutional level, we must report data on disparities, create guidelines for pain management, and reevaluate the educational curriculum and culture to assess how certain biases could be propagated. The lack of straightforward guidelines leads to unclear indications for opioid prescriptions, exacerbating provider-level differences in prescribing. At the policy level, legislation that promotes workplace diversity, increases training for and access to pain specialists, and incentivizes data collection and reporting could help reduce disparities.³⁵ Equitable access to prescriptions and care is essential. Pharmacies often understock opioids in minority neighborhoods, meaning that even if a patient is prescribed an opioid on discharge, he or she might have difficulty filling the prescription.³⁶

One could question whether fewer opioid prescriptions for Black patients protects against the harms of opioid overprescribing, and therefore is not reflective of harmful inequity.³⁷ Ongoing national programs aim to reduce the harmful effects of opioids, which is reflected in the reduction in opioid prescribing over time in our institution. Our point is that differences in prescribing could reflect practices that do result in patient harm, such as less adequately controlled pain among Black patients.^1,3 Undertreated pain has negative health and social consequences and further contributes to substance-use stigma within minority communities.³⁸ Moreover, Black people who describe more discrimination in medical settings were more likely to report subsequent opioid misuse.³⁹

Although the above mechanisms might partially explain our findings among Black patients, the higher rate of prescribing for Asian patients is more challenging to explain. Our models adjusted for clinical factors. Notably, our Asian patients had the highest baseline comorbidity index, oldest mean age, and highest cancer rates, and it is possible that we were unable to fully account for illness severity or related pain needs (Table 1). It also is possible—although speculative—that factors such as language, provider concordance, and the type of disease process all contribute.⁴⁰ Some researchers have proposed a “stereotype content model” that seeks to establish a pathway among social structure (status of a patient) to clinician stereotypes (is this patient warm and/or competent) to emotional prejudices (envy, pride) and ultimately to discrimination (active/passive, help/harm).²³Our study has limitations. Our model was limited by the available data collected on our patients. Covariates including primary care follow-up, pain scores, and overdose history were not available. Furthermore, our categorization of race/ethnicity was based on self-reported data. We had 676 patients with race/ethnicity specified as other/unknown. We recognize the heterogeneity within these racial/ethnic categorizations. For example, within the LatinX or Asian communities, there are large differences based on region, country, ethnic, or cultural groups. Our study only included patients presenting to a hospital in San Francisco, which is different from the racial/ethnic makeup of other cities across the nation. Our electronic health record capture of history of opioid use disorder and mood disorders is contingent on individual clinician documentation. We did not account for provider-level differences, which is an important part of variation in prescribing differences. We also did not examine differences at the diagnosis-specific level. Finally, we could not determine the indication or appropriateness of opioid prescriptions.

Future studies will be necessary to characterize this relationship at a diagnosis-specific level and to describe causal pathways. Within our own institution, these findings present an opportunity for positive change. We hope to continue to explore the etiology of these disparities and identify areas where differences could impact patient outcomes, such as pain control. It is essential to develop appropriate recommendations for inpatient and discharge opioid prescribing to help minimize disparities and to mitigate potential harms of overprescribing. All health systems should continue to collect data on their own disparities in opioid prescribing and educate clinicians on promoting more equitable practices.

The authors thank Sneha Daya, MD, Sachin Shah, MD, MPH, and the UCSF Division of Hospital Medicine Data Core.

Within the nationwide effort to combat the opioid epidemic and reduce opioid prescribing, researchers have described different prescribing patterns for non-White racial and ethnic groups, including Black and LatinX populations. This remains a largely unexplored area within hospital medicine. Earlier studies of racial disparities demonstrate how some patients are assessed less often for pain and prescribed fewer opioids from the emergency department, surgical settings, and outpatient primary care practices. Researchers also have documented racial and ethnic disparities in analgesia for cancer pain and chronic noncancer pain.^1-11 Studies have demonstrated that White patients are more likely to receive opioid prescriptions compared with Black patients. Even with similar documented pain scores, there is evidence that Black patients receive fewer analgesics compared with White patients. For example, a recent study found that Black and Hispanic patients presenting to the emergency room for renal colic received less opioid medication compared with White patients.³ A study across 22 sites in Northern California found that racial minorities with long-bone fractures received fewer opioids at discharge than White patients.¹

It is unknown whether differential prescribing patterns by race exist among patients hospitalized on general medicine services. The objective of our study was to assess whether race and ethnicity were associated with the likelihood of opioids being prescribed and the duration of opioids prescribed when these patients are discharged from the hospital. Quantifying and seeking to understand these differences are the first steps toward ensuring racial and ethnic health equity in patient care.

Study Population and Data Sources

We identified all adults (age ≥18 years) discharged from the acute care inpatient general medicine services between June 1, 2012, and November 30, 2018, at the University of California, San Francisco (UCSF) Helen Diller Medical Center at Parnassus Heights, a 785-bed urban academic teaching hospital. All data were obtained from the hospital’s Epic-based electronic medical record (Epic Systems Corporation). Data elements were extracted from Clarity, the relationship database that stores Epic inpatient data. Patients discharged from the inpatient cardiology or bone marrow transplant services were not included. We excluded patients who did not receive opioids in the last 24 hours of their hospitalization. Patients with cancer-related pain diagnoses or sickle cell disease pain crises and patients who were discharged to hospice or followed by palliative care were excluded from the study based on International Classification of Diseases, Tenth Revision (ICD-10) codes (available on request) or service codes, when available, or admitting provider electronic health record documentation (Appendix Figure 1). Palliative care and hospice patients have significantly different pain needs, with management often directed by specialists. Patients with sickle cell disease are disproportionately Black and have distinct opioid prescribing patterns.^12,13 We also excluded discharge opioid prescriptions that were a resumption of the patient’s opioid prescription before admission based on medication documentation. Only new prescriptions signed by the discharging hospitalist, including different doses and formulations, were included in this study.

We performed a subgroup analysis of patients who were not prescribed opioids before their admission based on medication reconciliation but were started on opioids while hospitalized.

Primary Outcomes

We examined two primary outcomes: whether a patient received an opioid prescription at discharge, and, for patients prescribed opioids, the number of days prescribed. Days of opioids at discharge were calculated as total morphine milligram equivalents (MMEs) prescribed divided by MMEs administered during the final 24 hours of hospitalization. This metric was used as a patient-specific approach to calculating how long an opioid prescription will last after discharge, standardized according to the actual opioid requirements from hospitalization.¹⁴ If a patient was discharged with prescriptions for several opioids, the longest single prescription duration was used.

Primary Predictors

The primary predictor was the patient’s primary self-reported race/ethnicity, categorized as White, Black, LatinX, Asian, Native Hawaiian or other Pacific Islander, American Indian or Alaska Native, and other/unknown. Other/unknown included patients who were listed as other, declined, or who were otherwise unspecified. Self-reported race/ethnicity is obtained through reporting to the registrar. These race/ethnicity groupings were done in concordance with US Census Bureau definitions. Researchers classified patients as LatinX if they had Hispanic documented as their ethnicity, no matter their racial identification. These categorizations were chosen to be consistent with the existing literature, recognizing the role of a combined race/ethnicity definition for Hispanic or LatinX populations.¹⁵ These definitions of race/ethnicity are self-reported and reflect socially—not genetically defined—groupings.¹⁶ This variable serves as a surrogate for the structural factors that contribute to racism, the determining factor for racially disparate outcomes.¹⁷

Covariate Data Collection

Additional data were obtained regarding patient demographics, hospitalization factors, and medical diagnoses. Demographic factors included age, sex, and limited English proficiency (LEP) status. LEP was defined as having a primary language other than English and requiring an interpreter. Hospitalization factors included length of stay, whether they required intensive care unit (ICU) management, average daily MMEs administered during their entire hospitalization, MMEs administered during the final 24 hours of their hospitalization, whether the patient was on a teaching service or direct-care hospitalist service, their disposition on discharge, and year. Medical diagnosis variables included the adjusted Elixhauser Comorbidity Index based on ICD-10 codes; whether the patient was taking opioids at admission; and specific diagnoses of cancer, posttraumatic stress disorder (PTSD), and mood, anxiety, or psychotic disorder based on ICD-10 documentation.¹⁸

Statistical Analysis

All statistical analyses were performed using Stata software version 16 (StataCorp LP). Baseline demographic variables, hospitalization factors, and medical diagnosis variables were stratified by race/ethnicity. Within group comparisons were performed using chi-square or analysis of varianace (ANOVA) testing. For regression analyses, we fit two models. First, we fit a multivariable logistic regression model on all patients who received opioids during the last 24 hours of their hospitalization to examine the association between patient race/ethnicity and whether a patient received opioids at discharge, adjusting for additional patient, hospitalization, and medical covariates. Then we fit a negative binomial regression model on patients who were prescribed opioids at discharge to examine the association between patient race/ethnicity and the amount of opioids prescribed at discharge, adjusting for covariates. We used a negative binomial model because of the overdispersed distribution of discharge opioid prescriptions and only examined patients with an opioid prescription at discharge. We included the listed variables in our model because they were all found a priori to be associated with discharge opioid prescriptions.¹⁹ Instead of using days of opioids based on the last 24 hours, we performed a secondary analysis using the actual days of opioids supplied as the outcome. For example, a prescription of 12 tablets with every 6 hours dosing would be 3 days’ duration.

For both models, subgroup analyses were performed using the adjusted models restricted to patients newly prescribed opioids during their hospitalization and who were not previously taking opioids based on admission medication reconciliation. After testing for effect modification, this subgroup analysis was performed to reduce selection bias associated with earlier opioid use.

For all models, we reported predicted population opioid prescribing rates from the average marginal effects (AME).²⁰ Marginal effects were used because ours was a population level study and the outcome of interest was relatively common, limiting the effective interpretation of odds ratios.²¹ Marginal effects allow us to observe the instantaneous effect a given independent variable has on a dependent variable, while holding all other variables constant. It is implemented using the margins command in Stata. Marginal effects enable us to present our results as differences in probabilities, which is a more accurate way to describe the differences found among patient groups. Further, marginal effects are less sensitive to changes in model specifications.²²The UCSF Institutional Review Board for Human Subjects Research approved this study with a waiver of informed consent.

Unadjusted Results

We identified 10,953 patients who received opioids during the last 24 hours of hospitalization (see Appendix Figure 1 for study consort diagram). The patient population was 52.2% White, 18.4% Black, 11.5% Latinx, 10.1% Asian, 6.2% other/unknown, 0.9% Native Hawaiian/Other Pacific Islander, and 0.8% American Indian/Alaska Native (Table 1, Appendix Table 1). Black patients had fewer cancer diagnoses and the highest rate of prescribed opioids on admission. Asian patients were older and more likely to be female, and had higher rates of cancer, the highest median comorbidity index, and the smallest median daily MME during both the last 24 hours and total duration of hospitalization. Representative of general medicine patients, the most common principal discharge diagnoses in our dataset were pneumonia, cellulitis, altered mental status, sepsis, and abdominal pain.

Overall, 5541 (50.6%) patients who received opioids in the last 24 hours of their hospitalization received an opioid prescription at discharge. There were significant differences among racial/ethnic groups receiving an opioid prescription at discharge. Black patients were less likely to be discharged with an opioid compared with White patients (47.7% vs 50.3%; P < .001) (Table 2). The median discharge prescription duration for all patients was 9.3 days (interquartile range [IQR], 3.8-20.0). Black patients received the fewest median days of opioids at 7.5 days (IQR, 3.2-16.7) compared with White patients at 8.8 days (IQR, 3.7-20.0; P < .001) (Table 2).

Adjusted Regression Results

Demographic, clinical, and diagnosis specific factors were significantly associated with opioid prescriptions, including previous opioid use, sex, and a concurrent cancer diagnosis. There were fewer opioid prescriptions over time (Figure).

Following multivariable logistic regression for the association between race/ethnicity and opioid on discharge and controlling for covariates, we found that Black patients were less likely to receive an opioid prescription on discharge compared with White patients (predicted population rate, 47.6% vs 50.7%; AME −3.1%; 95% CI, −5.5% to −0.8%). Asian patients were more likely to receive a prescription on discharge compared with White patients (predicted population rate, 55.6% vs 50.7%; AME +4.9; 95% CI, 1.5%-8.3%).

Following multivariable negative binomial regression for the association between race/ethnicity and the number of opioid days on discharge, we found that Black patients received a shorter duration of opioid days compared with White patients (predicted days of opioids on discharge, 15.7 days vs 17.8 days; AME −2.1 days; 95% CI, −3.3 to −0.87) (Table 3). There were no significant differences among patients and the other racial/ethnic groups.

Our secondary analysis from the negative binomial regression with the days of opioids supplied metric yielded similar results to our primary analysis showing that Black patients received statically significantly fewer days of opioid therapy compared with White patients (Appendix Table 2).

Subgroup Regression Results

After testing for effect modification, which was negative, we examined the relationships for patients started on opioids during their hospitalization (Appendix Table 3 and Appendix Table 4). There were 5101 patients with newly prescribed opioids during their hospitalization. Adjusting for covariates, we found that Black patients were less likely to receive opioids at discharge compared with White patients (predicted population rate, 34.9% vs 40.4%; AME −5.5%; 95% CI, −9.2% to −1.9%). American Indian or Alaska Native patients were more likely to receive opioids on discharge (predicted population rate, 58.3% vs 40.4%; AME +17.9%; 95% CI, 1.0%-34.8%). We also found that Asian patients received more days of opioids on discharge (predicted days of opioid on discharge, 16.7 vs 13.7 days; AME +3.0 days; 95% CI, 0.6-5.3 days) (Appendix Table 4, Appendix Figure 2).

We found that Black patients discharged from the general medicine service were less likely to receive opioids and received shorter courses on discharge compared with White patients, adjusting for demographic, hospitalization, and medical diagnosis variables. Asian patients were more likely to receive an opioid prescription at discharge—a finding not reported in the literature on opioid prescribing disparities in most other practice settings.¹

Previous studies have shown racial disparities in pain management in emergency and surgical settings, but these relationships have not been characterized in an inpatient medicine population. Medicine patients comprise the majority of admitted patients in the United States and reflect a wide diversity of medical conditions, many requiring opioids for pain management. Determining the etiology of these differential prescribing patterns was not within the scope of our study, but earlier studies demonstrate a number of reasons why these patterns exist across racial and ethnic groups in other practice settings.^23,24 These reports give us insight into potential mechanisms for our study population.

Differences in pain management likely represent the multiple structural mechanisms by which racism operates.¹⁷ Drawing from the existing literature and the socioecological model, we hypothesize the ways that individual, interpersonal relationships, organizations, communities, and public policy impact opioid prescribing.^25,26 Using this model and considering the framework of Critical Race Theory (CRT), we can work towards understanding how race and ethnicity stand in as surrogates for racism and how this manifests in different outcomes and identify areas for intervention. CRT draws attention to race consciousness, contemporary orientation, centering in the margins, and praxis. In the context of this analysis, we recognize race consciousness and the interactions among factors such as race/ethnicity, language, and diagnoses such as PTSD.²⁷ This approach is necessary because racism is a multilevel construct influenced by macrolevel factors.²⁸

Individually and interpersonally, there is clinician-driven bias in pain assessment, which is activated under times of stress and diagnostic uncertainty and is amplified by a lack of clear guidelines for pain management prescriptions.^23,29-32 Institutional and organizational culture contribute to disparities through ingrained culture, practice patterns, and resource allocation.^29,33 Last, public policy and the larger sociopolitical environment worsen disparities through nondiverse workforces, state and federal guidelines, criminal justice policy, supply chain regulation, and access to care.

As individual clinicians, departments, and health systems leaders, we must identify areas for intervention. At the individual and interpersonal levels, there is evidence that taking implicit association tests could help clinicians become more aware of their negative associations, and empathy-inducing, perspective-taking interventions can reduce pain treatment bias.^31,34 At the institutional level, we must report data on disparities, create guidelines for pain management, and reevaluate the educational curriculum and culture to assess how certain biases could be propagated. The lack of straightforward guidelines leads to unclear indications for opioid prescriptions, exacerbating provider-level differences in prescribing. At the policy level, legislation that promotes workplace diversity, increases training for and access to pain specialists, and incentivizes data collection and reporting could help reduce disparities.³⁵ Equitable access to prescriptions and care is essential. Pharmacies often understock opioids in minority neighborhoods, meaning that even if a patient is prescribed an opioid on discharge, he or she might have difficulty filling the prescription.³⁶

One could question whether fewer opioid prescriptions for Black patients protects against the harms of opioid overprescribing, and therefore is not reflective of harmful inequity.³⁷ Ongoing national programs aim to reduce the harmful effects of opioids, which is reflected in the reduction in opioid prescribing over time in our institution. Our point is that differences in prescribing could reflect practices that do result in patient harm, such as less adequately controlled pain among Black patients.^1,3 Undertreated pain has negative health and social consequences and further contributes to substance-use stigma within minority communities.³⁸ Moreover, Black people who describe more discrimination in medical settings were more likely to report subsequent opioid misuse.³⁹

Although the above mechanisms might partially explain our findings among Black patients, the higher rate of prescribing for Asian patients is more challenging to explain. Our models adjusted for clinical factors. Notably, our Asian patients had the highest baseline comorbidity index, oldest mean age, and highest cancer rates, and it is possible that we were unable to fully account for illness severity or related pain needs (Table 1). It also is possible—although speculative—that factors such as language, provider concordance, and the type of disease process all contribute.⁴⁰ Some researchers have proposed a “stereotype content model” that seeks to establish a pathway among social structure (status of a patient) to clinician stereotypes (is this patient warm and/or competent) to emotional prejudices (envy, pride) and ultimately to discrimination (active/passive, help/harm).²³Our study has limitations. Our model was limited by the available data collected on our patients. Covariates including primary care follow-up, pain scores, and overdose history were not available. Furthermore, our categorization of race/ethnicity was based on self-reported data. We had 676 patients with race/ethnicity specified as other/unknown. We recognize the heterogeneity within these racial/ethnic categorizations. For example, within the LatinX or Asian communities, there are large differences based on region, country, ethnic, or cultural groups. Our study only included patients presenting to a hospital in San Francisco, which is different from the racial/ethnic makeup of other cities across the nation. Our electronic health record capture of history of opioid use disorder and mood disorders is contingent on individual clinician documentation. We did not account for provider-level differences, which is an important part of variation in prescribing differences. We also did not examine differences at the diagnosis-specific level. Finally, we could not determine the indication or appropriateness of opioid prescriptions.

Future studies will be necessary to characterize this relationship at a diagnosis-specific level and to describe causal pathways. Within our own institution, these findings present an opportunity for positive change. We hope to continue to explore the etiology of these disparities and identify areas where differences could impact patient outcomes, such as pain control. It is essential to develop appropriate recommendations for inpatient and discharge opioid prescribing to help minimize disparities and to mitigate potential harms of overprescribing. All health systems should continue to collect data on their own disparities in opioid prescribing and educate clinicians on promoting more equitable practices.

The authors thank Sneha Daya, MD, Sachin Shah, MD, MPH, and the UCSF Division of Hospital Medicine Data Core.

Within the nationwide effort to combat the opioid epidemic and reduce opioid prescribing, researchers have described different prescribing patterns for non-White racial and ethnic groups, including Black and LatinX populations. This remains a largely unexplored area within hospital medicine. Earlier studies of racial disparities demonstrate how some patients are assessed less often for pain and prescribed fewer opioids from the emergency department, surgical settings, and outpatient primary care practices. Researchers also have documented racial and ethnic disparities in analgesia for cancer pain and chronic noncancer pain.^1-11 Studies have demonstrated that White patients are more likely to receive opioid prescriptions compared with Black patients. Even with similar documented pain scores, there is evidence that Black patients receive fewer analgesics compared with White patients. For example, a recent study found that Black and Hispanic patients presenting to the emergency room for renal colic received less opioid medication compared with White patients.³ A study across 22 sites in Northern California found that racial minorities with long-bone fractures received fewer opioids at discharge than White patients.¹

It is unknown whether differential prescribing patterns by race exist among patients hospitalized on general medicine services. The objective of our study was to assess whether race and ethnicity were associated with the likelihood of opioids being prescribed and the duration of opioids prescribed when these patients are discharged from the hospital. Quantifying and seeking to understand these differences are the first steps toward ensuring racial and ethnic health equity in patient care.

Study Population and Data Sources

We identified all adults (age ≥18 years) discharged from the acute care inpatient general medicine services between June 1, 2012, and November 30, 2018, at the University of California, San Francisco (UCSF) Helen Diller Medical Center at Parnassus Heights, a 785-bed urban academic teaching hospital. All data were obtained from the hospital’s Epic-based electronic medical record (Epic Systems Corporation). Data elements were extracted from Clarity, the relationship database that stores Epic inpatient data. Patients discharged from the inpatient cardiology or bone marrow transplant services were not included. We excluded patients who did not receive opioids in the last 24 hours of their hospitalization. Patients with cancer-related pain diagnoses or sickle cell disease pain crises and patients who were discharged to hospice or followed by palliative care were excluded from the study based on International Classification of Diseases, Tenth Revision (ICD-10) codes (available on request) or service codes, when available, or admitting provider electronic health record documentation (Appendix Figure 1). Palliative care and hospice patients have significantly different pain needs, with management often directed by specialists. Patients with sickle cell disease are disproportionately Black and have distinct opioid prescribing patterns.^12,13 We also excluded discharge opioid prescriptions that were a resumption of the patient’s opioid prescription before admission based on medication documentation. Only new prescriptions signed by the discharging hospitalist, including different doses and formulations, were included in this study.

We performed a subgroup analysis of patients who were not prescribed opioids before their admission based on medication reconciliation but were started on opioids while hospitalized.

Primary Outcomes

We examined two primary outcomes: whether a patient received an opioid prescription at discharge, and, for patients prescribed opioids, the number of days prescribed. Days of opioids at discharge were calculated as total morphine milligram equivalents (MMEs) prescribed divided by MMEs administered during the final 24 hours of hospitalization. This metric was used as a patient-specific approach to calculating how long an opioid prescription will last after discharge, standardized according to the actual opioid requirements from hospitalization.¹⁴ If a patient was discharged with prescriptions for several opioids, the longest single prescription duration was used.

Primary Predictors

The primary predictor was the patient’s primary self-reported race/ethnicity, categorized as White, Black, LatinX, Asian, Native Hawaiian or other Pacific Islander, American Indian or Alaska Native, and other/unknown. Other/unknown included patients who were listed as other, declined, or who were otherwise unspecified. Self-reported race/ethnicity is obtained through reporting to the registrar. These race/ethnicity groupings were done in concordance with US Census Bureau definitions. Researchers classified patients as LatinX if they had Hispanic documented as their ethnicity, no matter their racial identification. These categorizations were chosen to be consistent with the existing literature, recognizing the role of a combined race/ethnicity definition for Hispanic or LatinX populations.¹⁵ These definitions of race/ethnicity are self-reported and reflect socially—not genetically defined—groupings.¹⁶ This variable serves as a surrogate for the structural factors that contribute to racism, the determining factor for racially disparate outcomes.¹⁷

Covariate Data Collection

Additional data were obtained regarding patient demographics, hospitalization factors, and medical diagnoses. Demographic factors included age, sex, and limited English proficiency (LEP) status. LEP was defined as having a primary language other than English and requiring an interpreter. Hospitalization factors included length of stay, whether they required intensive care unit (ICU) management, average daily MMEs administered during their entire hospitalization, MMEs administered during the final 24 hours of their hospitalization, whether the patient was on a teaching service or direct-care hospitalist service, their disposition on discharge, and year. Medical diagnosis variables included the adjusted Elixhauser Comorbidity Index based on ICD-10 codes; whether the patient was taking opioids at admission; and specific diagnoses of cancer, posttraumatic stress disorder (PTSD), and mood, anxiety, or psychotic disorder based on ICD-10 documentation.¹⁸

Statistical Analysis

All statistical analyses were performed using Stata software version 16 (StataCorp LP). Baseline demographic variables, hospitalization factors, and medical diagnosis variables were stratified by race/ethnicity. Within group comparisons were performed using chi-square or analysis of varianace (ANOVA) testing. For regression analyses, we fit two models. First, we fit a multivariable logistic regression model on all patients who received opioids during the last 24 hours of their hospitalization to examine the association between patient race/ethnicity and whether a patient received opioids at discharge, adjusting for additional patient, hospitalization, and medical covariates. Then we fit a negative binomial regression model on patients who were prescribed opioids at discharge to examine the association between patient race/ethnicity and the amount of opioids prescribed at discharge, adjusting for covariates. We used a negative binomial model because of the overdispersed distribution of discharge opioid prescriptions and only examined patients with an opioid prescription at discharge. We included the listed variables in our model because they were all found a priori to be associated with discharge opioid prescriptions.¹⁹ Instead of using days of opioids based on the last 24 hours, we performed a secondary analysis using the actual days of opioids supplied as the outcome. For example, a prescription of 12 tablets with every 6 hours dosing would be 3 days’ duration.

For both models, subgroup analyses were performed using the adjusted models restricted to patients newly prescribed opioids during their hospitalization and who were not previously taking opioids based on admission medication reconciliation. After testing for effect modification, this subgroup analysis was performed to reduce selection bias associated with earlier opioid use.

For all models, we reported predicted population opioid prescribing rates from the average marginal effects (AME).²⁰ Marginal effects were used because ours was a population level study and the outcome of interest was relatively common, limiting the effective interpretation of odds ratios.²¹ Marginal effects allow us to observe the instantaneous effect a given independent variable has on a dependent variable, while holding all other variables constant. It is implemented using the margins command in Stata. Marginal effects enable us to present our results as differences in probabilities, which is a more accurate way to describe the differences found among patient groups. Further, marginal effects are less sensitive to changes in model specifications.²²The UCSF Institutional Review Board for Human Subjects Research approved this study with a waiver of informed consent.

Unadjusted Results

We identified 10,953 patients who received opioids during the last 24 hours of hospitalization (see Appendix Figure 1 for study consort diagram). The patient population was 52.2% White, 18.4% Black, 11.5% Latinx, 10.1% Asian, 6.2% other/unknown, 0.9% Native Hawaiian/Other Pacific Islander, and 0.8% American Indian/Alaska Native (Table 1, Appendix Table 1). Black patients had fewer cancer diagnoses and the highest rate of prescribed opioids on admission. Asian patients were older and more likely to be female, and had higher rates of cancer, the highest median comorbidity index, and the smallest median daily MME during both the last 24 hours and total duration of hospitalization. Representative of general medicine patients, the most common principal discharge diagnoses in our dataset were pneumonia, cellulitis, altered mental status, sepsis, and abdominal pain.

Overall, 5541 (50.6%) patients who received opioids in the last 24 hours of their hospitalization received an opioid prescription at discharge. There were significant differences among racial/ethnic groups receiving an opioid prescription at discharge. Black patients were less likely to be discharged with an opioid compared with White patients (47.7% vs 50.3%; P < .001) (Table 2). The median discharge prescription duration for all patients was 9.3 days (interquartile range [IQR], 3.8-20.0). Black patients received the fewest median days of opioids at 7.5 days (IQR, 3.2-16.7) compared with White patients at 8.8 days (IQR, 3.7-20.0; P < .001) (Table 2).

Adjusted Regression Results

Demographic, clinical, and diagnosis specific factors were significantly associated with opioid prescriptions, including previous opioid use, sex, and a concurrent cancer diagnosis. There were fewer opioid prescriptions over time (Figure).

Following multivariable logistic regression for the association between race/ethnicity and opioid on discharge and controlling for covariates, we found that Black patients were less likely to receive an opioid prescription on discharge compared with White patients (predicted population rate, 47.6% vs 50.7%; AME −3.1%; 95% CI, −5.5% to −0.8%). Asian patients were more likely to receive a prescription on discharge compared with White patients (predicted population rate, 55.6% vs 50.7%; AME +4.9; 95% CI, 1.5%-8.3%).

Following multivariable negative binomial regression for the association between race/ethnicity and the number of opioid days on discharge, we found that Black patients received a shorter duration of opioid days compared with White patients (predicted days of opioids on discharge, 15.7 days vs 17.8 days; AME −2.1 days; 95% CI, −3.3 to −0.87) (Table 3). There were no significant differences among patients and the other racial/ethnic groups.

Our secondary analysis from the negative binomial regression with the days of opioids supplied metric yielded similar results to our primary analysis showing that Black patients received statically significantly fewer days of opioid therapy compared with White patients (Appendix Table 2).

Subgroup Regression Results

After testing for effect modification, which was negative, we examined the relationships for patients started on opioids during their hospitalization (Appendix Table 3 and Appendix Table 4). There were 5101 patients with newly prescribed opioids during their hospitalization. Adjusting for covariates, we found that Black patients were less likely to receive opioids at discharge compared with White patients (predicted population rate, 34.9% vs 40.4%; AME −5.5%; 95% CI, −9.2% to −1.9%). American Indian or Alaska Native patients were more likely to receive opioids on discharge (predicted population rate, 58.3% vs 40.4%; AME +17.9%; 95% CI, 1.0%-34.8%). We also found that Asian patients received more days of opioids on discharge (predicted days of opioid on discharge, 16.7 vs 13.7 days; AME +3.0 days; 95% CI, 0.6-5.3 days) (Appendix Table 4, Appendix Figure 2).

We found that Black patients discharged from the general medicine service were less likely to receive opioids and received shorter courses on discharge compared with White patients, adjusting for demographic, hospitalization, and medical diagnosis variables. Asian patients were more likely to receive an opioid prescription at discharge—a finding not reported in the literature on opioid prescribing disparities in most other practice settings.¹

Previous studies have shown racial disparities in pain management in emergency and surgical settings, but these relationships have not been characterized in an inpatient medicine population. Medicine patients comprise the majority of admitted patients in the United States and reflect a wide diversity of medical conditions, many requiring opioids for pain management. Determining the etiology of these differential prescribing patterns was not within the scope of our study, but earlier studies demonstrate a number of reasons why these patterns exist across racial and ethnic groups in other practice settings.^23,24 These reports give us insight into potential mechanisms for our study population.

Differences in pain management likely represent the multiple structural mechanisms by which racism operates.¹⁷ Drawing from the existing literature and the socioecological model, we hypothesize the ways that individual, interpersonal relationships, organizations, communities, and public policy impact opioid prescribing.^25,26 Using this model and considering the framework of Critical Race Theory (CRT), we can work towards understanding how race and ethnicity stand in as surrogates for racism and how this manifests in different outcomes and identify areas for intervention. CRT draws attention to race consciousness, contemporary orientation, centering in the margins, and praxis. In the context of this analysis, we recognize race consciousness and the interactions among factors such as race/ethnicity, language, and diagnoses such as PTSD.²⁷ This approach is necessary because racism is a multilevel construct influenced by macrolevel factors.²⁸

Individually and interpersonally, there is clinician-driven bias in pain assessment, which is activated under times of stress and diagnostic uncertainty and is amplified by a lack of clear guidelines for pain management prescriptions.^23,29-32 Institutional and organizational culture contribute to disparities through ingrained culture, practice patterns, and resource allocation.^29,33 Last, public policy and the larger sociopolitical environment worsen disparities through nondiverse workforces, state and federal guidelines, criminal justice policy, supply chain regulation, and access to care.

As individual clinicians, departments, and health systems leaders, we must identify areas for intervention. At the individual and interpersonal levels, there is evidence that taking implicit association tests could help clinicians become more aware of their negative associations, and empathy-inducing, perspective-taking interventions can reduce pain treatment bias.^31,34 At the institutional level, we must report data on disparities, create guidelines for pain management, and reevaluate the educational curriculum and culture to assess how certain biases could be propagated. The lack of straightforward guidelines leads to unclear indications for opioid prescriptions, exacerbating provider-level differences in prescribing. At the policy level, legislation that promotes workplace diversity, increases training for and access to pain specialists, and incentivizes data collection and reporting could help reduce disparities.³⁵ Equitable access to prescriptions and care is essential. Pharmacies often understock opioids in minority neighborhoods, meaning that even if a patient is prescribed an opioid on discharge, he or she might have difficulty filling the prescription.³⁶

One could question whether fewer opioid prescriptions for Black patients protects against the harms of opioid overprescribing, and therefore is not reflective of harmful inequity.³⁷ Ongoing national programs aim to reduce the harmful effects of opioids, which is reflected in the reduction in opioid prescribing over time in our institution. Our point is that differences in prescribing could reflect practices that do result in patient harm, such as less adequately controlled pain among Black patients.^1,3 Undertreated pain has negative health and social consequences and further contributes to substance-use stigma within minority communities.³⁸ Moreover, Black people who describe more discrimination in medical settings were more likely to report subsequent opioid misuse.³⁹

Although the above mechanisms might partially explain our findings among Black patients, the higher rate of prescribing for Asian patients is more challenging to explain. Our models adjusted for clinical factors. Notably, our Asian patients had the highest baseline comorbidity index, oldest mean age, and highest cancer rates, and it is possible that we were unable to fully account for illness severity or related pain needs (Table 1). It also is possible—although speculative—that factors such as language, provider concordance, and the type of disease process all contribute.⁴⁰ Some researchers have proposed a “stereotype content model” that seeks to establish a pathway among social structure (status of a patient) to clinician stereotypes (is this patient warm and/or competent) to emotional prejudices (envy, pride) and ultimately to discrimination (active/passive, help/harm).²³Our study has limitations. Our model was limited by the available data collected on our patients. Covariates including primary care follow-up, pain scores, and overdose history were not available. Furthermore, our categorization of race/ethnicity was based on self-reported data. We had 676 patients with race/ethnicity specified as other/unknown. We recognize the heterogeneity within these racial/ethnic categorizations. For example, within the LatinX or Asian communities, there are large differences based on region, country, ethnic, or cultural groups. Our study only included patients presenting to a hospital in San Francisco, which is different from the racial/ethnic makeup of other cities across the nation. Our electronic health record capture of history of opioid use disorder and mood disorders is contingent on individual clinician documentation. We did not account for provider-level differences, which is an important part of variation in prescribing differences. We also did not examine differences at the diagnosis-specific level. Finally, we could not determine the indication or appropriateness of opioid prescriptions.

Future studies will be necessary to characterize this relationship at a diagnosis-specific level and to describe causal pathways. Within our own institution, these findings present an opportunity for positive change. We hope to continue to explore the etiology of these disparities and identify areas where differences could impact patient outcomes, such as pain control. It is essential to develop appropriate recommendations for inpatient and discharge opioid prescribing to help minimize disparities and to mitigate potential harms of overprescribing. All health systems should continue to collect data on their own disparities in opioid prescribing and educate clinicians on promoting more equitable practices.

The authors thank Sneha Daya, MD, Sachin Shah, MD, MPH, and the UCSF Division of Hospital Medicine Data Core.

Sepsis is the leading cause of in-hospital mortality in the United States.¹ Sepsis is present on admission in 85% of cases, and each hour delay in antibiotic treatment is associated with 4% to 7% increased odds of mortality.^2,3 Prompt identification and treatment of sepsis is essential for reducing morbidity and mortality, but identifying sepsis during triage is challenging.²

Risk stratification scores that rely solely on data readily available at the bedside have been developed to quickly identify those at greatest risk of poor outcomes from sepsis in real time. The quick Sequential Organ Failure Assessment (qSOFA) score, the National Early Warning System (NEWS2), and the Shock Index are easy-to-calculate measures that use routinely collected clinical data that are not subject to laboratory delay. These scores can be incorporated into electronic health record (EHR)-based alerts and can be calculated longitudinally to track the risk of poor outcomes over time. qSOFA was developed to quantify patient risk at bedside in non-intensive care unit (ICU) settings, but there is no consensus about its ability to predict adverse outcomes such as mortality and ICU admission.^4-6 The United Kingdom’s National Health Service uses NEWS2 to identify patients at risk for sepsis.⁷ NEWS has been shown to have similar or better sensitivity in identifying poorer outcomes in sepsis patients compared with systemic inflammatory response syndrome (SIRS) criteria and qSOFA.^4,8-11 However, since the latest update of NEWS2 in 2017, there has been little study of its predictive ability. The Shock Index is a simple bedside score (heart rate divided by systolic blood pressure) that was developed to detect changes in cardiovascular performance before systemic shock onset. Although it was not developed for infection and has not been regularly applied in the sepsis literature, the Shock Index might be useful for identifying patients at increased risk of poor outcomes. Patients with higher and sustained Shock Index scores are more likely to experience morbidity, such as hyperlactatemia, vasopressor use, and organ failure, and also have an increased risk of mortality.^12-14

Although the predictive abilities of these bedside risk stratification scores have been assessed individually using standard binary cut-points, the comparative performance of qSOFA, the Shock Index, and NEWS2 has not been evaluated in patients presenting to an emergency department (ED) with suspected sepsis. Our objective was to provide a head-to-head comparison of the test characteristics of qSOFA, the Shock Index, and NEWS2 calculated at ED triage for predicting in-hospital mortality and ED-to-ICU admission in patients with suspected sepsis to help health systems and providers select screening measures.

Design and Setting

We conducted a retrospective cohort study of ED patients who presented with suspected sepsis to the University of California San Francisco (UCSF) Helen Diller Medical Center at Parnassus Heights between June 1, 2012, and December 31, 2018. Our institution is a 785-bed academic teaching hospital with approximately 30,000 ED encounters per year. The study was approved with a waiver of informed consent by the UCSF Human Research Protection Program.

Participants

We use an Epic-based EHR platform (Epic 2017, Epic Systems Corporation) for clinical care, which was implemented on June 1, 2012. All data elements were obtained from Clarity, the relational database that stores Epic’s inpatient data. The study included encounters for patients age ≥18 years who had blood cultures ordered within 24 hours of ED presentation and administration of intravenous antibiotics within 24 hours. Repeat encounters were treated independently in our analysis.

Outcomes and Measures

We compared the ability of qSOFA, the Shock Index, and NEWS2 to predict in-hospital mortality and admission to the ICU from the ED (ED-to-ICU admission). We used the most abnormal vital signs and clinical assessments gathered within 30 minutes of ED presentation to identify patients who were qSOFA-positive, Shock Index-positive, and NEWS2-positive based on standard cut-points of risk. Data elements used to calculate qSOFA, Shock Index, and NEWS2 included blood pressure, heart rate, respiratory rate, Glasgow Coma Scale (GCS) score, oxygen saturation, requirement for supplemental oxygen, and temperature (Table 1). Patients were considered positive if they had a qSOFAscore ≥2, Shock Index of >0.7, and NEWS2 ≥5 based on triage vital signs.^7,15,16 We considered patients to have altered mental status, a criterion used for NEWS2, if they had a GCS score <15 instead of using the “alert, verbal, confusion, pain, unresponsive” scale, which is not captured in our EHR, a method that has been used in earlier studies.^17,18 Missing assessments were considered normal. Although our primary analysis focused on the scores calculated within 30 minutes of ED presentation, we performed a sensitivity analysis examining scores calculated within 1 hour of ED presentation in the event of a delay in gathering triage vital sign data.

We compared demographic and clinical characteristics of patients who were positive for qSOFA, the Shock Index, and NEWS2. Demographic data were extracted from the EHR and included primary language, age, sex, and insurance status. All International Classification of Diseases (ICD)-9/10 diagnosis codes were pulled from Clarity billing tables. We used the Elixhauser comorbidity groupings¹⁹ of ICD-9/10 codes present on admission to identify preexisting comorbidities and underlying organ dysfunction. To estimate burden of comorbid illnesses, we calculated the validated van Walraven comorbidity index,²⁰ which provides an estimated risk of in-hospital death based on documented Elixhauser comorbidities. Admission level of care (acute, stepdown, or intensive care) was collected for inpatient admissions to assess initial illness severity.²¹ We also evaluated discharge disposition and in-hospital mortality. Index blood culture results were collected, and dates and timestamps of mechanical ventilation, fluid, vasopressor, and antibiotic administration were obtained for the duration of the encounter.

UCSF uses an automated, real-time, algorithm-based severe sepsis alert that is triggered when a patient meets ≥2 SIRS criteria and again when the patient meets severe sepsis or septic shock criteria (ie, ≥2 SIRS criteria in addition to end-organ dysfunction and/or fluid nonresponsive hypotension). This sepsis screening alert was in use for the duration of our study.²²

Statistical Analysis

We performed a subgroup analysis among those who were diagnosed with sepsis, according to the 2016 Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) criteria. Sepsis is defined as a change in Sequential Organ Failure Assessment (SOFA) score of ≥2 points within the first 48 hours.²³ Additionally, patients meeting Sepsis-3 criteria needed to (1) receive ≥4 days of sequential antibiotic therapy or experience death or discharge to hospice before 4 days of antibiotic therapy or (2) have a validated sepsis discharge billing code. These parameters were added to increase the specificity of our sample.²⁴

All statistical analyses were conducted using Stata 14 (StataCorp). We summarized differences in demographic and clinical characteristics among the populations meeting each severity score but elected not to conduct hypothesis testing because patients could be positive for one or more scores. We calculated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for each score to predict in-hospital mortality and ED-to-ICU admission. To allow comparison with other studies, we also created a composite outcome of either in-hospital mortality or ED-to-ICU admission. To assess score discrimination to predict in-hospital mortality and ED-to-ICU admission, we calculated the area under the receiver operating characteristic curve (AUROC) along with asymptotic normal 95% CI using the “roctab” command considering a binary cut-point, as well as the full range of scores measured in the cohort. The AUROC range from 0.50 to 1.00 and a score in the 0.70 to 0.80 range can be considered fair.²⁵ We assessed significant differences between severity score AUROCs using the DeLong method²⁶ implemented through Stata 14’s “roccomp” command. As a sensitivity analysis, we explored whether the standard cut-points for qSOFA, the Shock Index, and NEWS2 provided the highest AUROC in our population by calculating test characteristics for several score cut-points.

Within our sample 23,837 ED patients had blood cultures ordered within 24 hours of ED presentation and were considered to have suspected sepsis. The mean age of the cohort was 60.8 years, and 1,612 (6.8%) had positive blood cultures. A total of 12,928 patients (54.2%) were found to have sepsis. We documented 1,427 in-hospital deaths (6.0%) and 3,149 (13.2%) ED-to-ICU admissions. At ED triage 1,921 (8.1%) were qSOFA-positive, 4,273 (17.9%) were Shock Index-positive, and 11,832 (49.6%) were NEWS2-positive. At ED triage, blood pressure, heart rate, respiratory rate, and oxygen saturated were documented in >99% of patients, 93.5% had temperature documented, and 28.5% had GCS recorded. If the window of assessment was widened to 1 hour, GCS was only documented among 44.2% of those with suspected sepsis.

Demographic Characteristics and Clinical Course

We identified significant differences when comparing demographic and clinical characteristics among patients who scored positive for the three severity measures at triage (Table 2). Although no hypothesis testing was conducted because patients could meet one or more scores, qSOFA-positive patients were older (median 70, 66, and 64 years, respectively), more likely to have Medicare as the primary payor (67.6% vs 59.7% vs 56.6%), to have chronic renal failure (26.1%, 23.1%, and 23.3%, respectively), to have a greater degree of underlying comorbidities based on the van Walraven Comorbidity Index (median 15, 12, and 11, respectively), and to be admitted to the ICU from the ED (48.1%, 36.3%, and 21.0%, respectively) compared with those positive for the Shock Index or NEWS2.

qSOFA-positive patients received antibiotics more quickly than those who were Shock Index-positive or NEWS2-positive (median 1.5, 1.8, and 2.8 hours after admission, respectively). In addition, those who were qSOFA-positive were more likely to have a positive blood culture (10.9%, 9.4%, and 8.5%, respectively) and to receive an EHR-based diagnosis of sepsis (77.0%, 69.6%, and 60.9%, respectively) than those who were Shock Index- or NEWS2-positive. Those who were qSOFA-positive also were more likely to be mechanically ventilated during their hospital stay (25.4%, 19.2%, and 10.8%, respectively) and to receive vasopressors (33.5%, 22.5%, and 12.2%, respectively). In-hospital mortality also was more common among those who were qSOFA-positive at triage (23.4%, 15.3%, and 9.2%, respectively).

Because both qSOFA and NEWS2 incorporate GCS, we explored baseline characteristics of patients with GCS documented at triage (n = 6,794). These patients were older (median age 63 and 61 years, P < .0001), more likely to be male (54.9% and 53.4%, P = .0031), more likely to have renal failure (22.8% and 20.1%, P < .0001), more likely to have liver disease (14.2% and 12.8%, P = .006), had a higher van Walraven comorbidity score on presentation (median 10 and 8, P < .0001), and were more likely to go directly to the ICU from the ED (20.2% and 10.6%, P < .0001). However, among the 6,397 GCS scores documented at triage, only 1,579 (24.7%) were abnormal.

Test Characteristics of qSOFA, Shock Index, and NEWS2 for Predicting In-hospital Mortality and ED-to-ICU Admission

Among 23,837 patients with suspected sepsis, NEWS2 had the highest sensitivity for predicting in-hospital mortality (76.0%; 95% CI, 73.7%-78.2%) and ED-to-ICU admission (78.9%; 95% CI, 77.5%-80.4%) but had the lowest specificity for in-hospital mortality (52.0%; 95% CI, 51.4%-52.7%) and for ED-to-ICU admission (54.8%; 95% CI, 54.1%-55.5%) (Table 3). qSOFA had the lowest sensitivity for in-hospital mortality (31.5%; 95% CI, 29.1%-33.9%) and ED-to-ICU admission (29.3%; 95% CI, 27.7%-30.9%) but the highest specificity for in-hospital mortality (93.4%; 95% CI, 93.1%-93.8%) and ED-to-ICU admission (95.2%; 95% CI, 94.9%-95.5%). The Shock Index had a sensitivity that fell between qSOFA and NEWS2 for in-hospital mortality (45.8%; 95% CI, 43.2%-48.5%) and ED-to-ICU admission (49.2%; 95% CI, 47.5%-51.0%). The specificity of the Shock Index also was between qSOFA and NEWS2 for in-hospital mortality (83.9%; 95% CI, 83.4%-84.3%) and ED-to-ICU admission (86.8%; 95% CI, 86.4%-87.3%). All three scores exhibited relatively low PPV, ranging from 9.2% to 23.4% for in-hospital mortality and 21.0% to 48.0% for ED-to-ICU triage. Conversely, all three scores exhibited relatively high NPV, ranging from 95.5% to 97.1% for in-hospital mortality and 89.8% to 94.5% for ED-to-ICU triage. The patterns in sensitivity and specificity for in-hospital mortality and ED-to-ICU admission were similar among the 12,928 patients who received an EHR-based sepsis diagnosis with the tests generally demonstrating lower specificities, higher PPVs, and lower NPVs (Table 3).

When considering a binary cutoff, the Shock Index exhibited the highest AUROC for in-hospital mortality (0.648; 95% CI, 0.635-0.662) and had a significantly higher AUROC than qSOFA (AUROC, 0.625; 95% CI, 0.612-0.637; P = .0005), but there was no difference compared with NEWS2 (AUROC, 0.640; 95% CI, 0.628-0.652; P = .2112). NEWS2 had a significantly higher AUROC than qSOFA for predicting in-hospital mortality (P = .0227). The Shock Index also exhibited the highest AUROC for ED-to-ICU admission (0.680; 95% CI, 0.617-0.689), which was significantly higher than the AUROC for qSOFA (P < .0001) and NEWS2 (P = 0.0151). NEWS2 had a significantly higher AUROC than qSOFA for predicting ED-to-ICU admission (P < .0001). Similar findings were seen in patients found to have sepsis. When considering the range of possible scores measured in our cohort, qSOFA and NEWS2 exhibited higher AUROCs for in-hospital mortality and ED-to-ICU admission than the Shock Index among patients with suspected infection and the subgroup with a sepsis diagnosis (Figure).

In this retrospective cohort study of 23,837 patients who presented to the ED with suspected sepsis, the standard qSOFA threshold was met least frequently, followed by the Shock Index and NEWS2. NEWS2 had the highest sensitivity but the lowest specificity for predicting in-hospital mortality and ED-to-ICU admission, making it a challenging bedside risk stratification scale for identifying patients at risk of poor clinical outcomes. When comparing predictive performance among the three scales, qSOFA had the highest specificity and the Shock Index had the highest AUROC for in-hospital mortality and ED-to-ICU admission in this cohort of patients with suspected sepsis. These trends in sensitivity, specificity, and AUROC were consistent among those who met EHR criteria for a sepsis diagnosis. In the analysis of the three scoring systems using all available cut-points, qSOFA and NEWS2 had the highest AUROCs, followed by the Shock Index.

Considering the rapid progression from organ dysfunction to death in sepsis patients, as well as the difficulty establishing a sepsis diagnosis at triage,²³ providers must quickly identify patients at increased risk of poor outcomes when they present to the ED. Sepsis alerts often are built using SIRS criteria,²⁷ including the one used for sepsis surveillance at UCSF since 2012,²² but the white blood cell count criterion is subject to a laboratory lag and could lead to a delay in identification. Implementation of a point-of-care bedside score alert that uses readily available clinical data could allow providers to identify patients at greatest risk of poor outcomes immediately at ED presentation and triage, which motivated us to explore the predictive performance of qSOFA, the Shock Index, and NEWS2.

Our study is the first to provide a head-to-head comparison of the predictive performance of qSOFA, the Shock Index, and NEWS2, three easy-to-calculate bedside risk scores that use EHR data collected among patients with suspected sepsis. The Sepsis-3 guidelines recommend qSOFA to quickly identify non-ICU patients at greatest risk of poor outcomes because the measure exhibited predictive performance similar to the more extensive SOFA score outside the ICU.^16,23 Although some studies have confirmed qSOFA’s high predictive performance,^28-31 our test characteristics and AUROC findings are in line with other published analyses.^4,6,10,17 The UK National Health Service is using NEWS2 to screen for patients at risk of poor outcomes from sepsis. Several analyses that assessed the predictive ability of NEWS have reported estimates in line with our findings.^4,10,32 The Shock Index was introduced in 1967 and provided a metric to evaluate hemodynamic stability based on heart rate and systolic blood pressure.³³ The Shock Index has been studied in several contexts, including sepsis,³⁴ and studies show that a sustained Shock Index is associated with increased odds of vasopressor administration, higher prevalence of hyperlactatemia, and increased risk of poor outcomes in the ICU.^13,14

For our study, we were particularly interested in exploring how the Shock Index would compare with more frequently used severity scores such as qSOFA and NEWS2 among patients with suspected sepsis, given the simplicity of its calculation and the easy availability of required data. In our cohort of 23,837 patients, only 159 people had missing blood pressure and only 71 had omitted heart rate. In contrast, both qSOFA and NEWS2 include an assessment of level of consciousness that can be subject to variability in assessment methods and EHR documentation across institutions.¹¹ In our cohort, GCS within 30 minutes of ED presentation was missing in 72 patients, which could have led to incomplete calculation of qSOFA and NEWS2 if a missing value was not actually within normal limits.

Several investigations relate qSOFA to NEWS but few compare qSOFA with the newer NEWS2, and even fewer evaluate the Shock Index with any of these scores.^{10,11,18,29,35-37} In general, studies have shown that NEWS exhibits a higher AUROC for predicting mortality, sepsis with organ dysfunction, and ICU admission, often as a composite outcome.^{4,11,18,37,38} A handful of studies compare the Shock Index to SIRS; however, little has been done to compare the Shock Index to qSOFA or NEWS2, scores that have been used specifically for sepsis and might be more predictive of poor outcomes than SIRS.³³ In our study, the Shock Index had a higher AUROC than either qSOFA or NEWS2 for predicting in-hospital mortality and ED-to-ICU admission measured as separate outcomes and as a composite outcome using standard cut-points for these scores.

When selecting a severity score to apply in an institution, it is important to carefully evaluate the score’s test characteristics, in addition to considering the availability of reliable data. Tests with high sensitivity and NPV for the population being studied can be useful to rule out disease or risk of poor outcome, while tests with high specificity and PPV can be useful to rule in disease or risk of poor outcome.³⁹ When considering specificity, qSOFA’s performance was superior to the Shock Index and NEWS2 in our study, but a small percentage of the population was identified using a cut-point of qSOFA ≥2. If we used qSOFA and applied this standard cut-point at our institution, we could be confident that those identified were at increased risk, but we would miss a significant number of patients who would experience a poor outcome. When considering sensitivity, performance of NEWS2 was superior to qSOFA and the Shock Index in our study, but one-half of the population was identified using a cut-point of NEWS2 ≥5. If we were to apply this standard NEWS2 cut-point at our institution, we would assume that one-half of our population was at risk, which might drive resource use towards patients who will not experience a poor outcome. Although none of the scores exhibited a robust AUROC measure, the Shock Index had the highest AUROC for in-hospital mortality and ED-to-ICU admission when using the standard binary cut-point, and its sensitivity and specificity is between that of qSOFA and NEWS2, potentially making it a score to use in settings where qSOFA and NEWS2 score components, such as altered mentation, are not reliably collected. Finally, our sensitivity analysis varying the binary cut-point of each score within our population demonstrated that the standard cut-points might not be as useful within a specific population and might need to be tailored for implementation, balancing sensitivity, specificity, PPV, and NPV to meet local priorities and ICU capacity.

Our study has limitations. It is a single-center, retrospective analysis, factors that could reduce generalizability. However, it does include a large and diverse patient population spanning several years. Missing GCS data could have affected the predictive ability of qSOFA and NEWS2 in our cohort. We could not reliably perform imputation of GCS because of the high missingness and therefore we assumed missing was normal, as was done in the Sepsis-3 derivation studies.¹⁶ Previous studies have attempted to impute GCS and have not observed improved performance of qSOFA to predict mortality.⁴⁰ Because manually collected variables such as GCS are less reliably documented in the EHR, there might be limitations in their use for triage risk scores.

Although the current analysis focused on the predictive performance of qSOFA, the Shock Index, and NEWS2 at triage, performance of these scores could affect the ED team’s treatment decisions before handoff to the hospitalist team and the expected level of care the patient will receive after in-patient admission. These tests also have the advantage of being easy to calculate at the bedside over time, which could provide an objective assessment of longitudinal predicted prognosis. Future work should assess the longitudinal performance of each of these scores among those with suspected sepsis and to determine the impact using these scores would have on clinical and resource utilization outcomes.

Local priorities should drive selection of a screening tool, balancing sensitivity, specificity, PPV, and NPV to achieve the institution’s goals. qSOFA, Shock Index, and NEWS2 are risk stratification tools that can be easily implemented at ED triage using data available at the bedside. Although none of these scores performed strongly when comparing AUROCs, qSOFA was highly specific for identifying patients with poor outcomes, and NEWS2 was the most sensitive for ruling out those at high risk among patients with suspected sepsis. The Shock Index exhibited a sensitivity and specificity that fell between qSOFA and NEWS2 and also might be considered to identify those at increased risk, given its ease of implementation, particularly in settings where altered mentation is unreliably or inconsistently documented.

The authors thank the UCSF Division of Hospital Medicine Data Core for their assistance with data acquisition.

Sepsis is the leading cause of in-hospital mortality in the United States.¹ Sepsis is present on admission in 85% of cases, and each hour delay in antibiotic treatment is associated with 4% to 7% increased odds of mortality.^2,3 Prompt identification and treatment of sepsis is essential for reducing morbidity and mortality, but identifying sepsis during triage is challenging.²

Risk stratification scores that rely solely on data readily available at the bedside have been developed to quickly identify those at greatest risk of poor outcomes from sepsis in real time. The quick Sequential Organ Failure Assessment (qSOFA) score, the National Early Warning System (NEWS2), and the Shock Index are easy-to-calculate measures that use routinely collected clinical data that are not subject to laboratory delay. These scores can be incorporated into electronic health record (EHR)-based alerts and can be calculated longitudinally to track the risk of poor outcomes over time. qSOFA was developed to quantify patient risk at bedside in non-intensive care unit (ICU) settings, but there is no consensus about its ability to predict adverse outcomes such as mortality and ICU admission.^4-6 The United Kingdom’s National Health Service uses NEWS2 to identify patients at risk for sepsis.⁷ NEWS has been shown to have similar or better sensitivity in identifying poorer outcomes in sepsis patients compared with systemic inflammatory response syndrome (SIRS) criteria and qSOFA.^4,8-11 However, since the latest update of NEWS2 in 2017, there has been little study of its predictive ability. The Shock Index is a simple bedside score (heart rate divided by systolic blood pressure) that was developed to detect changes in cardiovascular performance before systemic shock onset. Although it was not developed for infection and has not been regularly applied in the sepsis literature, the Shock Index might be useful for identifying patients at increased risk of poor outcomes. Patients with higher and sustained Shock Index scores are more likely to experience morbidity, such as hyperlactatemia, vasopressor use, and organ failure, and also have an increased risk of mortality.^12-14

Although the predictive abilities of these bedside risk stratification scores have been assessed individually using standard binary cut-points, the comparative performance of qSOFA, the Shock Index, and NEWS2 has not been evaluated in patients presenting to an emergency department (ED) with suspected sepsis. Our objective was to provide a head-to-head comparison of the test characteristics of qSOFA, the Shock Index, and NEWS2 calculated at ED triage for predicting in-hospital mortality and ED-to-ICU admission in patients with suspected sepsis to help health systems and providers select screening measures.

Design and Setting

We conducted a retrospective cohort study of ED patients who presented with suspected sepsis to the University of California San Francisco (UCSF) Helen Diller Medical Center at Parnassus Heights between June 1, 2012, and December 31, 2018. Our institution is a 785-bed academic teaching hospital with approximately 30,000 ED encounters per year. The study was approved with a waiver of informed consent by the UCSF Human Research Protection Program.

Participants

We use an Epic-based EHR platform (Epic 2017, Epic Systems Corporation) for clinical care, which was implemented on June 1, 2012. All data elements were obtained from Clarity, the relational database that stores Epic’s inpatient data. The study included encounters for patients age ≥18 years who had blood cultures ordered within 24 hours of ED presentation and administration of intravenous antibiotics within 24 hours. Repeat encounters were treated independently in our analysis.

Outcomes and Measures

We compared the ability of qSOFA, the Shock Index, and NEWS2 to predict in-hospital mortality and admission to the ICU from the ED (ED-to-ICU admission). We used the most abnormal vital signs and clinical assessments gathered within 30 minutes of ED presentation to identify patients who were qSOFA-positive, Shock Index-positive, and NEWS2-positive based on standard cut-points of risk. Data elements used to calculate qSOFA, Shock Index, and NEWS2 included blood pressure, heart rate, respiratory rate, Glasgow Coma Scale (GCS) score, oxygen saturation, requirement for supplemental oxygen, and temperature (Table 1). Patients were considered positive if they had a qSOFAscore ≥2, Shock Index of >0.7, and NEWS2 ≥5 based on triage vital signs.^7,15,16 We considered patients to have altered mental status, a criterion used for NEWS2, if they had a GCS score <15 instead of using the “alert, verbal, confusion, pain, unresponsive” scale, which is not captured in our EHR, a method that has been used in earlier studies.^17,18 Missing assessments were considered normal. Although our primary analysis focused on the scores calculated within 30 minutes of ED presentation, we performed a sensitivity analysis examining scores calculated within 1 hour of ED presentation in the event of a delay in gathering triage vital sign data.

We compared demographic and clinical characteristics of patients who were positive for qSOFA, the Shock Index, and NEWS2. Demographic data were extracted from the EHR and included primary language, age, sex, and insurance status. All International Classification of Diseases (ICD)-9/10 diagnosis codes were pulled from Clarity billing tables. We used the Elixhauser comorbidity groupings¹⁹ of ICD-9/10 codes present on admission to identify preexisting comorbidities and underlying organ dysfunction. To estimate burden of comorbid illnesses, we calculated the validated van Walraven comorbidity index,²⁰ which provides an estimated risk of in-hospital death based on documented Elixhauser comorbidities. Admission level of care (acute, stepdown, or intensive care) was collected for inpatient admissions to assess initial illness severity.²¹ We also evaluated discharge disposition and in-hospital mortality. Index blood culture results were collected, and dates and timestamps of mechanical ventilation, fluid, vasopressor, and antibiotic administration were obtained for the duration of the encounter.

UCSF uses an automated, real-time, algorithm-based severe sepsis alert that is triggered when a patient meets ≥2 SIRS criteria and again when the patient meets severe sepsis or septic shock criteria (ie, ≥2 SIRS criteria in addition to end-organ dysfunction and/or fluid nonresponsive hypotension). This sepsis screening alert was in use for the duration of our study.²²

Statistical Analysis

We performed a subgroup analysis among those who were diagnosed with sepsis, according to the 2016 Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) criteria. Sepsis is defined as a change in Sequential Organ Failure Assessment (SOFA) score of ≥2 points within the first 48 hours.²³ Additionally, patients meeting Sepsis-3 criteria needed to (1) receive ≥4 days of sequential antibiotic therapy or experience death or discharge to hospice before 4 days of antibiotic therapy or (2) have a validated sepsis discharge billing code. These parameters were added to increase the specificity of our sample.²⁴

All statistical analyses were conducted using Stata 14 (StataCorp). We summarized differences in demographic and clinical characteristics among the populations meeting each severity score but elected not to conduct hypothesis testing because patients could be positive for one or more scores. We calculated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for each score to predict in-hospital mortality and ED-to-ICU admission. To allow comparison with other studies, we also created a composite outcome of either in-hospital mortality or ED-to-ICU admission. To assess score discrimination to predict in-hospital mortality and ED-to-ICU admission, we calculated the area under the receiver operating characteristic curve (AUROC) along with asymptotic normal 95% CI using the “roctab” command considering a binary cut-point, as well as the full range of scores measured in the cohort. The AUROC range from 0.50 to 1.00 and a score in the 0.70 to 0.80 range can be considered fair.²⁵ We assessed significant differences between severity score AUROCs using the DeLong method²⁶ implemented through Stata 14’s “roccomp” command. As a sensitivity analysis, we explored whether the standard cut-points for qSOFA, the Shock Index, and NEWS2 provided the highest AUROC in our population by calculating test characteristics for several score cut-points.

Within our sample 23,837 ED patients had blood cultures ordered within 24 hours of ED presentation and were considered to have suspected sepsis. The mean age of the cohort was 60.8 years, and 1,612 (6.8%) had positive blood cultures. A total of 12,928 patients (54.2%) were found to have sepsis. We documented 1,427 in-hospital deaths (6.0%) and 3,149 (13.2%) ED-to-ICU admissions. At ED triage 1,921 (8.1%) were qSOFA-positive, 4,273 (17.9%) were Shock Index-positive, and 11,832 (49.6%) were NEWS2-positive. At ED triage, blood pressure, heart rate, respiratory rate, and oxygen saturated were documented in >99% of patients, 93.5% had temperature documented, and 28.5% had GCS recorded. If the window of assessment was widened to 1 hour, GCS was only documented among 44.2% of those with suspected sepsis.

Demographic Characteristics and Clinical Course

We identified significant differences when comparing demographic and clinical characteristics among patients who scored positive for the three severity measures at triage (Table 2). Although no hypothesis testing was conducted because patients could meet one or more scores, qSOFA-positive patients were older (median 70, 66, and 64 years, respectively), more likely to have Medicare as the primary payor (67.6% vs 59.7% vs 56.6%), to have chronic renal failure (26.1%, 23.1%, and 23.3%, respectively), to have a greater degree of underlying comorbidities based on the van Walraven Comorbidity Index (median 15, 12, and 11, respectively), and to be admitted to the ICU from the ED (48.1%, 36.3%, and 21.0%, respectively) compared with those positive for the Shock Index or NEWS2.

qSOFA-positive patients received antibiotics more quickly than those who were Shock Index-positive or NEWS2-positive (median 1.5, 1.8, and 2.8 hours after admission, respectively). In addition, those who were qSOFA-positive were more likely to have a positive blood culture (10.9%, 9.4%, and 8.5%, respectively) and to receive an EHR-based diagnosis of sepsis (77.0%, 69.6%, and 60.9%, respectively) than those who were Shock Index- or NEWS2-positive. Those who were qSOFA-positive also were more likely to be mechanically ventilated during their hospital stay (25.4%, 19.2%, and 10.8%, respectively) and to receive vasopressors (33.5%, 22.5%, and 12.2%, respectively). In-hospital mortality also was more common among those who were qSOFA-positive at triage (23.4%, 15.3%, and 9.2%, respectively).

Because both qSOFA and NEWS2 incorporate GCS, we explored baseline characteristics of patients with GCS documented at triage (n = 6,794). These patients were older (median age 63 and 61 years, P < .0001), more likely to be male (54.9% and 53.4%, P = .0031), more likely to have renal failure (22.8% and 20.1%, P < .0001), more likely to have liver disease (14.2% and 12.8%, P = .006), had a higher van Walraven comorbidity score on presentation (median 10 and 8, P < .0001), and were more likely to go directly to the ICU from the ED (20.2% and 10.6%, P < .0001). However, among the 6,397 GCS scores documented at triage, only 1,579 (24.7%) were abnormal.

Test Characteristics of qSOFA, Shock Index, and NEWS2 for Predicting In-hospital Mortality and ED-to-ICU Admission

Among 23,837 patients with suspected sepsis, NEWS2 had the highest sensitivity for predicting in-hospital mortality (76.0%; 95% CI, 73.7%-78.2%) and ED-to-ICU admission (78.9%; 95% CI, 77.5%-80.4%) but had the lowest specificity for in-hospital mortality (52.0%; 95% CI, 51.4%-52.7%) and for ED-to-ICU admission (54.8%; 95% CI, 54.1%-55.5%) (Table 3). qSOFA had the lowest sensitivity for in-hospital mortality (31.5%; 95% CI, 29.1%-33.9%) and ED-to-ICU admission (29.3%; 95% CI, 27.7%-30.9%) but the highest specificity for in-hospital mortality (93.4%; 95% CI, 93.1%-93.8%) and ED-to-ICU admission (95.2%; 95% CI, 94.9%-95.5%). The Shock Index had a sensitivity that fell between qSOFA and NEWS2 for in-hospital mortality (45.8%; 95% CI, 43.2%-48.5%) and ED-to-ICU admission (49.2%; 95% CI, 47.5%-51.0%). The specificity of the Shock Index also was between qSOFA and NEWS2 for in-hospital mortality (83.9%; 95% CI, 83.4%-84.3%) and ED-to-ICU admission (86.8%; 95% CI, 86.4%-87.3%). All three scores exhibited relatively low PPV, ranging from 9.2% to 23.4% for in-hospital mortality and 21.0% to 48.0% for ED-to-ICU triage. Conversely, all three scores exhibited relatively high NPV, ranging from 95.5% to 97.1% for in-hospital mortality and 89.8% to 94.5% for ED-to-ICU triage. The patterns in sensitivity and specificity for in-hospital mortality and ED-to-ICU admission were similar among the 12,928 patients who received an EHR-based sepsis diagnosis with the tests generally demonstrating lower specificities, higher PPVs, and lower NPVs (Table 3).

When considering a binary cutoff, the Shock Index exhibited the highest AUROC for in-hospital mortality (0.648; 95% CI, 0.635-0.662) and had a significantly higher AUROC than qSOFA (AUROC, 0.625; 95% CI, 0.612-0.637; P = .0005), but there was no difference compared with NEWS2 (AUROC, 0.640; 95% CI, 0.628-0.652; P = .2112). NEWS2 had a significantly higher AUROC than qSOFA for predicting in-hospital mortality (P = .0227). The Shock Index also exhibited the highest AUROC for ED-to-ICU admission (0.680; 95% CI, 0.617-0.689), which was significantly higher than the AUROC for qSOFA (P < .0001) and NEWS2 (P = 0.0151). NEWS2 had a significantly higher AUROC than qSOFA for predicting ED-to-ICU admission (P < .0001). Similar findings were seen in patients found to have sepsis. When considering the range of possible scores measured in our cohort, qSOFA and NEWS2 exhibited higher AUROCs for in-hospital mortality and ED-to-ICU admission than the Shock Index among patients with suspected infection and the subgroup with a sepsis diagnosis (Figure).

In this retrospective cohort study of 23,837 patients who presented to the ED with suspected sepsis, the standard qSOFA threshold was met least frequently, followed by the Shock Index and NEWS2. NEWS2 had the highest sensitivity but the lowest specificity for predicting in-hospital mortality and ED-to-ICU admission, making it a challenging bedside risk stratification scale for identifying patients at risk of poor clinical outcomes. When comparing predictive performance among the three scales, qSOFA had the highest specificity and the Shock Index had the highest AUROC for in-hospital mortality and ED-to-ICU admission in this cohort of patients with suspected sepsis. These trends in sensitivity, specificity, and AUROC were consistent among those who met EHR criteria for a sepsis diagnosis. In the analysis of the three scoring systems using all available cut-points, qSOFA and NEWS2 had the highest AUROCs, followed by the Shock Index.

Considering the rapid progression from organ dysfunction to death in sepsis patients, as well as the difficulty establishing a sepsis diagnosis at triage,²³ providers must quickly identify patients at increased risk of poor outcomes when they present to the ED. Sepsis alerts often are built using SIRS criteria,²⁷ including the one used for sepsis surveillance at UCSF since 2012,²² but the white blood cell count criterion is subject to a laboratory lag and could lead to a delay in identification. Implementation of a point-of-care bedside score alert that uses readily available clinical data could allow providers to identify patients at greatest risk of poor outcomes immediately at ED presentation and triage, which motivated us to explore the predictive performance of qSOFA, the Shock Index, and NEWS2.

Our study is the first to provide a head-to-head comparison of the predictive performance of qSOFA, the Shock Index, and NEWS2, three easy-to-calculate bedside risk scores that use EHR data collected among patients with suspected sepsis. The Sepsis-3 guidelines recommend qSOFA to quickly identify non-ICU patients at greatest risk of poor outcomes because the measure exhibited predictive performance similar to the more extensive SOFA score outside the ICU.^16,23 Although some studies have confirmed qSOFA’s high predictive performance,^28-31 our test characteristics and AUROC findings are in line with other published analyses.^4,6,10,17 The UK National Health Service is using NEWS2 to screen for patients at risk of poor outcomes from sepsis. Several analyses that assessed the predictive ability of NEWS have reported estimates in line with our findings.^4,10,32 The Shock Index was introduced in 1967 and provided a metric to evaluate hemodynamic stability based on heart rate and systolic blood pressure.³³ The Shock Index has been studied in several contexts, including sepsis,³⁴ and studies show that a sustained Shock Index is associated with increased odds of vasopressor administration, higher prevalence of hyperlactatemia, and increased risk of poor outcomes in the ICU.^13,14

For our study, we were particularly interested in exploring how the Shock Index would compare with more frequently used severity scores such as qSOFA and NEWS2 among patients with suspected sepsis, given the simplicity of its calculation and the easy availability of required data. In our cohort of 23,837 patients, only 159 people had missing blood pressure and only 71 had omitted heart rate. In contrast, both qSOFA and NEWS2 include an assessment of level of consciousness that can be subject to variability in assessment methods and EHR documentation across institutions.¹¹ In our cohort, GCS within 30 minutes of ED presentation was missing in 72 patients, which could have led to incomplete calculation of qSOFA and NEWS2 if a missing value was not actually within normal limits.

Several investigations relate qSOFA to NEWS but few compare qSOFA with the newer NEWS2, and even fewer evaluate the Shock Index with any of these scores.^{10,11,18,29,35-37} In general, studies have shown that NEWS exhibits a higher AUROC for predicting mortality, sepsis with organ dysfunction, and ICU admission, often as a composite outcome.^{4,11,18,37,38} A handful of studies compare the Shock Index to SIRS; however, little has been done to compare the Shock Index to qSOFA or NEWS2, scores that have been used specifically for sepsis and might be more predictive of poor outcomes than SIRS.³³ In our study, the Shock Index had a higher AUROC than either qSOFA or NEWS2 for predicting in-hospital mortality and ED-to-ICU admission measured as separate outcomes and as a composite outcome using standard cut-points for these scores.

When selecting a severity score to apply in an institution, it is important to carefully evaluate the score’s test characteristics, in addition to considering the availability of reliable data. Tests with high sensitivity and NPV for the population being studied can be useful to rule out disease or risk of poor outcome, while tests with high specificity and PPV can be useful to rule in disease or risk of poor outcome.³⁹ When considering specificity, qSOFA’s performance was superior to the Shock Index and NEWS2 in our study, but a small percentage of the population was identified using a cut-point of qSOFA ≥2. If we used qSOFA and applied this standard cut-point at our institution, we could be confident that those identified were at increased risk, but we would miss a significant number of patients who would experience a poor outcome. When considering sensitivity, performance of NEWS2 was superior to qSOFA and the Shock Index in our study, but one-half of the population was identified using a cut-point of NEWS2 ≥5. If we were to apply this standard NEWS2 cut-point at our institution, we would assume that one-half of our population was at risk, which might drive resource use towards patients who will not experience a poor outcome. Although none of the scores exhibited a robust AUROC measure, the Shock Index had the highest AUROC for in-hospital mortality and ED-to-ICU admission when using the standard binary cut-point, and its sensitivity and specificity is between that of qSOFA and NEWS2, potentially making it a score to use in settings where qSOFA and NEWS2 score components, such as altered mentation, are not reliably collected. Finally, our sensitivity analysis varying the binary cut-point of each score within our population demonstrated that the standard cut-points might not be as useful within a specific population and might need to be tailored for implementation, balancing sensitivity, specificity, PPV, and NPV to meet local priorities and ICU capacity.

Our study has limitations. It is a single-center, retrospective analysis, factors that could reduce generalizability. However, it does include a large and diverse patient population spanning several years. Missing GCS data could have affected the predictive ability of qSOFA and NEWS2 in our cohort. We could not reliably perform imputation of GCS because of the high missingness and therefore we assumed missing was normal, as was done in the Sepsis-3 derivation studies.¹⁶ Previous studies have attempted to impute GCS and have not observed improved performance of qSOFA to predict mortality.⁴⁰ Because manually collected variables such as GCS are less reliably documented in the EHR, there might be limitations in their use for triage risk scores.

Although the current analysis focused on the predictive performance of qSOFA, the Shock Index, and NEWS2 at triage, performance of these scores could affect the ED team’s treatment decisions before handoff to the hospitalist team and the expected level of care the patient will receive after in-patient admission. These tests also have the advantage of being easy to calculate at the bedside over time, which could provide an objective assessment of longitudinal predicted prognosis. Future work should assess the longitudinal performance of each of these scores among those with suspected sepsis and to determine the impact using these scores would have on clinical and resource utilization outcomes.

Local priorities should drive selection of a screening tool, balancing sensitivity, specificity, PPV, and NPV to achieve the institution’s goals. qSOFA, Shock Index, and NEWS2 are risk stratification tools that can be easily implemented at ED triage using data available at the bedside. Although none of these scores performed strongly when comparing AUROCs, qSOFA was highly specific for identifying patients with poor outcomes, and NEWS2 was the most sensitive for ruling out those at high risk among patients with suspected sepsis. The Shock Index exhibited a sensitivity and specificity that fell between qSOFA and NEWS2 and also might be considered to identify those at increased risk, given its ease of implementation, particularly in settings where altered mentation is unreliably or inconsistently documented.

The authors thank the UCSF Division of Hospital Medicine Data Core for their assistance with data acquisition.

Sepsis is the leading cause of in-hospital mortality in the United States.¹ Sepsis is present on admission in 85% of cases, and each hour delay in antibiotic treatment is associated with 4% to 7% increased odds of mortality.^2,3 Prompt identification and treatment of sepsis is essential for reducing morbidity and mortality, but identifying sepsis during triage is challenging.²

Risk stratification scores that rely solely on data readily available at the bedside have been developed to quickly identify those at greatest risk of poor outcomes from sepsis in real time. The quick Sequential Organ Failure Assessment (qSOFA) score, the National Early Warning System (NEWS2), and the Shock Index are easy-to-calculate measures that use routinely collected clinical data that are not subject to laboratory delay. These scores can be incorporated into electronic health record (EHR)-based alerts and can be calculated longitudinally to track the risk of poor outcomes over time. qSOFA was developed to quantify patient risk at bedside in non-intensive care unit (ICU) settings, but there is no consensus about its ability to predict adverse outcomes such as mortality and ICU admission.^4-6 The United Kingdom’s National Health Service uses NEWS2 to identify patients at risk for sepsis.⁷ NEWS has been shown to have similar or better sensitivity in identifying poorer outcomes in sepsis patients compared with systemic inflammatory response syndrome (SIRS) criteria and qSOFA.^4,8-11 However, since the latest update of NEWS2 in 2017, there has been little study of its predictive ability. The Shock Index is a simple bedside score (heart rate divided by systolic blood pressure) that was developed to detect changes in cardiovascular performance before systemic shock onset. Although it was not developed for infection and has not been regularly applied in the sepsis literature, the Shock Index might be useful for identifying patients at increased risk of poor outcomes. Patients with higher and sustained Shock Index scores are more likely to experience morbidity, such as hyperlactatemia, vasopressor use, and organ failure, and also have an increased risk of mortality.^12-14

Although the predictive abilities of these bedside risk stratification scores have been assessed individually using standard binary cut-points, the comparative performance of qSOFA, the Shock Index, and NEWS2 has not been evaluated in patients presenting to an emergency department (ED) with suspected sepsis. Our objective was to provide a head-to-head comparison of the test characteristics of qSOFA, the Shock Index, and NEWS2 calculated at ED triage for predicting in-hospital mortality and ED-to-ICU admission in patients with suspected sepsis to help health systems and providers select screening measures.

Design and Setting

We conducted a retrospective cohort study of ED patients who presented with suspected sepsis to the University of California San Francisco (UCSF) Helen Diller Medical Center at Parnassus Heights between June 1, 2012, and December 31, 2018. Our institution is a 785-bed academic teaching hospital with approximately 30,000 ED encounters per year. The study was approved with a waiver of informed consent by the UCSF Human Research Protection Program.

Participants

We use an Epic-based EHR platform (Epic 2017, Epic Systems Corporation) for clinical care, which was implemented on June 1, 2012. All data elements were obtained from Clarity, the relational database that stores Epic’s inpatient data. The study included encounters for patients age ≥18 years who had blood cultures ordered within 24 hours of ED presentation and administration of intravenous antibiotics within 24 hours. Repeat encounters were treated independently in our analysis.

Outcomes and Measures

We compared the ability of qSOFA, the Shock Index, and NEWS2 to predict in-hospital mortality and admission to the ICU from the ED (ED-to-ICU admission). We used the most abnormal vital signs and clinical assessments gathered within 30 minutes of ED presentation to identify patients who were qSOFA-positive, Shock Index-positive, and NEWS2-positive based on standard cut-points of risk. Data elements used to calculate qSOFA, Shock Index, and NEWS2 included blood pressure, heart rate, respiratory rate, Glasgow Coma Scale (GCS) score, oxygen saturation, requirement for supplemental oxygen, and temperature (Table 1). Patients were considered positive if they had a qSOFAscore ≥2, Shock Index of >0.7, and NEWS2 ≥5 based on triage vital signs.^7,15,16 We considered patients to have altered mental status, a criterion used for NEWS2, if they had a GCS score <15 instead of using the “alert, verbal, confusion, pain, unresponsive” scale, which is not captured in our EHR, a method that has been used in earlier studies.^17,18 Missing assessments were considered normal. Although our primary analysis focused on the scores calculated within 30 minutes of ED presentation, we performed a sensitivity analysis examining scores calculated within 1 hour of ED presentation in the event of a delay in gathering triage vital sign data.

We compared demographic and clinical characteristics of patients who were positive for qSOFA, the Shock Index, and NEWS2. Demographic data were extracted from the EHR and included primary language, age, sex, and insurance status. All International Classification of Diseases (ICD)-9/10 diagnosis codes were pulled from Clarity billing tables. We used the Elixhauser comorbidity groupings¹⁹ of ICD-9/10 codes present on admission to identify preexisting comorbidities and underlying organ dysfunction. To estimate burden of comorbid illnesses, we calculated the validated van Walraven comorbidity index,²⁰ which provides an estimated risk of in-hospital death based on documented Elixhauser comorbidities. Admission level of care (acute, stepdown, or intensive care) was collected for inpatient admissions to assess initial illness severity.²¹ We also evaluated discharge disposition and in-hospital mortality. Index blood culture results were collected, and dates and timestamps of mechanical ventilation, fluid, vasopressor, and antibiotic administration were obtained for the duration of the encounter.

UCSF uses an automated, real-time, algorithm-based severe sepsis alert that is triggered when a patient meets ≥2 SIRS criteria and again when the patient meets severe sepsis or septic shock criteria (ie, ≥2 SIRS criteria in addition to end-organ dysfunction and/or fluid nonresponsive hypotension). This sepsis screening alert was in use for the duration of our study.²²

Statistical Analysis

We performed a subgroup analysis among those who were diagnosed with sepsis, according to the 2016 Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) criteria. Sepsis is defined as a change in Sequential Organ Failure Assessment (SOFA) score of ≥2 points within the first 48 hours.²³ Additionally, patients meeting Sepsis-3 criteria needed to (1) receive ≥4 days of sequential antibiotic therapy or experience death or discharge to hospice before 4 days of antibiotic therapy or (2) have a validated sepsis discharge billing code. These parameters were added to increase the specificity of our sample.²⁴

All statistical analyses were conducted using Stata 14 (StataCorp). We summarized differences in demographic and clinical characteristics among the populations meeting each severity score but elected not to conduct hypothesis testing because patients could be positive for one or more scores. We calculated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for each score to predict in-hospital mortality and ED-to-ICU admission. To allow comparison with other studies, we also created a composite outcome of either in-hospital mortality or ED-to-ICU admission. To assess score discrimination to predict in-hospital mortality and ED-to-ICU admission, we calculated the area under the receiver operating characteristic curve (AUROC) along with asymptotic normal 95% CI using the “roctab” command considering a binary cut-point, as well as the full range of scores measured in the cohort. The AUROC range from 0.50 to 1.00 and a score in the 0.70 to 0.80 range can be considered fair.²⁵ We assessed significant differences between severity score AUROCs using the DeLong method²⁶ implemented through Stata 14’s “roccomp” command. As a sensitivity analysis, we explored whether the standard cut-points for qSOFA, the Shock Index, and NEWS2 provided the highest AUROC in our population by calculating test characteristics for several score cut-points.

Within our sample 23,837 ED patients had blood cultures ordered within 24 hours of ED presentation and were considered to have suspected sepsis. The mean age of the cohort was 60.8 years, and 1,612 (6.8%) had positive blood cultures. A total of 12,928 patients (54.2%) were found to have sepsis. We documented 1,427 in-hospital deaths (6.0%) and 3,149 (13.2%) ED-to-ICU admissions. At ED triage 1,921 (8.1%) were qSOFA-positive, 4,273 (17.9%) were Shock Index-positive, and 11,832 (49.6%) were NEWS2-positive. At ED triage, blood pressure, heart rate, respiratory rate, and oxygen saturated were documented in >99% of patients, 93.5% had temperature documented, and 28.5% had GCS recorded. If the window of assessment was widened to 1 hour, GCS was only documented among 44.2% of those with suspected sepsis.

Demographic Characteristics and Clinical Course

We identified significant differences when comparing demographic and clinical characteristics among patients who scored positive for the three severity measures at triage (Table 2). Although no hypothesis testing was conducted because patients could meet one or more scores, qSOFA-positive patients were older (median 70, 66, and 64 years, respectively), more likely to have Medicare as the primary payor (67.6% vs 59.7% vs 56.6%), to have chronic renal failure (26.1%, 23.1%, and 23.3%, respectively), to have a greater degree of underlying comorbidities based on the van Walraven Comorbidity Index (median 15, 12, and 11, respectively), and to be admitted to the ICU from the ED (48.1%, 36.3%, and 21.0%, respectively) compared with those positive for the Shock Index or NEWS2.

qSOFA-positive patients received antibiotics more quickly than those who were Shock Index-positive or NEWS2-positive (median 1.5, 1.8, and 2.8 hours after admission, respectively). In addition, those who were qSOFA-positive were more likely to have a positive blood culture (10.9%, 9.4%, and 8.5%, respectively) and to receive an EHR-based diagnosis of sepsis (77.0%, 69.6%, and 60.9%, respectively) than those who were Shock Index- or NEWS2-positive. Those who were qSOFA-positive also were more likely to be mechanically ventilated during their hospital stay (25.4%, 19.2%, and 10.8%, respectively) and to receive vasopressors (33.5%, 22.5%, and 12.2%, respectively). In-hospital mortality also was more common among those who were qSOFA-positive at triage (23.4%, 15.3%, and 9.2%, respectively).

Because both qSOFA and NEWS2 incorporate GCS, we explored baseline characteristics of patients with GCS documented at triage (n = 6,794). These patients were older (median age 63 and 61 years, P < .0001), more likely to be male (54.9% and 53.4%, P = .0031), more likely to have renal failure (22.8% and 20.1%, P < .0001), more likely to have liver disease (14.2% and 12.8%, P = .006), had a higher van Walraven comorbidity score on presentation (median 10 and 8, P < .0001), and were more likely to go directly to the ICU from the ED (20.2% and 10.6%, P < .0001). However, among the 6,397 GCS scores documented at triage, only 1,579 (24.7%) were abnormal.

Test Characteristics of qSOFA, Shock Index, and NEWS2 for Predicting In-hospital Mortality and ED-to-ICU Admission

Among 23,837 patients with suspected sepsis, NEWS2 had the highest sensitivity for predicting in-hospital mortality (76.0%; 95% CI, 73.7%-78.2%) and ED-to-ICU admission (78.9%; 95% CI, 77.5%-80.4%) but had the lowest specificity for in-hospital mortality (52.0%; 95% CI, 51.4%-52.7%) and for ED-to-ICU admission (54.8%; 95% CI, 54.1%-55.5%) (Table 3). qSOFA had the lowest sensitivity for in-hospital mortality (31.5%; 95% CI, 29.1%-33.9%) and ED-to-ICU admission (29.3%; 95% CI, 27.7%-30.9%) but the highest specificity for in-hospital mortality (93.4%; 95% CI, 93.1%-93.8%) and ED-to-ICU admission (95.2%; 95% CI, 94.9%-95.5%). The Shock Index had a sensitivity that fell between qSOFA and NEWS2 for in-hospital mortality (45.8%; 95% CI, 43.2%-48.5%) and ED-to-ICU admission (49.2%; 95% CI, 47.5%-51.0%). The specificity of the Shock Index also was between qSOFA and NEWS2 for in-hospital mortality (83.9%; 95% CI, 83.4%-84.3%) and ED-to-ICU admission (86.8%; 95% CI, 86.4%-87.3%). All three scores exhibited relatively low PPV, ranging from 9.2% to 23.4% for in-hospital mortality and 21.0% to 48.0% for ED-to-ICU triage. Conversely, all three scores exhibited relatively high NPV, ranging from 95.5% to 97.1% for in-hospital mortality and 89.8% to 94.5% for ED-to-ICU triage. The patterns in sensitivity and specificity for in-hospital mortality and ED-to-ICU admission were similar among the 12,928 patients who received an EHR-based sepsis diagnosis with the tests generally demonstrating lower specificities, higher PPVs, and lower NPVs (Table 3).

When considering a binary cutoff, the Shock Index exhibited the highest AUROC for in-hospital mortality (0.648; 95% CI, 0.635-0.662) and had a significantly higher AUROC than qSOFA (AUROC, 0.625; 95% CI, 0.612-0.637; P = .0005), but there was no difference compared with NEWS2 (AUROC, 0.640; 95% CI, 0.628-0.652; P = .2112). NEWS2 had a significantly higher AUROC than qSOFA for predicting in-hospital mortality (P = .0227). The Shock Index also exhibited the highest AUROC for ED-to-ICU admission (0.680; 95% CI, 0.617-0.689), which was significantly higher than the AUROC for qSOFA (P < .0001) and NEWS2 (P = 0.0151). NEWS2 had a significantly higher AUROC than qSOFA for predicting ED-to-ICU admission (P < .0001). Similar findings were seen in patients found to have sepsis. When considering the range of possible scores measured in our cohort, qSOFA and NEWS2 exhibited higher AUROCs for in-hospital mortality and ED-to-ICU admission than the Shock Index among patients with suspected infection and the subgroup with a sepsis diagnosis (Figure).

In this retrospective cohort study of 23,837 patients who presented to the ED with suspected sepsis, the standard qSOFA threshold was met least frequently, followed by the Shock Index and NEWS2. NEWS2 had the highest sensitivity but the lowest specificity for predicting in-hospital mortality and ED-to-ICU admission, making it a challenging bedside risk stratification scale for identifying patients at risk of poor clinical outcomes. When comparing predictive performance among the three scales, qSOFA had the highest specificity and the Shock Index had the highest AUROC for in-hospital mortality and ED-to-ICU admission in this cohort of patients with suspected sepsis. These trends in sensitivity, specificity, and AUROC were consistent among those who met EHR criteria for a sepsis diagnosis. In the analysis of the three scoring systems using all available cut-points, qSOFA and NEWS2 had the highest AUROCs, followed by the Shock Index.

Considering the rapid progression from organ dysfunction to death in sepsis patients, as well as the difficulty establishing a sepsis diagnosis at triage,²³ providers must quickly identify patients at increased risk of poor outcomes when they present to the ED. Sepsis alerts often are built using SIRS criteria,²⁷ including the one used for sepsis surveillance at UCSF since 2012,²² but the white blood cell count criterion is subject to a laboratory lag and could lead to a delay in identification. Implementation of a point-of-care bedside score alert that uses readily available clinical data could allow providers to identify patients at greatest risk of poor outcomes immediately at ED presentation and triage, which motivated us to explore the predictive performance of qSOFA, the Shock Index, and NEWS2.

Our study is the first to provide a head-to-head comparison of the predictive performance of qSOFA, the Shock Index, and NEWS2, three easy-to-calculate bedside risk scores that use EHR data collected among patients with suspected sepsis. The Sepsis-3 guidelines recommend qSOFA to quickly identify non-ICU patients at greatest risk of poor outcomes because the measure exhibited predictive performance similar to the more extensive SOFA score outside the ICU.^16,23 Although some studies have confirmed qSOFA’s high predictive performance,^28-31 our test characteristics and AUROC findings are in line with other published analyses.^4,6,10,17 The UK National Health Service is using NEWS2 to screen for patients at risk of poor outcomes from sepsis. Several analyses that assessed the predictive ability of NEWS have reported estimates in line with our findings.^4,10,32 The Shock Index was introduced in 1967 and provided a metric to evaluate hemodynamic stability based on heart rate and systolic blood pressure.³³ The Shock Index has been studied in several contexts, including sepsis,³⁴ and studies show that a sustained Shock Index is associated with increased odds of vasopressor administration, higher prevalence of hyperlactatemia, and increased risk of poor outcomes in the ICU.^13,14

For our study, we were particularly interested in exploring how the Shock Index would compare with more frequently used severity scores such as qSOFA and NEWS2 among patients with suspected sepsis, given the simplicity of its calculation and the easy availability of required data. In our cohort of 23,837 patients, only 159 people had missing blood pressure and only 71 had omitted heart rate. In contrast, both qSOFA and NEWS2 include an assessment of level of consciousness that can be subject to variability in assessment methods and EHR documentation across institutions.¹¹ In our cohort, GCS within 30 minutes of ED presentation was missing in 72 patients, which could have led to incomplete calculation of qSOFA and NEWS2 if a missing value was not actually within normal limits.

Several investigations relate qSOFA to NEWS but few compare qSOFA with the newer NEWS2, and even fewer evaluate the Shock Index with any of these scores.^{10,11,18,29,35-37} In general, studies have shown that NEWS exhibits a higher AUROC for predicting mortality, sepsis with organ dysfunction, and ICU admission, often as a composite outcome.^{4,11,18,37,38} A handful of studies compare the Shock Index to SIRS; however, little has been done to compare the Shock Index to qSOFA or NEWS2, scores that have been used specifically for sepsis and might be more predictive of poor outcomes than SIRS.³³ In our study, the Shock Index had a higher AUROC than either qSOFA or NEWS2 for predicting in-hospital mortality and ED-to-ICU admission measured as separate outcomes and as a composite outcome using standard cut-points for these scores.

When selecting a severity score to apply in an institution, it is important to carefully evaluate the score’s test characteristics, in addition to considering the availability of reliable data. Tests with high sensitivity and NPV for the population being studied can be useful to rule out disease or risk of poor outcome, while tests with high specificity and PPV can be useful to rule in disease or risk of poor outcome.³⁹ When considering specificity, qSOFA’s performance was superior to the Shock Index and NEWS2 in our study, but a small percentage of the population was identified using a cut-point of qSOFA ≥2. If we used qSOFA and applied this standard cut-point at our institution, we could be confident that those identified were at increased risk, but we would miss a significant number of patients who would experience a poor outcome. When considering sensitivity, performance of NEWS2 was superior to qSOFA and the Shock Index in our study, but one-half of the population was identified using a cut-point of NEWS2 ≥5. If we were to apply this standard NEWS2 cut-point at our institution, we would assume that one-half of our population was at risk, which might drive resource use towards patients who will not experience a poor outcome. Although none of the scores exhibited a robust AUROC measure, the Shock Index had the highest AUROC for in-hospital mortality and ED-to-ICU admission when using the standard binary cut-point, and its sensitivity and specificity is between that of qSOFA and NEWS2, potentially making it a score to use in settings where qSOFA and NEWS2 score components, such as altered mentation, are not reliably collected. Finally, our sensitivity analysis varying the binary cut-point of each score within our population demonstrated that the standard cut-points might not be as useful within a specific population and might need to be tailored for implementation, balancing sensitivity, specificity, PPV, and NPV to meet local priorities and ICU capacity.

Our study has limitations. It is a single-center, retrospective analysis, factors that could reduce generalizability. However, it does include a large and diverse patient population spanning several years. Missing GCS data could have affected the predictive ability of qSOFA and NEWS2 in our cohort. We could not reliably perform imputation of GCS because of the high missingness and therefore we assumed missing was normal, as was done in the Sepsis-3 derivation studies.¹⁶ Previous studies have attempted to impute GCS and have not observed improved performance of qSOFA to predict mortality.⁴⁰ Because manually collected variables such as GCS are less reliably documented in the EHR, there might be limitations in their use for triage risk scores.

Although the current analysis focused on the predictive performance of qSOFA, the Shock Index, and NEWS2 at triage, performance of these scores could affect the ED team’s treatment decisions before handoff to the hospitalist team and the expected level of care the patient will receive after in-patient admission. These tests also have the advantage of being easy to calculate at the bedside over time, which could provide an objective assessment of longitudinal predicted prognosis. Future work should assess the longitudinal performance of each of these scores among those with suspected sepsis and to determine the impact using these scores would have on clinical and resource utilization outcomes.

Local priorities should drive selection of a screening tool, balancing sensitivity, specificity, PPV, and NPV to achieve the institution’s goals. qSOFA, Shock Index, and NEWS2 are risk stratification tools that can be easily implemented at ED triage using data available at the bedside. Although none of these scores performed strongly when comparing AUROCs, qSOFA was highly specific for identifying patients with poor outcomes, and NEWS2 was the most sensitive for ruling out those at high risk among patients with suspected sepsis. The Shock Index exhibited a sensitivity and specificity that fell between qSOFA and NEWS2 and also might be considered to identify those at increased risk, given its ease of implementation, particularly in settings where altered mentation is unreliably or inconsistently documented.

The authors thank the UCSF Division of Hospital Medicine Data Core for their assistance with data acquisition.

The coronavirus disease of 2019 (COVID-19) pandemic has resulted in a surge in hospitalizations of patients with a novel, serious, and highly contagious infectious disease for which there is yet no proven treatment. Currently, much of the focus has been on intensive care unit (ICU) and ventilator capacity for the sickest of these patients who develop respiratory failure. However, most hospitalized patients are being cared for in general medical units.¹ Some evidence exists to describe adaptations to capacity needs outside of medical wards,^2-4 but few studies have specifically addressed the ward setting. Therefore, there is a pressing need for evidence to describe how to expand capacity and deliver medical ward–based care.

To better understand how inpatient care in the United States is adapting to the COVID-19 pandemic, we surveyed 72 sites participating in the Hospital Medicine Reengineering Network (HOMERuN), a national consortium of hospital medicine groups.⁵ We report results of this survey, carried out between April 3 and April 5, 2020.

Sites and Subjects

HOMERuN is a collaborative network of hospitalists from across the United States whose primary goal is to catalyze research and share best practices across hospital medicine groups. Using surveys of Hospital Medicine leaders, targeted medical record review, and other methods, HOMERuN’s funded research interests to date have included care transitions, workforce issues, patient and family engagement, and diagnostic errors. Sites participating in HOMERuN sites are relatively large urban academic medical centers (Appendix).

Survey Development and Deployment

We designed a focused survey that aimed to provide a snapshot of evolving operational and clinical aspects of COVID-19 care (Appendix). Domains included COVID-19 testing turnaround times, personal protective equipment (PPE) stewardship,⁶ features of respiratory isolation units (RIUs; ie, dedicated units for patients with known or suspected COVID-19), and observed effects on clinical care. We tested the instrument to ensure feasibility and clarity internally, performed brief cognitive testing with several hospital medicine leaders in HOMERuN, then disseminated the survey by email on April 3, with two follow-up emails on 2 subsequent days. Our study was deemed non–human subjects research by the University of California, San Francisco, Committee on Human Research. Descriptive statistics were used to characterize survey responses.

Of 72 hospitals surveyed, 51 (71%) responded. Mean hospital bed count was 940, three were safety-net hospitals, and one was a community-based teaching center; responding and nonresponding hospitals did not differ significantly in terms of bed count (Appendix).

Health System Adaptations, Testing, and PPE Status

Nearly all responding hospitals (46 of 51; 90%) had RIUs for patients with known or suspected COVID-19 (Table 1). Nearly all hospitals took steps to keep potentially sick healthcare providers from infecting others (eg, staying home if sick or exposed). Among respondents, 32% had rapid response teams, 24% had respiratory therapy teams, and 29% had case management teams that were dedicated to COVID-19 care. Thirty-two (63%) had developed models, such as ethics or palliative care consult services, to assist with difficult resource-allocation decisions (eg, how to prioritize ventilator use if demand exceeded supply). Twenty-three (45%) had developed post-acute care monitoring programs dedicated to COVID-19 patients.

At the time of our survey, only 2 sites (4%) reported COVID-19 test time turnaround under 1 hour, and 15 (30%) reported turnaround in less than 6 hours. Of the 29 sites able to provide estimates of PPE stockpile, 14 (48%) reported a supply of 2 weeks or less. The most common approaches to PPE stewardship focused on reuse of masks and face shields if not obviously soiled, centralizing PPE distribution, and disinfecting or sterilizing masks. Ten sites (20%) were utilizing 3-D printed masks, while 10% used homemade face shields or masks.

Characteristics of COVID-19 RIUs

Forty-six hospitals (90% of all respondents) in our cohort had developed RIUs at the time of survey administration. The earliest RIU implementation date was February 10, 2020, and the most recent was launched on the day of our survey. Admission to RIUs was primarily based on clinical factors associated with known or suspected COVID-19 infection (Table 2). The number of non–critical care RIU beds among locations at that time ranged from 10 or less to more than 50. The mean number of hospitalist attendings caring for patients in the RIUs was 10.2, with a mean 4.1 advanced practice providers, 5.5 residents, and 0 medical students. The number of planned patients per attending was typically 5 to 15. Nurses and physicians typically rounded separately. Medical distancing (eg, reducing patient room entry) was accomplished most commonly by grouped timing of medication administration (76% of sites), video links to room outside of rounding times (54% of sites), the use of video or telemedicine during rounds (17%), and clustering of activities such as medication administration or phlebotomy. The most common criteria prompting discharge from the RIU were a negative COVID-19 test (59%) and hospital discharge (57%), though comments from many respondents suggested that discharge criteria were changing rapidly.

Effects of Isolation Measures on In-Room Encounters and Diagnostic Processes

More than 90% of sites reported decreases in in-room encounter frequency across all provider types whether as a result of policies in place or not. Reductions were reported among hospitalists, advanced practice providers, residents, consultants, and therapists (Table 3). Reduced room entry most often resulted from an established or developing policy, but many noted reduced room entry without formal policies in place. Nearly all sites reported moving specialty consultations to phone or video evaluations. Diagnostic error was commonly reported, with missed non–COVID-19 medical diagnoses among COVID-19 infected patients being reported by 22 sites (46%) and missed COVID-19 diagnoses in patients admitted for other reasons by 22 sites (45%).

In this study of medical wards at academic medical centers, we found that, in response to the COVID-19 pandemic, hospitals made several changes in a short period of time to adapt to the crisis. These included implementation and rapid expansion of dedicated RIUs, greatly expanded use of inpatient telehealth for patient assessments and consultation, implementation of other approaches to minimize room entry (such as grouping in-room activities), and deployment of ethics consultation services to help manage issues around potential scarcity of life-saving measures such as ventilators. We also found that availability of PPE and timely testing was limited. Finally, a large proportion of sites reported potential diagnostic problems in the assessment of both patients suspected and those not suspected of having COVID-19.

RIUs are emerging as a primary modality for caring for non-ICU COVID-19 patients, though they never involved medical students; we hope the role of students in particular will increase as new models of training emerge in response to the pandemic.⁷ In contrast, telemedicine evolved rapidly to hold a substantial role in RIUs, with both ward and specialty teams using video visit technology to communicate with patients. COVID-19 has been viewed as a perfect use case for outpatient telemedicine,⁸ and a growing number of studies are examining its outpatient use^9,10; however, to date, somewhat less attention has been paid to inpatient deployment. Although our data suggest telemedicine has found a prominent place in RIUs, it remains to be seen whether it is associated with differences in patient or provider outcomes. For example, deficiencies in the physical examination, limited face-to-face contact, and lack of physical presence could all affect the patient–provider relationship, patient engagement, and the accuracy of the diagnostic process.

Our data suggest the possibility of missing non–COVID-19 diagnoses in patients suspected of COVID-19 and missing COVID-19 in those admitted for nonrespiratory reasons. The latter may be addressed as routine COVID-19 screening of admitted patients becomes commonplace. For the former, however, it is possible that physicians are “anchoring” their thinking on COVID-19 to the exclusion of other diagnoses, that physicians are not fully aware of complications unique to COVID-19 infection (such as thromboembolism), and/or that the above-mentioned limitations of telemedicine have decreased diagnostic performance.

Although PPE stockpile data were not easily available for some sites, a distressingly large number reported stockpiles of 2 weeks or less, with reuse being the most common approach to extending PPE supply. We also found it concerning that 43% of hospital leaders did not know their stockpile data; we believe this is an important question that hospital leaders need to be asking. Most sites in our study reported test turnaround times of longer than 6 hours; lack of rapid COVID-19 testing further stresses PPE stockpile and may slow patients’ transition out of the RIU or discharge to home.

Our study has several limitations, including the evolving nature of the pandemic and rapid adaptations of care systems in the pandemic’s surge phase. However, we attempted to frame our questions in ways that provided a focused snapshot of care. Furthermore, respondents may not have had exhaustive knowledge of their institution’s COVID-19 response strategies, but most were the directors of their hospitalist services, and we encouraged the respondents to confer with others to gather high-fidelity data. Finally, as a survey of large academic medical centers, our results may not apply to nonacademic centers.

Approaches to caring for non-ICU patients during the COVID-19 pandemic are rapidly evolving. Expansion of RIUs and developing the workforce to support them has been a primary focus, with rapid innovation in use of technology emerging as a critical adaptation while PPE limitations persist and needs for “medical distancing” continue to grow. Although rates of missed COVID-19 diagnoses will likely be reduced with testing and systems improvements, physicians and systems will also need to consider how to utilize emerging technology in ways that can improve clinical care and provider safety while aiding diagnostic thinking. This survey illustrates the rapid adaptations made by our hospitals in response to the pandemic; ongoing adaptation will likely be needed to optimally care for hospitalized patients with COVID-19 while the pandemic continues to evolve.

Thanks to members of the HOMERuN COVID-19 Collaborative Group: Baylor Scott & White Medical Center – Temple, Texas - Tresa McNeal MD; Beth Israel Deaconess Medical Center - Shani Herzig MD MPH, Joseph Li MD, Julius Yang MD PhD; Brigham and Women’s Hospital - Christopher Roy MD, Jeffrey Schnipper MD MPH; Cedars-Sinai Medical Center - Ed Seferian MD, ; ChristianaCare - Surekha Bhamidipati MD; Cleveland Clinic - Matthew Pappas MD MPH; Dartmouth-Hitchcock Medical Center - Jonathan Lurie MD MS; Dell Medical School at The University of Texas at Austin - Chris Moriates MD, Luci Leykum MD MBA MSc; Denver Health and Hospitals Authority - Diana Mancini MD; Emory University Hospital - Dan Hunt MD; Johns Hopkins Hospital - Daniel J Brotman MD, Zishan K Siddiqui MD, Shaker Eid MD MBA; Maine Medical Center - Daniel A Meyer MD, Robert Trowbridge MD; Massachusetts General Hospital - Melissa Mattison MD; Mayo Clinic Rochester – Caroline Burton MD, Sagar Dugani MD PhD; Medical College of Wisconsin - Sanjay Bhandari MD; Miriam Hospital - Kwame Dapaah-Afriyie MD MBA; Mount Sinai Hospital - Andrew Dunn MD; NorthShore - David Lovinger MD; Northwestern Memorial Hospital - Kevin O’Leary MD MS; Ohio State University Wexner Medical Center - Eric Schumacher DO; Oregon Health & Science University - Angela Alday MD; Penn Medicine - Ryan Greysen MD MHS MA; Rutgers- Robert Wood Johnson University Hospital - Michael Steinberg MD MPH; Stanford University School of Medicine - Neera Ahuja MD; Tulane Hospital and University Medical Center - Geraldine Ménard MD; UC San Diego Health - Ian Jenkins MD; UC Los Angeles Health - Michael Lazarus MD, Magdalena E. Ptaszny, MD; UC San Francisco Health - Bradley A Sharpe, MD, Margaret Fang MD MPH; UK HealthCare - Mark Williams MD MHM, John Romond MD; University of Chicago – David Meltzer MD PhD, Gregory Ruhnke MD; University of Colorado - Marisha Burden MD; University of Florida - Nila Radhakrishnan MD; University of Iowa Hospitals and Clinics - Kevin Glenn MD MS; University of Miami - Efren Manjarrez MD; University of Michigan - Vineet Chopra MD MSc, Valerie Vaughn MD MSc; University of Missouri-Columbia Hospital - Hasan Naqvi MD; University of Nebraska Medical Center - Chad Vokoun MD; University of North Carolina at Chapel Hill - David Hemsey MD; University of Pittsburgh Medical Center - Gena Marie Walker MD; University of Vermont Medical Center - Steven Grant MD; University of Washington Medical Center - Christopher Kim MD MBA, Andrew White MD; University of Washington-Harborview Medical Center - Maralyssa Bann MD; University of Wisconsin Hospital and Clinics - David Sterken MD, Farah Kaiksow MD MPP, Ann Sheehy MD MS, Jordan Kenik MD MPH; UW Northwest Campus - Ben Wolpaw MD; Vanderbilt University Medical Center - Sunil Kripalani MD MSc, Eduard E Vasilevskis MD, Kathleene T Wooldridge MD MPH; Wake Forest Baptist Health - Erik Summers MD; Washington University St. Louis - Michael Lin MD; Weill Cornell - Justin Choi MD; Yale New Haven Hospital - William Cushing MA, Chris Sankey MD; Zuckerberg San Francisco General Hospital - Sumant Ranji MD.

The coronavirus disease of 2019 (COVID-19) pandemic has resulted in a surge in hospitalizations of patients with a novel, serious, and highly contagious infectious disease for which there is yet no proven treatment. Currently, much of the focus has been on intensive care unit (ICU) and ventilator capacity for the sickest of these patients who develop respiratory failure. However, most hospitalized patients are being cared for in general medical units.¹ Some evidence exists to describe adaptations to capacity needs outside of medical wards,^2-4 but few studies have specifically addressed the ward setting. Therefore, there is a pressing need for evidence to describe how to expand capacity and deliver medical ward–based care.

To better understand how inpatient care in the United States is adapting to the COVID-19 pandemic, we surveyed 72 sites participating in the Hospital Medicine Reengineering Network (HOMERuN), a national consortium of hospital medicine groups.⁵ We report results of this survey, carried out between April 3 and April 5, 2020.

Sites and Subjects

HOMERuN is a collaborative network of hospitalists from across the United States whose primary goal is to catalyze research and share best practices across hospital medicine groups. Using surveys of Hospital Medicine leaders, targeted medical record review, and other methods, HOMERuN’s funded research interests to date have included care transitions, workforce issues, patient and family engagement, and diagnostic errors. Sites participating in HOMERuN sites are relatively large urban academic medical centers (Appendix).

Survey Development and Deployment

We designed a focused survey that aimed to provide a snapshot of evolving operational and clinical aspects of COVID-19 care (Appendix). Domains included COVID-19 testing turnaround times, personal protective equipment (PPE) stewardship,⁶ features of respiratory isolation units (RIUs; ie, dedicated units for patients with known or suspected COVID-19), and observed effects on clinical care. We tested the instrument to ensure feasibility and clarity internally, performed brief cognitive testing with several hospital medicine leaders in HOMERuN, then disseminated the survey by email on April 3, with two follow-up emails on 2 subsequent days. Our study was deemed non–human subjects research by the University of California, San Francisco, Committee on Human Research. Descriptive statistics were used to characterize survey responses.

Of 72 hospitals surveyed, 51 (71%) responded. Mean hospital bed count was 940, three were safety-net hospitals, and one was a community-based teaching center; responding and nonresponding hospitals did not differ significantly in terms of bed count (Appendix).

Health System Adaptations, Testing, and PPE Status

Nearly all responding hospitals (46 of 51; 90%) had RIUs for patients with known or suspected COVID-19 (Table 1). Nearly all hospitals took steps to keep potentially sick healthcare providers from infecting others (eg, staying home if sick or exposed). Among respondents, 32% had rapid response teams, 24% had respiratory therapy teams, and 29% had case management teams that were dedicated to COVID-19 care. Thirty-two (63%) had developed models, such as ethics or palliative care consult services, to assist with difficult resource-allocation decisions (eg, how to prioritize ventilator use if demand exceeded supply). Twenty-three (45%) had developed post-acute care monitoring programs dedicated to COVID-19 patients.

At the time of our survey, only 2 sites (4%) reported COVID-19 test time turnaround under 1 hour, and 15 (30%) reported turnaround in less than 6 hours. Of the 29 sites able to provide estimates of PPE stockpile, 14 (48%) reported a supply of 2 weeks or less. The most common approaches to PPE stewardship focused on reuse of masks and face shields if not obviously soiled, centralizing PPE distribution, and disinfecting or sterilizing masks. Ten sites (20%) were utilizing 3-D printed masks, while 10% used homemade face shields or masks.

Characteristics of COVID-19 RIUs

Forty-six hospitals (90% of all respondents) in our cohort had developed RIUs at the time of survey administration. The earliest RIU implementation date was February 10, 2020, and the most recent was launched on the day of our survey. Admission to RIUs was primarily based on clinical factors associated with known or suspected COVID-19 infection (Table 2). The number of non–critical care RIU beds among locations at that time ranged from 10 or less to more than 50. The mean number of hospitalist attendings caring for patients in the RIUs was 10.2, with a mean 4.1 advanced practice providers, 5.5 residents, and 0 medical students. The number of planned patients per attending was typically 5 to 15. Nurses and physicians typically rounded separately. Medical distancing (eg, reducing patient room entry) was accomplished most commonly by grouped timing of medication administration (76% of sites), video links to room outside of rounding times (54% of sites), the use of video or telemedicine during rounds (17%), and clustering of activities such as medication administration or phlebotomy. The most common criteria prompting discharge from the RIU were a negative COVID-19 test (59%) and hospital discharge (57%), though comments from many respondents suggested that discharge criteria were changing rapidly.

Effects of Isolation Measures on In-Room Encounters and Diagnostic Processes

More than 90% of sites reported decreases in in-room encounter frequency across all provider types whether as a result of policies in place or not. Reductions were reported among hospitalists, advanced practice providers, residents, consultants, and therapists (Table 3). Reduced room entry most often resulted from an established or developing policy, but many noted reduced room entry without formal policies in place. Nearly all sites reported moving specialty consultations to phone or video evaluations. Diagnostic error was commonly reported, with missed non–COVID-19 medical diagnoses among COVID-19 infected patients being reported by 22 sites (46%) and missed COVID-19 diagnoses in patients admitted for other reasons by 22 sites (45%).

In this study of medical wards at academic medical centers, we found that, in response to the COVID-19 pandemic, hospitals made several changes in a short period of time to adapt to the crisis. These included implementation and rapid expansion of dedicated RIUs, greatly expanded use of inpatient telehealth for patient assessments and consultation, implementation of other approaches to minimize room entry (such as grouping in-room activities), and deployment of ethics consultation services to help manage issues around potential scarcity of life-saving measures such as ventilators. We also found that availability of PPE and timely testing was limited. Finally, a large proportion of sites reported potential diagnostic problems in the assessment of both patients suspected and those not suspected of having COVID-19.

RIUs are emerging as a primary modality for caring for non-ICU COVID-19 patients, though they never involved medical students; we hope the role of students in particular will increase as new models of training emerge in response to the pandemic.⁷ In contrast, telemedicine evolved rapidly to hold a substantial role in RIUs, with both ward and specialty teams using video visit technology to communicate with patients. COVID-19 has been viewed as a perfect use case for outpatient telemedicine,⁸ and a growing number of studies are examining its outpatient use^9,10; however, to date, somewhat less attention has been paid to inpatient deployment. Although our data suggest telemedicine has found a prominent place in RIUs, it remains to be seen whether it is associated with differences in patient or provider outcomes. For example, deficiencies in the physical examination, limited face-to-face contact, and lack of physical presence could all affect the patient–provider relationship, patient engagement, and the accuracy of the diagnostic process.

Our data suggest the possibility of missing non–COVID-19 diagnoses in patients suspected of COVID-19 and missing COVID-19 in those admitted for nonrespiratory reasons. The latter may be addressed as routine COVID-19 screening of admitted patients becomes commonplace. For the former, however, it is possible that physicians are “anchoring” their thinking on COVID-19 to the exclusion of other diagnoses, that physicians are not fully aware of complications unique to COVID-19 infection (such as thromboembolism), and/or that the above-mentioned limitations of telemedicine have decreased diagnostic performance.

Although PPE stockpile data were not easily available for some sites, a distressingly large number reported stockpiles of 2 weeks or less, with reuse being the most common approach to extending PPE supply. We also found it concerning that 43% of hospital leaders did not know their stockpile data; we believe this is an important question that hospital leaders need to be asking. Most sites in our study reported test turnaround times of longer than 6 hours; lack of rapid COVID-19 testing further stresses PPE stockpile and may slow patients’ transition out of the RIU or discharge to home.

Our study has several limitations, including the evolving nature of the pandemic and rapid adaptations of care systems in the pandemic’s surge phase. However, we attempted to frame our questions in ways that provided a focused snapshot of care. Furthermore, respondents may not have had exhaustive knowledge of their institution’s COVID-19 response strategies, but most were the directors of their hospitalist services, and we encouraged the respondents to confer with others to gather high-fidelity data. Finally, as a survey of large academic medical centers, our results may not apply to nonacademic centers.

Approaches to caring for non-ICU patients during the COVID-19 pandemic are rapidly evolving. Expansion of RIUs and developing the workforce to support them has been a primary focus, with rapid innovation in use of technology emerging as a critical adaptation while PPE limitations persist and needs for “medical distancing” continue to grow. Although rates of missed COVID-19 diagnoses will likely be reduced with testing and systems improvements, physicians and systems will also need to consider how to utilize emerging technology in ways that can improve clinical care and provider safety while aiding diagnostic thinking. This survey illustrates the rapid adaptations made by our hospitals in response to the pandemic; ongoing adaptation will likely be needed to optimally care for hospitalized patients with COVID-19 while the pandemic continues to evolve.

Thanks to members of the HOMERuN COVID-19 Collaborative Group: Baylor Scott & White Medical Center – Temple, Texas - Tresa McNeal MD; Beth Israel Deaconess Medical Center - Shani Herzig MD MPH, Joseph Li MD, Julius Yang MD PhD; Brigham and Women’s Hospital - Christopher Roy MD, Jeffrey Schnipper MD MPH; Cedars-Sinai Medical Center - Ed Seferian MD, ; ChristianaCare - Surekha Bhamidipati MD; Cleveland Clinic - Matthew Pappas MD MPH; Dartmouth-Hitchcock Medical Center - Jonathan Lurie MD MS; Dell Medical School at The University of Texas at Austin - Chris Moriates MD, Luci Leykum MD MBA MSc; Denver Health and Hospitals Authority - Diana Mancini MD; Emory University Hospital - Dan Hunt MD; Johns Hopkins Hospital - Daniel J Brotman MD, Zishan K Siddiqui MD, Shaker Eid MD MBA; Maine Medical Center - Daniel A Meyer MD, Robert Trowbridge MD; Massachusetts General Hospital - Melissa Mattison MD; Mayo Clinic Rochester – Caroline Burton MD, Sagar Dugani MD PhD; Medical College of Wisconsin - Sanjay Bhandari MD; Miriam Hospital - Kwame Dapaah-Afriyie MD MBA; Mount Sinai Hospital - Andrew Dunn MD; NorthShore - David Lovinger MD; Northwestern Memorial Hospital - Kevin O’Leary MD MS; Ohio State University Wexner Medical Center - Eric Schumacher DO; Oregon Health & Science University - Angela Alday MD; Penn Medicine - Ryan Greysen MD MHS MA; Rutgers- Robert Wood Johnson University Hospital - Michael Steinberg MD MPH; Stanford University School of Medicine - Neera Ahuja MD; Tulane Hospital and University Medical Center - Geraldine Ménard MD; UC San Diego Health - Ian Jenkins MD; UC Los Angeles Health - Michael Lazarus MD, Magdalena E. Ptaszny, MD; UC San Francisco Health - Bradley A Sharpe, MD, Margaret Fang MD MPH; UK HealthCare - Mark Williams MD MHM, John Romond MD; University of Chicago – David Meltzer MD PhD, Gregory Ruhnke MD; University of Colorado - Marisha Burden MD; University of Florida - Nila Radhakrishnan MD; University of Iowa Hospitals and Clinics - Kevin Glenn MD MS; University of Miami - Efren Manjarrez MD; University of Michigan - Vineet Chopra MD MSc, Valerie Vaughn MD MSc; University of Missouri-Columbia Hospital - Hasan Naqvi MD; University of Nebraska Medical Center - Chad Vokoun MD; University of North Carolina at Chapel Hill - David Hemsey MD; University of Pittsburgh Medical Center - Gena Marie Walker MD; University of Vermont Medical Center - Steven Grant MD; University of Washington Medical Center - Christopher Kim MD MBA, Andrew White MD; University of Washington-Harborview Medical Center - Maralyssa Bann MD; University of Wisconsin Hospital and Clinics - David Sterken MD, Farah Kaiksow MD MPP, Ann Sheehy MD MS, Jordan Kenik MD MPH; UW Northwest Campus - Ben Wolpaw MD; Vanderbilt University Medical Center - Sunil Kripalani MD MSc, Eduard E Vasilevskis MD, Kathleene T Wooldridge MD MPH; Wake Forest Baptist Health - Erik Summers MD; Washington University St. Louis - Michael Lin MD; Weill Cornell - Justin Choi MD; Yale New Haven Hospital - William Cushing MA, Chris Sankey MD; Zuckerberg San Francisco General Hospital - Sumant Ranji MD.

The coronavirus disease of 2019 (COVID-19) pandemic has resulted in a surge in hospitalizations of patients with a novel, serious, and highly contagious infectious disease for which there is yet no proven treatment. Currently, much of the focus has been on intensive care unit (ICU) and ventilator capacity for the sickest of these patients who develop respiratory failure. However, most hospitalized patients are being cared for in general medical units.¹ Some evidence exists to describe adaptations to capacity needs outside of medical wards,^2-4 but few studies have specifically addressed the ward setting. Therefore, there is a pressing need for evidence to describe how to expand capacity and deliver medical ward–based care.

To better understand how inpatient care in the United States is adapting to the COVID-19 pandemic, we surveyed 72 sites participating in the Hospital Medicine Reengineering Network (HOMERuN), a national consortium of hospital medicine groups.⁵ We report results of this survey, carried out between April 3 and April 5, 2020.

Sites and Subjects

HOMERuN is a collaborative network of hospitalists from across the United States whose primary goal is to catalyze research and share best practices across hospital medicine groups. Using surveys of Hospital Medicine leaders, targeted medical record review, and other methods, HOMERuN’s funded research interests to date have included care transitions, workforce issues, patient and family engagement, and diagnostic errors. Sites participating in HOMERuN sites are relatively large urban academic medical centers (Appendix).

Survey Development and Deployment

We designed a focused survey that aimed to provide a snapshot of evolving operational and clinical aspects of COVID-19 care (Appendix). Domains included COVID-19 testing turnaround times, personal protective equipment (PPE) stewardship,⁶ features of respiratory isolation units (RIUs; ie, dedicated units for patients with known or suspected COVID-19), and observed effects on clinical care. We tested the instrument to ensure feasibility and clarity internally, performed brief cognitive testing with several hospital medicine leaders in HOMERuN, then disseminated the survey by email on April 3, with two follow-up emails on 2 subsequent days. Our study was deemed non–human subjects research by the University of California, San Francisco, Committee on Human Research. Descriptive statistics were used to characterize survey responses.

Of 72 hospitals surveyed, 51 (71%) responded. Mean hospital bed count was 940, three were safety-net hospitals, and one was a community-based teaching center; responding and nonresponding hospitals did not differ significantly in terms of bed count (Appendix).

Health System Adaptations, Testing, and PPE Status

Nearly all responding hospitals (46 of 51; 90%) had RIUs for patients with known or suspected COVID-19 (Table 1). Nearly all hospitals took steps to keep potentially sick healthcare providers from infecting others (eg, staying home if sick or exposed). Among respondents, 32% had rapid response teams, 24% had respiratory therapy teams, and 29% had case management teams that were dedicated to COVID-19 care. Thirty-two (63%) had developed models, such as ethics or palliative care consult services, to assist with difficult resource-allocation decisions (eg, how to prioritize ventilator use if demand exceeded supply). Twenty-three (45%) had developed post-acute care monitoring programs dedicated to COVID-19 patients.

At the time of our survey, only 2 sites (4%) reported COVID-19 test time turnaround under 1 hour, and 15 (30%) reported turnaround in less than 6 hours. Of the 29 sites able to provide estimates of PPE stockpile, 14 (48%) reported a supply of 2 weeks or less. The most common approaches to PPE stewardship focused on reuse of masks and face shields if not obviously soiled, centralizing PPE distribution, and disinfecting or sterilizing masks. Ten sites (20%) were utilizing 3-D printed masks, while 10% used homemade face shields or masks.

Characteristics of COVID-19 RIUs

Forty-six hospitals (90% of all respondents) in our cohort had developed RIUs at the time of survey administration. The earliest RIU implementation date was February 10, 2020, and the most recent was launched on the day of our survey. Admission to RIUs was primarily based on clinical factors associated with known or suspected COVID-19 infection (Table 2). The number of non–critical care RIU beds among locations at that time ranged from 10 or less to more than 50. The mean number of hospitalist attendings caring for patients in the RIUs was 10.2, with a mean 4.1 advanced practice providers, 5.5 residents, and 0 medical students. The number of planned patients per attending was typically 5 to 15. Nurses and physicians typically rounded separately. Medical distancing (eg, reducing patient room entry) was accomplished most commonly by grouped timing of medication administration (76% of sites), video links to room outside of rounding times (54% of sites), the use of video or telemedicine during rounds (17%), and clustering of activities such as medication administration or phlebotomy. The most common criteria prompting discharge from the RIU were a negative COVID-19 test (59%) and hospital discharge (57%), though comments from many respondents suggested that discharge criteria were changing rapidly.

Effects of Isolation Measures on In-Room Encounters and Diagnostic Processes

More than 90% of sites reported decreases in in-room encounter frequency across all provider types whether as a result of policies in place or not. Reductions were reported among hospitalists, advanced practice providers, residents, consultants, and therapists (Table 3). Reduced room entry most often resulted from an established or developing policy, but many noted reduced room entry without formal policies in place. Nearly all sites reported moving specialty consultations to phone or video evaluations. Diagnostic error was commonly reported, with missed non–COVID-19 medical diagnoses among COVID-19 infected patients being reported by 22 sites (46%) and missed COVID-19 diagnoses in patients admitted for other reasons by 22 sites (45%).

In this study of medical wards at academic medical centers, we found that, in response to the COVID-19 pandemic, hospitals made several changes in a short period of time to adapt to the crisis. These included implementation and rapid expansion of dedicated RIUs, greatly expanded use of inpatient telehealth for patient assessments and consultation, implementation of other approaches to minimize room entry (such as grouping in-room activities), and deployment of ethics consultation services to help manage issues around potential scarcity of life-saving measures such as ventilators. We also found that availability of PPE and timely testing was limited. Finally, a large proportion of sites reported potential diagnostic problems in the assessment of both patients suspected and those not suspected of having COVID-19.

RIUs are emerging as a primary modality for caring for non-ICU COVID-19 patients, though they never involved medical students; we hope the role of students in particular will increase as new models of training emerge in response to the pandemic.⁷ In contrast, telemedicine evolved rapidly to hold a substantial role in RIUs, with both ward and specialty teams using video visit technology to communicate with patients. COVID-19 has been viewed as a perfect use case for outpatient telemedicine,⁸ and a growing number of studies are examining its outpatient use^9,10; however, to date, somewhat less attention has been paid to inpatient deployment. Although our data suggest telemedicine has found a prominent place in RIUs, it remains to be seen whether it is associated with differences in patient or provider outcomes. For example, deficiencies in the physical examination, limited face-to-face contact, and lack of physical presence could all affect the patient–provider relationship, patient engagement, and the accuracy of the diagnostic process.

Our data suggest the possibility of missing non–COVID-19 diagnoses in patients suspected of COVID-19 and missing COVID-19 in those admitted for nonrespiratory reasons. The latter may be addressed as routine COVID-19 screening of admitted patients becomes commonplace. For the former, however, it is possible that physicians are “anchoring” their thinking on COVID-19 to the exclusion of other diagnoses, that physicians are not fully aware of complications unique to COVID-19 infection (such as thromboembolism), and/or that the above-mentioned limitations of telemedicine have decreased diagnostic performance.

Although PPE stockpile data were not easily available for some sites, a distressingly large number reported stockpiles of 2 weeks or less, with reuse being the most common approach to extending PPE supply. We also found it concerning that 43% of hospital leaders did not know their stockpile data; we believe this is an important question that hospital leaders need to be asking. Most sites in our study reported test turnaround times of longer than 6 hours; lack of rapid COVID-19 testing further stresses PPE stockpile and may slow patients’ transition out of the RIU or discharge to home.

Our study has several limitations, including the evolving nature of the pandemic and rapid adaptations of care systems in the pandemic’s surge phase. However, we attempted to frame our questions in ways that provided a focused snapshot of care. Furthermore, respondents may not have had exhaustive knowledge of their institution’s COVID-19 response strategies, but most were the directors of their hospitalist services, and we encouraged the respondents to confer with others to gather high-fidelity data. Finally, as a survey of large academic medical centers, our results may not apply to nonacademic centers.

Approaches to caring for non-ICU patients during the COVID-19 pandemic are rapidly evolving. Expansion of RIUs and developing the workforce to support them has been a primary focus, with rapid innovation in use of technology emerging as a critical adaptation while PPE limitations persist and needs for “medical distancing” continue to grow. Although rates of missed COVID-19 diagnoses will likely be reduced with testing and systems improvements, physicians and systems will also need to consider how to utilize emerging technology in ways that can improve clinical care and provider safety while aiding diagnostic thinking. This survey illustrates the rapid adaptations made by our hospitals in response to the pandemic; ongoing adaptation will likely be needed to optimally care for hospitalized patients with COVID-19 while the pandemic continues to evolve.

Thanks to members of the HOMERuN COVID-19 Collaborative Group: Baylor Scott & White Medical Center – Temple, Texas - Tresa McNeal MD; Beth Israel Deaconess Medical Center - Shani Herzig MD MPH, Joseph Li MD, Julius Yang MD PhD; Brigham and Women’s Hospital - Christopher Roy MD, Jeffrey Schnipper MD MPH; Cedars-Sinai Medical Center - Ed Seferian MD, ; ChristianaCare - Surekha Bhamidipati MD; Cleveland Clinic - Matthew Pappas MD MPH; Dartmouth-Hitchcock Medical Center - Jonathan Lurie MD MS; Dell Medical School at The University of Texas at Austin - Chris Moriates MD, Luci Leykum MD MBA MSc; Denver Health and Hospitals Authority - Diana Mancini MD; Emory University Hospital - Dan Hunt MD; Johns Hopkins Hospital - Daniel J Brotman MD, Zishan K Siddiqui MD, Shaker Eid MD MBA; Maine Medical Center - Daniel A Meyer MD, Robert Trowbridge MD; Massachusetts General Hospital - Melissa Mattison MD; Mayo Clinic Rochester – Caroline Burton MD, Sagar Dugani MD PhD; Medical College of Wisconsin - Sanjay Bhandari MD; Miriam Hospital - Kwame Dapaah-Afriyie MD MBA; Mount Sinai Hospital - Andrew Dunn MD; NorthShore - David Lovinger MD; Northwestern Memorial Hospital - Kevin O’Leary MD MS; Ohio State University Wexner Medical Center - Eric Schumacher DO; Oregon Health & Science University - Angela Alday MD; Penn Medicine - Ryan Greysen MD MHS MA; Rutgers- Robert Wood Johnson University Hospital - Michael Steinberg MD MPH; Stanford University School of Medicine - Neera Ahuja MD; Tulane Hospital and University Medical Center - Geraldine Ménard MD; UC San Diego Health - Ian Jenkins MD; UC Los Angeles Health - Michael Lazarus MD, Magdalena E. Ptaszny, MD; UC San Francisco Health - Bradley A Sharpe, MD, Margaret Fang MD MPH; UK HealthCare - Mark Williams MD MHM, John Romond MD; University of Chicago – David Meltzer MD PhD, Gregory Ruhnke MD; University of Colorado - Marisha Burden MD; University of Florida - Nila Radhakrishnan MD; University of Iowa Hospitals and Clinics - Kevin Glenn MD MS; University of Miami - Efren Manjarrez MD; University of Michigan - Vineet Chopra MD MSc, Valerie Vaughn MD MSc; University of Missouri-Columbia Hospital - Hasan Naqvi MD; University of Nebraska Medical Center - Chad Vokoun MD; University of North Carolina at Chapel Hill - David Hemsey MD; University of Pittsburgh Medical Center - Gena Marie Walker MD; University of Vermont Medical Center - Steven Grant MD; University of Washington Medical Center - Christopher Kim MD MBA, Andrew White MD; University of Washington-Harborview Medical Center - Maralyssa Bann MD; University of Wisconsin Hospital and Clinics - David Sterken MD, Farah Kaiksow MD MPP, Ann Sheehy MD MS, Jordan Kenik MD MPH; UW Northwest Campus - Ben Wolpaw MD; Vanderbilt University Medical Center - Sunil Kripalani MD MSc, Eduard E Vasilevskis MD, Kathleene T Wooldridge MD MPH; Wake Forest Baptist Health - Erik Summers MD; Washington University St. Louis - Michael Lin MD; Weill Cornell - Justin Choi MD; Yale New Haven Hospital - William Cushing MA, Chris Sankey MD; Zuckerberg San Francisco General Hospital - Sumant Ranji MD.

Sepsis is defined as a life-threatening organ dysfunction that occurs in response to systemic infection.^1,2 It is frequently fatal, common in hospital medicine, and a leading contributor to critical illness, morbidity, and healthcare expenditures.^2-5 While sepsis care and outcomes have improved in the past decade,^6,7 inpatient mortality remains high.⁸

A number of studies have sought to determine whether race plays a role in sepsis mortality. While Black patients with sepsis have frequently been identified as having the highest rates of death,^9-14 similar observations have been made for most non-White races/ethnicities.^13-15 Studies have also demonstrated higher rates of hospital-acquired infections among Asian and Latino patients.¹⁶

There are several possible explanations for why racial minorities experience disparate outcomes in sepsis, including access to care, comorbidities, implicit biases, and biological or environmental factors,^17-20 as well as characteristics of hospitals most likely to care for racial minorities.^13,15,21 One explanation that has not been explored is that racial disparities in sepsis are mediated by language. Limited English proficiency (LEP) has previously been associated with increased rates of adverse hospital events,²² longer length of stay,²³ and greater likelihood of readmission.²⁴ LEP has also been shown to represent a significant barrier to accessing healthcare and preventive screening.²⁵ The role of LEP in sepsis mortality, however, has yet to be examined.

The diverse patient population at the University of California, San Francisco (UCSF) provides a unique opportunity to build upon existing literature by further exploring racial differences in sepsis, specifically by investigating the role of LEP. The objective of this study was to determine the association between LEP and inpatient mortality among adults hospitalized with sepsis.

Setting

The study was conducted at the University of California, San Francisco, California (UCSF), an 800-bed tertiary care, academic medical center. It was approved by the UCSF Institutional Review Board with waiver of informed consent. UCSF cares for a population of patients who are racially and linguistically diverse, with high proportions of patients of East Asian descent and with LEP. According to recent United States census estimates, more than half of San Francisco County residents identify as non-White (35% Asians, 15% Hispanic/Latino, 6% Black), and 44% report speaking a language other than English at home.²⁶

Study Population and Data Collection

The UCSF Medical Center uses the electronic health record (EHR) Epic (Epic 2017, Epic Systems Corporation, Verona, Wisconsin). We obtained computerized EHR data from Clarity, the relational database that stores Epic’s inpatient data in thousands of tables. We identified all patients ≥18 years of age presenting to the emergency department (ED) between June 1, 2012 and December 31, 2016 with suspected serious infection, defined as having blood cultures ordered within 72 hours of ED presentation (N = 25,441). Patients who did not receive at least two doses of intravenous (IV) antibiotics within 48 hours were excluded, as they were unlikely to have serious infections.

We defined sepsis based on Sepsis-3 consensus guidelines² as a change in sequential [sepsis-related] organ failure assessment (SOFA) score ≥2 within the first 48 hours of ED presentation. The SOFA score is comprised of six variables representing different organ systems, each rated 0-4 based on the degree of dysfunction.² Patient vital signs, laboratory data, vasopressor medication doses, and ventilator settings were used to determine the exact timestamp at which each patient attained a change in SOFA score ≥2. Missing values were considered to be normal. To adjust for baseline organ dysfunction, SOFA elements associated with elevated bilirubin and/or creatinine were excluded for patients with chronic liver/kidney disease based on Elixhauser comorbidities.²⁷ We chose to focus on the first 48 hours in an attempt to capture patients with the most severe illnesses and the highest probability of true sepsis.

All primary and secondary International Classification of Diseases (ICD)-9/10 diagnosis codes were extracted from Clarity coding tables at the time of hospital discharge. Diagnosis codes signifying bacterial infection were grouped into the following categories based on type/location: pneumonia; bacteremia; urinary tract infection; and skin and soft tissue infection. All remaining diagnostic codes indicating bacterial infections at other sites were categorized as “Other”. If no codes indicating infection were present, patients were categorized as “None coded”. Patients with discharge diagnosis codes of “sepsis” were also identified. Dates and times of antibiotic administrations were obtained from the medications table. Time to first antibiotic was defined as the time in minutes from ED presentation to initiation of the first IV antibacterial medication. This variable was transformed using a natural log transformation based on best fit for normal distribution.

We limited our analyses to 8,974 patients who were diagnosed with sepsis as defined above and had either (1) ≥4 qualifying antibiotic days (QADs) or (2) an ICD-9/10 discharge diagnosis code of “sepsis” (Figure). QADs were defined based on the recent publication by Rhee et al. as having received four or more consecutive days of antibiotics, with the first dose given IV within 48 hours of presentation.²⁸ Patients who died or were discharged to hospice prior to the 4th QAD were also included. These additional parameters were added to increase specificity of the study sample for patients with true sepsis. Patients admitted to all levels of care (acute care, transitional care unit [TCU], intensive care unit [ICU]) and under all hospital services were included. There were no missing data for mortality, race, or language. We chose to focus on patients with sepsis in this initial study as this is a common diagnosis in hospital medicine that is enriched for high mortality.

Primary Outcome

The primary outcome of the study was inpatient mortality, which was obtained from the hospital encounters table in Clarity.

Primary Predictors

The primary predictor of interest was LEP. The encounter numbers from the dataset were used to link to self-reported demographic data, including “preferred language” and need for interpreter services. A manual chart review of 60 patients speaking the top six languages was conducted to verify the accuracy of the data on language and interpreter use (KNK). Defining the gold standard for LEP as having any chart note indicating non-English language and/or that an interpreter was used, the “interpreter needed” variable in Epic was found to have a positive predictive value for LEP of 100%. Therefore, patients in the study cohort were defined as having LEP if they met both of the following criteria: (1) a self-reported “preferred language” other than English and (2) having the “interpreter needed” variable indicating “yes”.

Covariate Data Collection

Additional data were obtained from the demographics tables, including age, race, sex, and insurance status. Race and ethnicity were combined into a single five-category variable including White, Asian, Black, Latino, and Other. This approach has been suggested as the best way to operationalize these variables²⁹ and has been utilized by similar studies in the literature.^9,14,15 We considered the Asian race to include all people of East Asian, Southeast Asian, or South Asian descent, which is consistent with the United States Census Bureau definition.³⁰ Patients identifying as Native Hawaiians/Pacific Islanders, Native Americans/Alaskan Natives, as well as those with unspecified race or ethnicity, were categorized as Other. Insurance status was categorized as Commercial, Medicare, Medicaid, or Other.

We estimated illness severity in several ways. First, the total qualifying SOFA score was calculated for each patient, which was defined as the total score achieved at the time that SOFA criteria were first met (≥2, within 48 hours). Second, we dichotomized patients based on whether they had received mechanical ventilation at any point during hospitalization. Finally, we used admission location as a surrogate marker for severity at the time of initial hospitalization.

To estimate the burden of baseline comorbidities, we calculated the van Walraven score (VWS),³¹ a validated modification of the Elixhauser Comorbidity Index.²⁷ This score conveys an estimated risk of in hospital death based on ICD-9/10 diagnosis codes for preexisting conditions, which ranges from <1% for the minimum score of –19 to >99% for the maximum score of 89.

Statistical Analyses

All statistical analyses were performed using Stata software version 15 (StataCorp LLC, College Station, Texas). Baseline demographics and patient characteristics were stratified by LEP. These were compared using two-sample t-tests or chi-squared tests of significance. Wilcoxon rank-sum tests were used for non-normally distributed variables. Inpatient mortality was compared across all races stratified by LEP using chi-squared tests of significance.

We fit a series of multivariable logistic regression models to examine the association between race and inpatient mortality adjusting for LEP and other patient/clinical characteristics. We first examined the unadjusted association between mortality and race; then adjusted for LEP alone; and finally adjusted for all covariates of interest, including LEP, age, sex, insurance status, year, admission level of care, VWS, total qualifying SOFA score, need for mechanical ventilation, site of infection, and time to first IV antibiotic. A subgroup analysis was also performed using the fully adjusted model restricted to patients who were mechanically ventilated. This population was selected because the patients (1) have among the highest severity of illness and (2) share a common barrier to communication, regardless of English proficiency.

Several potential interactions between LEP with other covariates were explored, including age, race, ICU admission level of care, and need for mechanical ventilation. Lastly, a mediation analysis was performed based on Baron & Kenny’s four-step model³² in order to calculate the proportion of the association between race and mortality explained by the proposed mediator (LEP).

To evaluate for the likelihood of residual confounding, we calculated an E-value, which is defined as the minimum strength of association that an unmeasured confounder would need to have with both the predictor and outcome variables, above and beyond the measured covariates, in order to fully explain away an observed predictor-outcome association.^33,34

We identified 8,974 patients hospitalized with sepsis based on the above inclusion criteria. This represented a medically complex, racially and linguistically diverse population (Table 1). The cohort was comprised of 24% Asian, 12% Black, and 11% Latino patients. Among those categorized as Other race, Native Americans/Alaskan Natives and Native Hawaiians/Pacific Islanders accounted for 4% (n = 31) and 21% (n = 159), respectively. A fifth of all patients had LEP (n = 1,716), 62% of whom were Asian (n = 1,064). Patients with LEP tended to be older, female, and to have a greater number of comorbid conditions (Table 1). The total qualifying SOFA score was also higher among patients with LEP (median 5; interquartile range [IQR]: 4-8 vs 5; IQR: 3-7; P <.001), though there was no association between LEP and mechanical ventilation (P = .22). The prevalence of LEP differed significantly across races, with 50% LEP among Asians, 32% among Latinos, 5% among White patients (P < .001). Only eight Black patients had LEP. More than 40 unique languages were represented in the cohort, with English, Cantonese, Spanish, Russian, and Mandarin accounting for ~95% (Appendix Table 1). Among Latino patients, 63% spoke English and 36% spoke Spanish.

In-hospital mortality was significantly higher among patients who had LEP (n = 268/1,716, 16%) compared to non-LEP patients (n = 678/7,258, 9%), with 80% greater unadjusted odds of mortality (OR 1.80; 95% CI: 1.54-2.09; P < .001). Notably we also found that Asian race was associated with a 1.57 unadjusted odds of mortality compared to White race (95% CI: 1.34-1.85; P < .001). Age, VWS, total qualifying SOFA score, mechanical ventilation, and admission level of care all exhibited a positive dose-response association with mortality (Appendix Table 2). In unadjusted analyses, there was no evidence of interaction between LEP and age (P = .38), LEP and race (P = .45), LEP and ICU admission level of care (P = .31), or LEP and mechanical ventilation (P = .19). Asian patients had the highest overall mortality (14% total, 17% with LEP). LEP was associated with increased unadjusted mortality among White, Asian, and Other races compared to their non-LEP counterparts (Appendix Figure 1). There was no significant difference in mortality between Latino patients with and without LEP. The sample size for Black patients with LEP (n = 8) was too small to draw conclusions about mortality.

Following multivariable logistic regression modeling for the association between race and mortality, we found that the increased odds of death among Asian patients was partially attenuated after adjusting for LEP (odds ratio [OR] 1.23, 95% CI: 1.02-1.48; P = .03; Table 2). Meanwhile, LEP was associated with a 1.66 odds of mortality (95% CI: 1.38-1.99; P < .001) after adjustment for race. In the full multivariable model adjusting for demographics and clinical characteristics, illness severity, and comorbidities, LEP was associated with a 31% increase in the odds of mortality compared to non-LEP (95% CI: 1.06-1.63; P = .02). In this model, the association between Asian race and mortality was now fully attenuated, with a point estimate near 1.0 (OR 0.98; 95% CI: 0.79-1.22; P = .87). Markers of illness severity, including total qualifying SOFA score (OR 1.23; 95% CI: 1.20-1.27; P < .001) and need for mechanical ventilation (OR 1.88; 95% CI: 1.52-2.33; P < .001), were both associated with greater odds of death. Based on a four-step mediation analysis, LEP was found to be a partial mediator to the association between Asian race and mortality (76% proportion explained). The E-value for the association between LEP and mortality was 1.95, with an E-value for the corresponding confidence interval of 1.29.

At a single US academic medical center serving a diverse population, we found that LEP was associated with sepsis mortality across all races except Black and Latino, conveying a 31% increase in the odds of death after adjusting for illness severity, comorbidities, and baseline characteristics. The higher mortality among Asian patients was largely mediated by LEP (76% proportion explained). While previous studies have variably found Black, Asian, Latino, and other non-White races/ethnicities to be at an increased risk of death from sepsis,^9-15 LEP has not been previously evaluated as a mediator of sepsis mortality. We were uniquely suited to uncover such an association due to the racial and linguistic diversity of our patient population. LEP has previously been implicated in poor health outcomes among hospitalized patients in general.^22-24 Future studies will be necessary to determine whether similar associations between LEP and mortality are observed among broader patient populations outside of sepsis.

There are a number of possible explanations for how LEP could mediate the association between race and mortality. First, LEP is known to be associated with greater difficulties in accessing medical care,²⁵ which could result in poorer baseline control of chronic comorbid conditions, fewer opportunities for preventive screening, and greater reluctance to seek medical attention when ill, theoretically leading to more severe presentations and worse outcomes. Indeed, LEP patients in our cohort had both a shorter median time to receiving their first antibiotic, as well as a higher total qualifying SOFA score, both of which may suggest more severe initial presentations. LEP is also known to contribute to, or exacerbate, the impact of low health literacy, which is itself associated with poor health.³⁵ Second, implicit biases may also have been present, as they are known to be common among healthcare providers and have been shown to negatively impact patient care.³⁶

Finally,it is possible that the association is related to the language barrier itself, which impacts providers’ ability to take an appropriate clinical history, and can lead to clinical errors or delays in care.³⁷ The fact that the association between LEP and mortality was eliminated when the analysis was restricted to mechanically ventilated patients seems to support this, since differences in language proficiency become irrelevant in this subgroup. While we are unable to comment on causality based on this observational study, we included a directed acyclic graph (DAG) in the supplemental materials, which shows one proposed model for describing these associations (Appendix Figure 2).

Assuming that the language barrier itself does, at least in part, drive the observed association, LEP represents a potentially modifiable risk factor that could be a target for quality improvement interventions. There is evidence that the use of medical interpreters among patients with LEP leads to greater satisfaction, fewer errors, and improved clinical outcomes;³⁸ however, several recent studies have documented underutilization of professional interpreter services, even when readily available.^39,40 At our institution, phone and video interpreter services are available 24/7 for approximately 150 languages. Due to limitations inherent to the EHR, we were unable to ascertain the extent to which these services were used in the present study. Heavy clinical workloads, connectivity issues, and missing or faulty equipment represent theoretical barriers to utilization of these services.

There are some limitations to our study. First, by utilizing a large database of electronic data, the quality of our analyses was reliant on the accuracy of the EHR. Demographic data such as language may have been subject to misclassification due to self-reporting. We attempted to minimize this by also including the need for interpreter services within the definition of LEP, which was validated by manual chart review. Second, generalizability is limited in this single-center study conducted at an institution with unique demographics, wherein nearly two-thirds of the LEP patients were Asian, and the Chinese-speaking population outnumbered those who speak Spanish.

Finally, the most important limitation to our study is the potential for residual confounding. While we attempted to mitigate this by adjusting for as many clinically relevant covariates as possible, there may still be unmeasured confounders to the association between LEP and mortality, such as access to outpatient care, functional status, interpreter use, and other markers of illness severity like the number and type of supportive therapies received. Based on our E-value calculations, with an observed OR of 1.31 for the association between LEP and mortality, an unmeasured confounder with an OR of 1.95 would fully explain away this association, while an OR of 1.29 would shift the confidence interval to include the null. These values suggest at least some risk of residual confounding. The fact that our fully adjusted model included multiple covariates, including several markers of illness severity, does somewhat lessen the likelihood of a confounder achieving these values, since they represent the minimum strength of an unmeasured confounder above and beyond the measured covariates. Regardless, the finding that patients with LEP are more likely to die from sepsis remains an important one, recognizing the need for further studies including multicenter investigations.

In this study, we showed that LEP was associated with sepsis mortality across nearly all races in our cohort. While Asian race was associated with a higher unadjusted odds of death compared to White race, this was attenuated after adjusting for LEP. This may suggest that some of the racial disparities in sepsis identified in prior studies were in fact mediated by language proficiency. Further studies will be required to explore the causal nature of this novel association. If modifiable factors are identified, this could represent a potential target for future quality improvement initiatives aimed at improving sepsis outcomes.

Disclaimer

The contents are solely the responsibility of the authors and do not necessarily represent the official views of the University of California, San Francisco or the National Institutes of Health.

Sepsis is defined as a life-threatening organ dysfunction that occurs in response to systemic infection.^1,2 It is frequently fatal, common in hospital medicine, and a leading contributor to critical illness, morbidity, and healthcare expenditures.^2-5 While sepsis care and outcomes have improved in the past decade,^6,7 inpatient mortality remains high.⁸

A number of studies have sought to determine whether race plays a role in sepsis mortality. While Black patients with sepsis have frequently been identified as having the highest rates of death,^9-14 similar observations have been made for most non-White races/ethnicities.^13-15 Studies have also demonstrated higher rates of hospital-acquired infections among Asian and Latino patients.¹⁶

There are several possible explanations for why racial minorities experience disparate outcomes in sepsis, including access to care, comorbidities, implicit biases, and biological or environmental factors,^17-20 as well as characteristics of hospitals most likely to care for racial minorities.^13,15,21 One explanation that has not been explored is that racial disparities in sepsis are mediated by language. Limited English proficiency (LEP) has previously been associated with increased rates of adverse hospital events,²² longer length of stay,²³ and greater likelihood of readmission.²⁴ LEP has also been shown to represent a significant barrier to accessing healthcare and preventive screening.²⁵ The role of LEP in sepsis mortality, however, has yet to be examined.

The diverse patient population at the University of California, San Francisco (UCSF) provides a unique opportunity to build upon existing literature by further exploring racial differences in sepsis, specifically by investigating the role of LEP. The objective of this study was to determine the association between LEP and inpatient mortality among adults hospitalized with sepsis.

Setting

The study was conducted at the University of California, San Francisco, California (UCSF), an 800-bed tertiary care, academic medical center. It was approved by the UCSF Institutional Review Board with waiver of informed consent. UCSF cares for a population of patients who are racially and linguistically diverse, with high proportions of patients of East Asian descent and with LEP. According to recent United States census estimates, more than half of San Francisco County residents identify as non-White (35% Asians, 15% Hispanic/Latino, 6% Black), and 44% report speaking a language other than English at home.²⁶

Study Population and Data Collection

The UCSF Medical Center uses the electronic health record (EHR) Epic (Epic 2017, Epic Systems Corporation, Verona, Wisconsin). We obtained computerized EHR data from Clarity, the relational database that stores Epic’s inpatient data in thousands of tables. We identified all patients ≥18 years of age presenting to the emergency department (ED) between June 1, 2012 and December 31, 2016 with suspected serious infection, defined as having blood cultures ordered within 72 hours of ED presentation (N = 25,441). Patients who did not receive at least two doses of intravenous (IV) antibiotics within 48 hours were excluded, as they were unlikely to have serious infections.

We defined sepsis based on Sepsis-3 consensus guidelines² as a change in sequential [sepsis-related] organ failure assessment (SOFA) score ≥2 within the first 48 hours of ED presentation. The SOFA score is comprised of six variables representing different organ systems, each rated 0-4 based on the degree of dysfunction.² Patient vital signs, laboratory data, vasopressor medication doses, and ventilator settings were used to determine the exact timestamp at which each patient attained a change in SOFA score ≥2. Missing values were considered to be normal. To adjust for baseline organ dysfunction, SOFA elements associated with elevated bilirubin and/or creatinine were excluded for patients with chronic liver/kidney disease based on Elixhauser comorbidities.²⁷ We chose to focus on the first 48 hours in an attempt to capture patients with the most severe illnesses and the highest probability of true sepsis.

All primary and secondary International Classification of Diseases (ICD)-9/10 diagnosis codes were extracted from Clarity coding tables at the time of hospital discharge. Diagnosis codes signifying bacterial infection were grouped into the following categories based on type/location: pneumonia; bacteremia; urinary tract infection; and skin and soft tissue infection. All remaining diagnostic codes indicating bacterial infections at other sites were categorized as “Other”. If no codes indicating infection were present, patients were categorized as “None coded”. Patients with discharge diagnosis codes of “sepsis” were also identified. Dates and times of antibiotic administrations were obtained from the medications table. Time to first antibiotic was defined as the time in minutes from ED presentation to initiation of the first IV antibacterial medication. This variable was transformed using a natural log transformation based on best fit for normal distribution.

We limited our analyses to 8,974 patients who were diagnosed with sepsis as defined above and had either (1) ≥4 qualifying antibiotic days (QADs) or (2) an ICD-9/10 discharge diagnosis code of “sepsis” (Figure). QADs were defined based on the recent publication by Rhee et al. as having received four or more consecutive days of antibiotics, with the first dose given IV within 48 hours of presentation.²⁸ Patients who died or were discharged to hospice prior to the 4th QAD were also included. These additional parameters were added to increase specificity of the study sample for patients with true sepsis. Patients admitted to all levels of care (acute care, transitional care unit [TCU], intensive care unit [ICU]) and under all hospital services were included. There were no missing data for mortality, race, or language. We chose to focus on patients with sepsis in this initial study as this is a common diagnosis in hospital medicine that is enriched for high mortality.

Primary Outcome

The primary outcome of the study was inpatient mortality, which was obtained from the hospital encounters table in Clarity.

Primary Predictors

The primary predictor of interest was LEP. The encounter numbers from the dataset were used to link to self-reported demographic data, including “preferred language” and need for interpreter services. A manual chart review of 60 patients speaking the top six languages was conducted to verify the accuracy of the data on language and interpreter use (KNK). Defining the gold standard for LEP as having any chart note indicating non-English language and/or that an interpreter was used, the “interpreter needed” variable in Epic was found to have a positive predictive value for LEP of 100%. Therefore, patients in the study cohort were defined as having LEP if they met both of the following criteria: (1) a self-reported “preferred language” other than English and (2) having the “interpreter needed” variable indicating “yes”.

Covariate Data Collection

Additional data were obtained from the demographics tables, including age, race, sex, and insurance status. Race and ethnicity were combined into a single five-category variable including White, Asian, Black, Latino, and Other. This approach has been suggested as the best way to operationalize these variables²⁹ and has been utilized by similar studies in the literature.^9,14,15 We considered the Asian race to include all people of East Asian, Southeast Asian, or South Asian descent, which is consistent with the United States Census Bureau definition.³⁰ Patients identifying as Native Hawaiians/Pacific Islanders, Native Americans/Alaskan Natives, as well as those with unspecified race or ethnicity, were categorized as Other. Insurance status was categorized as Commercial, Medicare, Medicaid, or Other.

We estimated illness severity in several ways. First, the total qualifying SOFA score was calculated for each patient, which was defined as the total score achieved at the time that SOFA criteria were first met (≥2, within 48 hours). Second, we dichotomized patients based on whether they had received mechanical ventilation at any point during hospitalization. Finally, we used admission location as a surrogate marker for severity at the time of initial hospitalization.

To estimate the burden of baseline comorbidities, we calculated the van Walraven score (VWS),³¹ a validated modification of the Elixhauser Comorbidity Index.²⁷ This score conveys an estimated risk of in hospital death based on ICD-9/10 diagnosis codes for preexisting conditions, which ranges from <1% for the minimum score of –19 to >99% for the maximum score of 89.

Statistical Analyses

All statistical analyses were performed using Stata software version 15 (StataCorp LLC, College Station, Texas). Baseline demographics and patient characteristics were stratified by LEP. These were compared using two-sample t-tests or chi-squared tests of significance. Wilcoxon rank-sum tests were used for non-normally distributed variables. Inpatient mortality was compared across all races stratified by LEP using chi-squared tests of significance.

We fit a series of multivariable logistic regression models to examine the association between race and inpatient mortality adjusting for LEP and other patient/clinical characteristics. We first examined the unadjusted association between mortality and race; then adjusted for LEP alone; and finally adjusted for all covariates of interest, including LEP, age, sex, insurance status, year, admission level of care, VWS, total qualifying SOFA score, need for mechanical ventilation, site of infection, and time to first IV antibiotic. A subgroup analysis was also performed using the fully adjusted model restricted to patients who were mechanically ventilated. This population was selected because the patients (1) have among the highest severity of illness and (2) share a common barrier to communication, regardless of English proficiency.

Several potential interactions between LEP with other covariates were explored, including age, race, ICU admission level of care, and need for mechanical ventilation. Lastly, a mediation analysis was performed based on Baron & Kenny’s four-step model³² in order to calculate the proportion of the association between race and mortality explained by the proposed mediator (LEP).

To evaluate for the likelihood of residual confounding, we calculated an E-value, which is defined as the minimum strength of association that an unmeasured confounder would need to have with both the predictor and outcome variables, above and beyond the measured covariates, in order to fully explain away an observed predictor-outcome association.^33,34

We identified 8,974 patients hospitalized with sepsis based on the above inclusion criteria. This represented a medically complex, racially and linguistically diverse population (Table 1). The cohort was comprised of 24% Asian, 12% Black, and 11% Latino patients. Among those categorized as Other race, Native Americans/Alaskan Natives and Native Hawaiians/Pacific Islanders accounted for 4% (n = 31) and 21% (n = 159), respectively. A fifth of all patients had LEP (n = 1,716), 62% of whom were Asian (n = 1,064). Patients with LEP tended to be older, female, and to have a greater number of comorbid conditions (Table 1). The total qualifying SOFA score was also higher among patients with LEP (median 5; interquartile range [IQR]: 4-8 vs 5; IQR: 3-7; P <.001), though there was no association between LEP and mechanical ventilation (P = .22). The prevalence of LEP differed significantly across races, with 50% LEP among Asians, 32% among Latinos, 5% among White patients (P < .001). Only eight Black patients had LEP. More than 40 unique languages were represented in the cohort, with English, Cantonese, Spanish, Russian, and Mandarin accounting for ~95% (Appendix Table 1). Among Latino patients, 63% spoke English and 36% spoke Spanish.

In-hospital mortality was significantly higher among patients who had LEP (n = 268/1,716, 16%) compared to non-LEP patients (n = 678/7,258, 9%), with 80% greater unadjusted odds of mortality (OR 1.80; 95% CI: 1.54-2.09; P < .001). Notably we also found that Asian race was associated with a 1.57 unadjusted odds of mortality compared to White race (95% CI: 1.34-1.85; P < .001). Age, VWS, total qualifying SOFA score, mechanical ventilation, and admission level of care all exhibited a positive dose-response association with mortality (Appendix Table 2). In unadjusted analyses, there was no evidence of interaction between LEP and age (P = .38), LEP and race (P = .45), LEP and ICU admission level of care (P = .31), or LEP and mechanical ventilation (P = .19). Asian patients had the highest overall mortality (14% total, 17% with LEP). LEP was associated with increased unadjusted mortality among White, Asian, and Other races compared to their non-LEP counterparts (Appendix Figure 1). There was no significant difference in mortality between Latino patients with and without LEP. The sample size for Black patients with LEP (n = 8) was too small to draw conclusions about mortality.

Following multivariable logistic regression modeling for the association between race and mortality, we found that the increased odds of death among Asian patients was partially attenuated after adjusting for LEP (odds ratio [OR] 1.23, 95% CI: 1.02-1.48; P = .03; Table 2). Meanwhile, LEP was associated with a 1.66 odds of mortality (95% CI: 1.38-1.99; P < .001) after adjustment for race. In the full multivariable model adjusting for demographics and clinical characteristics, illness severity, and comorbidities, LEP was associated with a 31% increase in the odds of mortality compared to non-LEP (95% CI: 1.06-1.63; P = .02). In this model, the association between Asian race and mortality was now fully attenuated, with a point estimate near 1.0 (OR 0.98; 95% CI: 0.79-1.22; P = .87). Markers of illness severity, including total qualifying SOFA score (OR 1.23; 95% CI: 1.20-1.27; P < .001) and need for mechanical ventilation (OR 1.88; 95% CI: 1.52-2.33; P < .001), were both associated with greater odds of death. Based on a four-step mediation analysis, LEP was found to be a partial mediator to the association between Asian race and mortality (76% proportion explained). The E-value for the association between LEP and mortality was 1.95, with an E-value for the corresponding confidence interval of 1.29.

At a single US academic medical center serving a diverse population, we found that LEP was associated with sepsis mortality across all races except Black and Latino, conveying a 31% increase in the odds of death after adjusting for illness severity, comorbidities, and baseline characteristics. The higher mortality among Asian patients was largely mediated by LEP (76% proportion explained). While previous studies have variably found Black, Asian, Latino, and other non-White races/ethnicities to be at an increased risk of death from sepsis,^9-15 LEP has not been previously evaluated as a mediator of sepsis mortality. We were uniquely suited to uncover such an association due to the racial and linguistic diversity of our patient population. LEP has previously been implicated in poor health outcomes among hospitalized patients in general.^22-24 Future studies will be necessary to determine whether similar associations between LEP and mortality are observed among broader patient populations outside of sepsis.

There are a number of possible explanations for how LEP could mediate the association between race and mortality. First, LEP is known to be associated with greater difficulties in accessing medical care,²⁵ which could result in poorer baseline control of chronic comorbid conditions, fewer opportunities for preventive screening, and greater reluctance to seek medical attention when ill, theoretically leading to more severe presentations and worse outcomes. Indeed, LEP patients in our cohort had both a shorter median time to receiving their first antibiotic, as well as a higher total qualifying SOFA score, both of which may suggest more severe initial presentations. LEP is also known to contribute to, or exacerbate, the impact of low health literacy, which is itself associated with poor health.³⁵ Second, implicit biases may also have been present, as they are known to be common among healthcare providers and have been shown to negatively impact patient care.³⁶

Finally,it is possible that the association is related to the language barrier itself, which impacts providers’ ability to take an appropriate clinical history, and can lead to clinical errors or delays in care.³⁷ The fact that the association between LEP and mortality was eliminated when the analysis was restricted to mechanically ventilated patients seems to support this, since differences in language proficiency become irrelevant in this subgroup. While we are unable to comment on causality based on this observational study, we included a directed acyclic graph (DAG) in the supplemental materials, which shows one proposed model for describing these associations (Appendix Figure 2).

Assuming that the language barrier itself does, at least in part, drive the observed association, LEP represents a potentially modifiable risk factor that could be a target for quality improvement interventions. There is evidence that the use of medical interpreters among patients with LEP leads to greater satisfaction, fewer errors, and improved clinical outcomes;³⁸ however, several recent studies have documented underutilization of professional interpreter services, even when readily available.^39,40 At our institution, phone and video interpreter services are available 24/7 for approximately 150 languages. Due to limitations inherent to the EHR, we were unable to ascertain the extent to which these services were used in the present study. Heavy clinical workloads, connectivity issues, and missing or faulty equipment represent theoretical barriers to utilization of these services.

There are some limitations to our study. First, by utilizing a large database of electronic data, the quality of our analyses was reliant on the accuracy of the EHR. Demographic data such as language may have been subject to misclassification due to self-reporting. We attempted to minimize this by also including the need for interpreter services within the definition of LEP, which was validated by manual chart review. Second, generalizability is limited in this single-center study conducted at an institution with unique demographics, wherein nearly two-thirds of the LEP patients were Asian, and the Chinese-speaking population outnumbered those who speak Spanish.

Finally, the most important limitation to our study is the potential for residual confounding. While we attempted to mitigate this by adjusting for as many clinically relevant covariates as possible, there may still be unmeasured confounders to the association between LEP and mortality, such as access to outpatient care, functional status, interpreter use, and other markers of illness severity like the number and type of supportive therapies received. Based on our E-value calculations, with an observed OR of 1.31 for the association between LEP and mortality, an unmeasured confounder with an OR of 1.95 would fully explain away this association, while an OR of 1.29 would shift the confidence interval to include the null. These values suggest at least some risk of residual confounding. The fact that our fully adjusted model included multiple covariates, including several markers of illness severity, does somewhat lessen the likelihood of a confounder achieving these values, since they represent the minimum strength of an unmeasured confounder above and beyond the measured covariates. Regardless, the finding that patients with LEP are more likely to die from sepsis remains an important one, recognizing the need for further studies including multicenter investigations.

In this study, we showed that LEP was associated with sepsis mortality across nearly all races in our cohort. While Asian race was associated with a higher unadjusted odds of death compared to White race, this was attenuated after adjusting for LEP. This may suggest that some of the racial disparities in sepsis identified in prior studies were in fact mediated by language proficiency. Further studies will be required to explore the causal nature of this novel association. If modifiable factors are identified, this could represent a potential target for future quality improvement initiatives aimed at improving sepsis outcomes.

Disclaimer

The contents are solely the responsibility of the authors and do not necessarily represent the official views of the University of California, San Francisco or the National Institutes of Health.

Sepsis is defined as a life-threatening organ dysfunction that occurs in response to systemic infection.^1,2 It is frequently fatal, common in hospital medicine, and a leading contributor to critical illness, morbidity, and healthcare expenditures.^2-5 While sepsis care and outcomes have improved in the past decade,^6,7 inpatient mortality remains high.⁸

A number of studies have sought to determine whether race plays a role in sepsis mortality. While Black patients with sepsis have frequently been identified as having the highest rates of death,^9-14 similar observations have been made for most non-White races/ethnicities.^13-15 Studies have also demonstrated higher rates of hospital-acquired infections among Asian and Latino patients.¹⁶

There are several possible explanations for why racial minorities experience disparate outcomes in sepsis, including access to care, comorbidities, implicit biases, and biological or environmental factors,^17-20 as well as characteristics of hospitals most likely to care for racial minorities.^13,15,21 One explanation that has not been explored is that racial disparities in sepsis are mediated by language. Limited English proficiency (LEP) has previously been associated with increased rates of adverse hospital events,²² longer length of stay,²³ and greater likelihood of readmission.²⁴ LEP has also been shown to represent a significant barrier to accessing healthcare and preventive screening.²⁵ The role of LEP in sepsis mortality, however, has yet to be examined.

The diverse patient population at the University of California, San Francisco (UCSF) provides a unique opportunity to build upon existing literature by further exploring racial differences in sepsis, specifically by investigating the role of LEP. The objective of this study was to determine the association between LEP and inpatient mortality among adults hospitalized with sepsis.

Setting

The study was conducted at the University of California, San Francisco, California (UCSF), an 800-bed tertiary care, academic medical center. It was approved by the UCSF Institutional Review Board with waiver of informed consent. UCSF cares for a population of patients who are racially and linguistically diverse, with high proportions of patients of East Asian descent and with LEP. According to recent United States census estimates, more than half of San Francisco County residents identify as non-White (35% Asians, 15% Hispanic/Latino, 6% Black), and 44% report speaking a language other than English at home.²⁶

Study Population and Data Collection

The UCSF Medical Center uses the electronic health record (EHR) Epic (Epic 2017, Epic Systems Corporation, Verona, Wisconsin). We obtained computerized EHR data from Clarity, the relational database that stores Epic’s inpatient data in thousands of tables. We identified all patients ≥18 years of age presenting to the emergency department (ED) between June 1, 2012 and December 31, 2016 with suspected serious infection, defined as having blood cultures ordered within 72 hours of ED presentation (N = 25,441). Patients who did not receive at least two doses of intravenous (IV) antibiotics within 48 hours were excluded, as they were unlikely to have serious infections.

We defined sepsis based on Sepsis-3 consensus guidelines² as a change in sequential [sepsis-related] organ failure assessment (SOFA) score ≥2 within the first 48 hours of ED presentation. The SOFA score is comprised of six variables representing different organ systems, each rated 0-4 based on the degree of dysfunction.² Patient vital signs, laboratory data, vasopressor medication doses, and ventilator settings were used to determine the exact timestamp at which each patient attained a change in SOFA score ≥2. Missing values were considered to be normal. To adjust for baseline organ dysfunction, SOFA elements associated with elevated bilirubin and/or creatinine were excluded for patients with chronic liver/kidney disease based on Elixhauser comorbidities.²⁷ We chose to focus on the first 48 hours in an attempt to capture patients with the most severe illnesses and the highest probability of true sepsis.

All primary and secondary International Classification of Diseases (ICD)-9/10 diagnosis codes were extracted from Clarity coding tables at the time of hospital discharge. Diagnosis codes signifying bacterial infection were grouped into the following categories based on type/location: pneumonia; bacteremia; urinary tract infection; and skin and soft tissue infection. All remaining diagnostic codes indicating bacterial infections at other sites were categorized as “Other”. If no codes indicating infection were present, patients were categorized as “None coded”. Patients with discharge diagnosis codes of “sepsis” were also identified. Dates and times of antibiotic administrations were obtained from the medications table. Time to first antibiotic was defined as the time in minutes from ED presentation to initiation of the first IV antibacterial medication. This variable was transformed using a natural log transformation based on best fit for normal distribution.

We limited our analyses to 8,974 patients who were diagnosed with sepsis as defined above and had either (1) ≥4 qualifying antibiotic days (QADs) or (2) an ICD-9/10 discharge diagnosis code of “sepsis” (Figure). QADs were defined based on the recent publication by Rhee et al. as having received four or more consecutive days of antibiotics, with the first dose given IV within 48 hours of presentation.²⁸ Patients who died or were discharged to hospice prior to the 4th QAD were also included. These additional parameters were added to increase specificity of the study sample for patients with true sepsis. Patients admitted to all levels of care (acute care, transitional care unit [TCU], intensive care unit [ICU]) and under all hospital services were included. There were no missing data for mortality, race, or language. We chose to focus on patients with sepsis in this initial study as this is a common diagnosis in hospital medicine that is enriched for high mortality.

Primary Outcome

The primary outcome of the study was inpatient mortality, which was obtained from the hospital encounters table in Clarity.

Primary Predictors

The primary predictor of interest was LEP. The encounter numbers from the dataset were used to link to self-reported demographic data, including “preferred language” and need for interpreter services. A manual chart review of 60 patients speaking the top six languages was conducted to verify the accuracy of the data on language and interpreter use (KNK). Defining the gold standard for LEP as having any chart note indicating non-English language and/or that an interpreter was used, the “interpreter needed” variable in Epic was found to have a positive predictive value for LEP of 100%. Therefore, patients in the study cohort were defined as having LEP if they met both of the following criteria: (1) a self-reported “preferred language” other than English and (2) having the “interpreter needed” variable indicating “yes”.

Covariate Data Collection

Additional data were obtained from the demographics tables, including age, race, sex, and insurance status. Race and ethnicity were combined into a single five-category variable including White, Asian, Black, Latino, and Other. This approach has been suggested as the best way to operationalize these variables²⁹ and has been utilized by similar studies in the literature.^9,14,15 We considered the Asian race to include all people of East Asian, Southeast Asian, or South Asian descent, which is consistent with the United States Census Bureau definition.³⁰ Patients identifying as Native Hawaiians/Pacific Islanders, Native Americans/Alaskan Natives, as well as those with unspecified race or ethnicity, were categorized as Other. Insurance status was categorized as Commercial, Medicare, Medicaid, or Other.

We estimated illness severity in several ways. First, the total qualifying SOFA score was calculated for each patient, which was defined as the total score achieved at the time that SOFA criteria were first met (≥2, within 48 hours). Second, we dichotomized patients based on whether they had received mechanical ventilation at any point during hospitalization. Finally, we used admission location as a surrogate marker for severity at the time of initial hospitalization.

To estimate the burden of baseline comorbidities, we calculated the van Walraven score (VWS),³¹ a validated modification of the Elixhauser Comorbidity Index.²⁷ This score conveys an estimated risk of in hospital death based on ICD-9/10 diagnosis codes for preexisting conditions, which ranges from <1% for the minimum score of –19 to >99% for the maximum score of 89.

Statistical Analyses

All statistical analyses were performed using Stata software version 15 (StataCorp LLC, College Station, Texas). Baseline demographics and patient characteristics were stratified by LEP. These were compared using two-sample t-tests or chi-squared tests of significance. Wilcoxon rank-sum tests were used for non-normally distributed variables. Inpatient mortality was compared across all races stratified by LEP using chi-squared tests of significance.

We fit a series of multivariable logistic regression models to examine the association between race and inpatient mortality adjusting for LEP and other patient/clinical characteristics. We first examined the unadjusted association between mortality and race; then adjusted for LEP alone; and finally adjusted for all covariates of interest, including LEP, age, sex, insurance status, year, admission level of care, VWS, total qualifying SOFA score, need for mechanical ventilation, site of infection, and time to first IV antibiotic. A subgroup analysis was also performed using the fully adjusted model restricted to patients who were mechanically ventilated. This population was selected because the patients (1) have among the highest severity of illness and (2) share a common barrier to communication, regardless of English proficiency.

Several potential interactions between LEP with other covariates were explored, including age, race, ICU admission level of care, and need for mechanical ventilation. Lastly, a mediation analysis was performed based on Baron & Kenny’s four-step model³² in order to calculate the proportion of the association between race and mortality explained by the proposed mediator (LEP).

To evaluate for the likelihood of residual confounding, we calculated an E-value, which is defined as the minimum strength of association that an unmeasured confounder would need to have with both the predictor and outcome variables, above and beyond the measured covariates, in order to fully explain away an observed predictor-outcome association.^33,34

We identified 8,974 patients hospitalized with sepsis based on the above inclusion criteria. This represented a medically complex, racially and linguistically diverse population (Table 1). The cohort was comprised of 24% Asian, 12% Black, and 11% Latino patients. Among those categorized as Other race, Native Americans/Alaskan Natives and Native Hawaiians/Pacific Islanders accounted for 4% (n = 31) and 21% (n = 159), respectively. A fifth of all patients had LEP (n = 1,716), 62% of whom were Asian (n = 1,064). Patients with LEP tended to be older, female, and to have a greater number of comorbid conditions (Table 1). The total qualifying SOFA score was also higher among patients with LEP (median 5; interquartile range [IQR]: 4-8 vs 5; IQR: 3-7; P <.001), though there was no association between LEP and mechanical ventilation (P = .22). The prevalence of LEP differed significantly across races, with 50% LEP among Asians, 32% among Latinos, 5% among White patients (P < .001). Only eight Black patients had LEP. More than 40 unique languages were represented in the cohort, with English, Cantonese, Spanish, Russian, and Mandarin accounting for ~95% (Appendix Table 1). Among Latino patients, 63% spoke English and 36% spoke Spanish.

In-hospital mortality was significantly higher among patients who had LEP (n = 268/1,716, 16%) compared to non-LEP patients (n = 678/7,258, 9%), with 80% greater unadjusted odds of mortality (OR 1.80; 95% CI: 1.54-2.09; P < .001). Notably we also found that Asian race was associated with a 1.57 unadjusted odds of mortality compared to White race (95% CI: 1.34-1.85; P < .001). Age, VWS, total qualifying SOFA score, mechanical ventilation, and admission level of care all exhibited a positive dose-response association with mortality (Appendix Table 2). In unadjusted analyses, there was no evidence of interaction between LEP and age (P = .38), LEP and race (P = .45), LEP and ICU admission level of care (P = .31), or LEP and mechanical ventilation (P = .19). Asian patients had the highest overall mortality (14% total, 17% with LEP). LEP was associated with increased unadjusted mortality among White, Asian, and Other races compared to their non-LEP counterparts (Appendix Figure 1). There was no significant difference in mortality between Latino patients with and without LEP. The sample size for Black patients with LEP (n = 8) was too small to draw conclusions about mortality.

Following multivariable logistic regression modeling for the association between race and mortality, we found that the increased odds of death among Asian patients was partially attenuated after adjusting for LEP (odds ratio [OR] 1.23, 95% CI: 1.02-1.48; P = .03; Table 2). Meanwhile, LEP was associated with a 1.66 odds of mortality (95% CI: 1.38-1.99; P < .001) after adjustment for race. In the full multivariable model adjusting for demographics and clinical characteristics, illness severity, and comorbidities, LEP was associated with a 31% increase in the odds of mortality compared to non-LEP (95% CI: 1.06-1.63; P = .02). In this model, the association between Asian race and mortality was now fully attenuated, with a point estimate near 1.0 (OR 0.98; 95% CI: 0.79-1.22; P = .87). Markers of illness severity, including total qualifying SOFA score (OR 1.23; 95% CI: 1.20-1.27; P < .001) and need for mechanical ventilation (OR 1.88; 95% CI: 1.52-2.33; P < .001), were both associated with greater odds of death. Based on a four-step mediation analysis, LEP was found to be a partial mediator to the association between Asian race and mortality (76% proportion explained). The E-value for the association between LEP and mortality was 1.95, with an E-value for the corresponding confidence interval of 1.29.

At a single US academic medical center serving a diverse population, we found that LEP was associated with sepsis mortality across all races except Black and Latino, conveying a 31% increase in the odds of death after adjusting for illness severity, comorbidities, and baseline characteristics. The higher mortality among Asian patients was largely mediated by LEP (76% proportion explained). While previous studies have variably found Black, Asian, Latino, and other non-White races/ethnicities to be at an increased risk of death from sepsis,^9-15 LEP has not been previously evaluated as a mediator of sepsis mortality. We were uniquely suited to uncover such an association due to the racial and linguistic diversity of our patient population. LEP has previously been implicated in poor health outcomes among hospitalized patients in general.^22-24 Future studies will be necessary to determine whether similar associations between LEP and mortality are observed among broader patient populations outside of sepsis.

There are a number of possible explanations for how LEP could mediate the association between race and mortality. First, LEP is known to be associated with greater difficulties in accessing medical care,²⁵ which could result in poorer baseline control of chronic comorbid conditions, fewer opportunities for preventive screening, and greater reluctance to seek medical attention when ill, theoretically leading to more severe presentations and worse outcomes. Indeed, LEP patients in our cohort had both a shorter median time to receiving their first antibiotic, as well as a higher total qualifying SOFA score, both of which may suggest more severe initial presentations. LEP is also known to contribute to, or exacerbate, the impact of low health literacy, which is itself associated with poor health.³⁵ Second, implicit biases may also have been present, as they are known to be common among healthcare providers and have been shown to negatively impact patient care.³⁶

Finally,it is possible that the association is related to the language barrier itself, which impacts providers’ ability to take an appropriate clinical history, and can lead to clinical errors or delays in care.³⁷ The fact that the association between LEP and mortality was eliminated when the analysis was restricted to mechanically ventilated patients seems to support this, since differences in language proficiency become irrelevant in this subgroup. While we are unable to comment on causality based on this observational study, we included a directed acyclic graph (DAG) in the supplemental materials, which shows one proposed model for describing these associations (Appendix Figure 2).

Assuming that the language barrier itself does, at least in part, drive the observed association, LEP represents a potentially modifiable risk factor that could be a target for quality improvement interventions. There is evidence that the use of medical interpreters among patients with LEP leads to greater satisfaction, fewer errors, and improved clinical outcomes;³⁸ however, several recent studies have documented underutilization of professional interpreter services, even when readily available.^39,40 At our institution, phone and video interpreter services are available 24/7 for approximately 150 languages. Due to limitations inherent to the EHR, we were unable to ascertain the extent to which these services were used in the present study. Heavy clinical workloads, connectivity issues, and missing or faulty equipment represent theoretical barriers to utilization of these services.

There are some limitations to our study. First, by utilizing a large database of electronic data, the quality of our analyses was reliant on the accuracy of the EHR. Demographic data such as language may have been subject to misclassification due to self-reporting. We attempted to minimize this by also including the need for interpreter services within the definition of LEP, which was validated by manual chart review. Second, generalizability is limited in this single-center study conducted at an institution with unique demographics, wherein nearly two-thirds of the LEP patients were Asian, and the Chinese-speaking population outnumbered those who speak Spanish.

Finally, the most important limitation to our study is the potential for residual confounding. While we attempted to mitigate this by adjusting for as many clinically relevant covariates as possible, there may still be unmeasured confounders to the association between LEP and mortality, such as access to outpatient care, functional status, interpreter use, and other markers of illness severity like the number and type of supportive therapies received. Based on our E-value calculations, with an observed OR of 1.31 for the association between LEP and mortality, an unmeasured confounder with an OR of 1.95 would fully explain away this association, while an OR of 1.29 would shift the confidence interval to include the null. These values suggest at least some risk of residual confounding. The fact that our fully adjusted model included multiple covariates, including several markers of illness severity, does somewhat lessen the likelihood of a confounder achieving these values, since they represent the minimum strength of an unmeasured confounder above and beyond the measured covariates. Regardless, the finding that patients with LEP are more likely to die from sepsis remains an important one, recognizing the need for further studies including multicenter investigations.

In this study, we showed that LEP was associated with sepsis mortality across nearly all races in our cohort. While Asian race was associated with a higher unadjusted odds of death compared to White race, this was attenuated after adjusting for LEP. This may suggest that some of the racial disparities in sepsis identified in prior studies were in fact mediated by language proficiency. Further studies will be required to explore the causal nature of this novel association. If modifiable factors are identified, this could represent a potential target for future quality improvement initiatives aimed at improving sepsis outcomes.

Disclaimer

The contents are solely the responsibility of the authors and do not necessarily represent the official views of the University of California, San Francisco or the National Institutes of Health.