Original Research

Inadequate bowel preparation (IBP) at the time of inpatient colonoscopy is common and associated with increased length of stay and cost of care.¹ The factors that contribute to IBP can be categorized into those that are modifiable and those that are nonmodifiable. While many factors have been associated with IBP, studies have been limited by small sample size or have combined inpatient/outpatient populations, thus limiting generalizability.^1-5 Moreover, most factors associated with IBP, such as socioeconomic status, male gender, increased age, and comorbidities, are nonmodifiable. No studies have explicitly focused on modifiable risk factors, such as medication use, colonoscopy timing, or assessed the potential impact of modifying these factors.

In a large, multihospital system, we examine the frequency of IBP among inpatients undergoing colonoscopy along with factors associated with IBP. We attempted to identify modifiable risk factors that were associated with IBP.

Potential Predictors of IBP

Demographic data such as patient age, gender, ethnicity, body mass index (BMI), and insurance/payor status were obtained from the electronic health record (EHR). International Classification of Disease 9^th and 10^th revision, Clinical Modifications (ICD-9/10-CM) codes were used to obtain patient comorbidities including diabetes, coronary artery disease, heart failure, cirrhosis, gastroparesis, hypothyroidism, inflammatory bowel disease, constipation, stroke, dementia, dysphagia, and nausea/vomiting. Use of opioid medications within three days before colonoscopy was extracted from the medication administration record. These variables were chosen as biologically plausible modifiers of bowel preparation or had previously been assessed in the literature.^1-6 The name and volume, classified as 4 L (GoLytely^®) and < 4 liters (MoviPrep^®) of bowel preparation, time of day when colonoscopy was performed, solid diet the day prior to colonoscopy, type of sedation used (conscious sedation or general anesthesia), and total colonoscopy time (defined as the time from scope insertion to removal) was recorded. Hospitalization-related variables, including the number of hospitalizations in the year before the current hospitalization, the year in which the colonoscopy was performed, and the number of days from admission to colonoscopy, were also obtained from the EHR.

Outcome Measures

An internally validated natural language algorithm, using Structured Queried Language was used to search through colonoscopy reports to identify adequacy of bowel preparation. ProVation^® software allows the gastroenterologist to use some terms to describe bowel preparation in a drop-down menu format. In addition to the Aronchik scale (which allows the gastroenterologist to rate bowel preparation on a five-point scale: “excellent,” “good,” “fair,” “poor,” and “inadequate”) it also allows the provider to use terms such as “adequate” or “adequate to detect polyps >5 mm” as well as “unsatisfactory.”⁷ Mirroring prior literature, bowel preparation quality was classified into “adequate” and “inadequate”; “good” and “excellent” on the Aronchik scale were categorized as adequate as was the term “adequate” in any form; “fair,” “poor,” or “inadequate” on the Aronchik scale were classified as inadequate as was the term “unsatisfactory.” We evaluated the hospital length of stay (LOS) as a secondary outcome measure.

Statistical Analysis

After describing the frequency of IBP, the quality of bowel preparation (adequate vs inadequate) was compared based on the predictors described above. Categorical variables were reported as frequencies with percentages and continuous variables were reported as medians with 25^th-75^th percentile values. The significance of the difference between the proportion or median values of those who had inadequate versus adequate bowel preparation was assessed. Two-sided chi-square analysis was used to assess the significance of differences between categorical variables and the Wilcoxon Rank-Sum test was used to assess the significance of differences between continuous variables.

Multivariate logistic regression analysis was performed to assess factors associated with hospital predictors and outcomes, after adjusting for all the aforementioned factors and clustering the effect based on the endoscopist. To evaluate the potential impact of modifiable factors on IBP, we performed counterfactual analysis, in which the observed distribution was compared to a hypothetical population in which all the modifiable risk factors were optimal.

Overall, 8,819 patients were included in our study population. They had a median age of 64 [53-76] years; 50.5% were female and 51% had an IBP. Patient characteristics and rates of IBP are presented in Table 1.

In unadjusted analyses, with regards to modifiable factors, opiate use within three days of colonoscopy was associated with a higher rate of IBP (55.4% vs 47.3%, P <.001), as was a lower volume (<4L) bowel preparation (55.3% vs 50.4%, P = .003). IBP was less frequent when colonoscopy was performed before noon vs afternoon (50.3% vs 57.4%, P < .001), and when patients were documented to receive a clear liquid diet or nil per os vs a solid diet the day prior to colonoscopy (50.3% vs 57.4%, P < .001). Overall bowel preparation quality improved over time (Figure 1). Median LOS was five [3-11] days. Patients who had IBP on their initial colonoscopy had a LOS one day longer than patients without IBP (six days vs five days, P < .001).

Multivariate Analysis

Table 2 shows the results of the multivariate analysis. The following modifiable factors were associated with IBP: opiate used within three days of the procedure (OR 1.31; 95% CI 1.8, 1.45), having the colonoscopy performed after12:00 pm (OR, 1.25; 95% CI, 1.10, 1.41), and consuming a solid diet the day prior to the colonoscopy (OR, 1.37; 95% CI, 1.18, 1.59). However, the volume of bowel preparation was not associated with IBP. The selected nonmodifiable factors that were found to be associated with IBP included age (increment of five years; OR, 1.04; 95% CI, 1.02, 1.05), male gender (OR, 1.33; 95% CI, 1.23, 1.44), Medicare insurance (OR, 1.17; 95% CI, 1.07, 1.28), Medicaid insurance (OR, 1.34; 95% CI, 1.07, 1.28), gastroparesis (OR, 1.62; 95% CI, 1.16, 2.27), nausea/vomiting (OR 1.21; 95% CI, 1.09, 1.34), and dysphagia (OR, 1.16; 95% CI, 1.01, 1.34).

Potential Impact of Modifiable Variables

We conducted a counterfactual analysis based on a multivariate model to assess the impact of each modifiable risk factor on the IBP rate (Figure 1). In the included study population, 44.9% received an opiate, 39.3% had a colonoscopy after 12:00 pm, and 9.1% received solid food the day prior to the procedure. Holding all other factors constant, if all patients were not prescribed opiates within three days of the procedure a 2.9% reduction in IBP would be expected. Similarly, if all patients underwent colonoscopy before noon, a 2.1% reduction in IBP rate would be expected. A 0.7% reduction would be expected if all patients were maintained on a liquid diet or nil per os. Combined, instituting all these changes (no opiates or solid diet before colonoscopy and performing all colonoscopies before noon) would produce a 5.6% reduction in IBP rate.

In this large, multihospital cohort, IBP was documented in half (51%) of 8,819 inpatient colonoscopies performed. Nonmodifiable patient characteristics independently associated with IBP were age, male gender, white race, Medicare and Medicaid insurance, nausea/vomiting, dysphagia, and gastroparesis. Modifiable factors included not consuming opiates within three days of colonoscopy, avoidance of a solid diet the day prior to colonoscopy and performing the colonoscopy before noon. The volume of bowel preparation consumed was not associated with IBP. In a counterfactual analysis, we found that if all three modifiable factors were optimized, the predicted rate of IBP would drop to 45%.

Many studies, including our analysis, have shown significant differences between the frequency of IBP in inpatient versus outpatient bowel preparations.^8-11 Therefore, it is crucial to study IBP in these settings separately. Three single-institution studies, including a total of 898 patients, have identified risk factors for inpatient IBP. Individual studies ranged in size from 130 to 524 patients with rates of IBP ranging from 22%-57%.^1-3 They found IBP to be associated with increasing age, lower income, ASA Grade >3, diabetes, coronary artery disease (CAD), nausea or vomiting, BMI >25, and chronic constipation. Modifiable factors included opiates, afternoon procedures, and runway times >6 hours.

We also found IBP to be associated with increasing age and male gender. However, we found no association with diabetes, chronic constipation, CAD or BMI. As we were able to adjust for a wider variety of variables, it is possible that we were able to account for residual confounding better than previous studies. For example, we found that having nausea/vomiting, dysphagia, and gastroparesis was associated with IBP. Gastroparesis with associated nausea and vomiting may be the mechanism by which diabetes increases the risk for IBP. Further studies are needed to assess if interventions or alternative bowel cleansing in these patients can result in improved IBP. Finally, in contrast to studies with smaller cohorts which found that lower volume bowel preps improved IBP in the right colon,^4,12 we found no association between IBP based and volume of bowel preparation consumed. Our impact analysis suggests that avoidance of opiates for at least three days before colonoscopy, avoidance of solid diet on the day before colonoscopy and performing all colonoscopies before noon would reduce the rate of IBP by 5.6%. While at first glance this does not appear to be a significant change, from a public health perspective with thousands of inpatient colonoscopies performed every year, it is crucial. We found that IBP was associated with an increased inpatient LOS of approximately one day. Assuming an average cost of one hospital day to be $2,000,¹³ this 5.6% improvement among our almost 9,000 patients, would translate into eliminating 494 unnecessary hospital days, or approximately $1 million in savings. More importantly, this savings comes without risk to patients and would result in an improvement in quality.

The factors mentioned above may not always be amenable to modification. For example, for patients with active gastrointestinal bleeding, postponing colonoscopy by one day for the sake of maintaining a patient on a clear diet may not be feasible. Similarly, performing colonoscopies in the morning is highly dependent on endoscopy suite availability and hospital logistics. Denying opiates to patients experiencing severe pain is not ethical. In many scenarios, however, these variables could be modified, and institutional efforts to support these practices could yield considerable savings. Future prospective studies are needed to verify the real impact of these changes.

Further discussion is needed to contextualize the finding that colonoscopies scheduled in the afternoon are associated with improved bowel preparation quality. Previous research—albeit in the outpatient setting—has demonstrated 11.8 hours as the maximum upper time limit for the time elapsed between the end of bowel preparation to colonoscopy.¹⁴ Another study found an inverse relationship between the quality of bowel preparation and the time after completion of the bowel preparation.¹⁵ This makes sense from a physiological perspective as delaying the time between completion of bowel preparation, and the procedure allows chyme from the small intestine to reaccumulate in the colon. Anecdotally, at our institution as well as at many others, the bowel preparations are ordered to start in the evening to allow the consumption of complete bowel preparation by midnight. As a result of this practice, only patients who have their colonoscopies scheduled before noon fall within the optimal period of 11.8 hours. In the outpatient setting, the use of split preparations has led to the obliteration of the difference in the quality of bowel preparation between morning and afternoon colonoscopies.¹⁶ Prospective trials are needed to evaluate the use of split preparations to improve the quality of afternoon inpatient colonoscopies.

In this large retrospective study evaluating bowel preparation quality in inpatients undergoing colonoscopy, we found that more than half of the patients have IBP and that IBP was associated with an extra day of hospitalization. Our study identifies those patients at highest risk and identifies modifiable risk factors for IBP. Specifically, we found that abstinence from opiates or solid diet before the colonoscopy, along with performing colonoscopies before noon were associated with improved outcomes. Prospective studies are needed to confirm the effects of these interventions on bowel preparation quality.

Disclosures

Carol A Burke, MD has received research funding from Ferring Pharmaceuticals. Other authors have no conflicts of interest to disclose.

Inadequate bowel preparation (IBP) at the time of inpatient colonoscopy is common and associated with increased length of stay and cost of care.¹ The factors that contribute to IBP can be categorized into those that are modifiable and those that are nonmodifiable. While many factors have been associated with IBP, studies have been limited by small sample size or have combined inpatient/outpatient populations, thus limiting generalizability.^1-5 Moreover, most factors associated with IBP, such as socioeconomic status, male gender, increased age, and comorbidities, are nonmodifiable. No studies have explicitly focused on modifiable risk factors, such as medication use, colonoscopy timing, or assessed the potential impact of modifying these factors.

In a large, multihospital system, we examine the frequency of IBP among inpatients undergoing colonoscopy along with factors associated with IBP. We attempted to identify modifiable risk factors that were associated with IBP.

Potential Predictors of IBP

Demographic data such as patient age, gender, ethnicity, body mass index (BMI), and insurance/payor status were obtained from the electronic health record (EHR). International Classification of Disease 9^th and 10^th revision, Clinical Modifications (ICD-9/10-CM) codes were used to obtain patient comorbidities including diabetes, coronary artery disease, heart failure, cirrhosis, gastroparesis, hypothyroidism, inflammatory bowel disease, constipation, stroke, dementia, dysphagia, and nausea/vomiting. Use of opioid medications within three days before colonoscopy was extracted from the medication administration record. These variables were chosen as biologically plausible modifiers of bowel preparation or had previously been assessed in the literature.^1-6 The name and volume, classified as 4 L (GoLytely^®) and < 4 liters (MoviPrep^®) of bowel preparation, time of day when colonoscopy was performed, solid diet the day prior to colonoscopy, type of sedation used (conscious sedation or general anesthesia), and total colonoscopy time (defined as the time from scope insertion to removal) was recorded. Hospitalization-related variables, including the number of hospitalizations in the year before the current hospitalization, the year in which the colonoscopy was performed, and the number of days from admission to colonoscopy, were also obtained from the EHR.

Outcome Measures

An internally validated natural language algorithm, using Structured Queried Language was used to search through colonoscopy reports to identify adequacy of bowel preparation. ProVation^® software allows the gastroenterologist to use some terms to describe bowel preparation in a drop-down menu format. In addition to the Aronchik scale (which allows the gastroenterologist to rate bowel preparation on a five-point scale: “excellent,” “good,” “fair,” “poor,” and “inadequate”) it also allows the provider to use terms such as “adequate” or “adequate to detect polyps >5 mm” as well as “unsatisfactory.”⁷ Mirroring prior literature, bowel preparation quality was classified into “adequate” and “inadequate”; “good” and “excellent” on the Aronchik scale were categorized as adequate as was the term “adequate” in any form; “fair,” “poor,” or “inadequate” on the Aronchik scale were classified as inadequate as was the term “unsatisfactory.” We evaluated the hospital length of stay (LOS) as a secondary outcome measure.

Statistical Analysis

After describing the frequency of IBP, the quality of bowel preparation (adequate vs inadequate) was compared based on the predictors described above. Categorical variables were reported as frequencies with percentages and continuous variables were reported as medians with 25^th-75^th percentile values. The significance of the difference between the proportion or median values of those who had inadequate versus adequate bowel preparation was assessed. Two-sided chi-square analysis was used to assess the significance of differences between categorical variables and the Wilcoxon Rank-Sum test was used to assess the significance of differences between continuous variables.

Multivariate logistic regression analysis was performed to assess factors associated with hospital predictors and outcomes, after adjusting for all the aforementioned factors and clustering the effect based on the endoscopist. To evaluate the potential impact of modifiable factors on IBP, we performed counterfactual analysis, in which the observed distribution was compared to a hypothetical population in which all the modifiable risk factors were optimal.

Overall, 8,819 patients were included in our study population. They had a median age of 64 [53-76] years; 50.5% were female and 51% had an IBP. Patient characteristics and rates of IBP are presented in Table 1.

In unadjusted analyses, with regards to modifiable factors, opiate use within three days of colonoscopy was associated with a higher rate of IBP (55.4% vs 47.3%, P <.001), as was a lower volume (<4L) bowel preparation (55.3% vs 50.4%, P = .003). IBP was less frequent when colonoscopy was performed before noon vs afternoon (50.3% vs 57.4%, P < .001), and when patients were documented to receive a clear liquid diet or nil per os vs a solid diet the day prior to colonoscopy (50.3% vs 57.4%, P < .001). Overall bowel preparation quality improved over time (Figure 1). Median LOS was five [3-11] days. Patients who had IBP on their initial colonoscopy had a LOS one day longer than patients without IBP (six days vs five days, P < .001).

Multivariate Analysis

Table 2 shows the results of the multivariate analysis. The following modifiable factors were associated with IBP: opiate used within three days of the procedure (OR 1.31; 95% CI 1.8, 1.45), having the colonoscopy performed after12:00 pm (OR, 1.25; 95% CI, 1.10, 1.41), and consuming a solid diet the day prior to the colonoscopy (OR, 1.37; 95% CI, 1.18, 1.59). However, the volume of bowel preparation was not associated with IBP. The selected nonmodifiable factors that were found to be associated with IBP included age (increment of five years; OR, 1.04; 95% CI, 1.02, 1.05), male gender (OR, 1.33; 95% CI, 1.23, 1.44), Medicare insurance (OR, 1.17; 95% CI, 1.07, 1.28), Medicaid insurance (OR, 1.34; 95% CI, 1.07, 1.28), gastroparesis (OR, 1.62; 95% CI, 1.16, 2.27), nausea/vomiting (OR 1.21; 95% CI, 1.09, 1.34), and dysphagia (OR, 1.16; 95% CI, 1.01, 1.34).

Potential Impact of Modifiable Variables

We conducted a counterfactual analysis based on a multivariate model to assess the impact of each modifiable risk factor on the IBP rate (Figure 1). In the included study population, 44.9% received an opiate, 39.3% had a colonoscopy after 12:00 pm, and 9.1% received solid food the day prior to the procedure. Holding all other factors constant, if all patients were not prescribed opiates within three days of the procedure a 2.9% reduction in IBP would be expected. Similarly, if all patients underwent colonoscopy before noon, a 2.1% reduction in IBP rate would be expected. A 0.7% reduction would be expected if all patients were maintained on a liquid diet or nil per os. Combined, instituting all these changes (no opiates or solid diet before colonoscopy and performing all colonoscopies before noon) would produce a 5.6% reduction in IBP rate.

In this large, multihospital cohort, IBP was documented in half (51%) of 8,819 inpatient colonoscopies performed. Nonmodifiable patient characteristics independently associated with IBP were age, male gender, white race, Medicare and Medicaid insurance, nausea/vomiting, dysphagia, and gastroparesis. Modifiable factors included not consuming opiates within three days of colonoscopy, avoidance of a solid diet the day prior to colonoscopy and performing the colonoscopy before noon. The volume of bowel preparation consumed was not associated with IBP. In a counterfactual analysis, we found that if all three modifiable factors were optimized, the predicted rate of IBP would drop to 45%.

Many studies, including our analysis, have shown significant differences between the frequency of IBP in inpatient versus outpatient bowel preparations.^8-11 Therefore, it is crucial to study IBP in these settings separately. Three single-institution studies, including a total of 898 patients, have identified risk factors for inpatient IBP. Individual studies ranged in size from 130 to 524 patients with rates of IBP ranging from 22%-57%.^1-3 They found IBP to be associated with increasing age, lower income, ASA Grade >3, diabetes, coronary artery disease (CAD), nausea or vomiting, BMI >25, and chronic constipation. Modifiable factors included opiates, afternoon procedures, and runway times >6 hours.

We also found IBP to be associated with increasing age and male gender. However, we found no association with diabetes, chronic constipation, CAD or BMI. As we were able to adjust for a wider variety of variables, it is possible that we were able to account for residual confounding better than previous studies. For example, we found that having nausea/vomiting, dysphagia, and gastroparesis was associated with IBP. Gastroparesis with associated nausea and vomiting may be the mechanism by which diabetes increases the risk for IBP. Further studies are needed to assess if interventions or alternative bowel cleansing in these patients can result in improved IBP. Finally, in contrast to studies with smaller cohorts which found that lower volume bowel preps improved IBP in the right colon,^4,12 we found no association between IBP based and volume of bowel preparation consumed. Our impact analysis suggests that avoidance of opiates for at least three days before colonoscopy, avoidance of solid diet on the day before colonoscopy and performing all colonoscopies before noon would reduce the rate of IBP by 5.6%. While at first glance this does not appear to be a significant change, from a public health perspective with thousands of inpatient colonoscopies performed every year, it is crucial. We found that IBP was associated with an increased inpatient LOS of approximately one day. Assuming an average cost of one hospital day to be $2,000,¹³ this 5.6% improvement among our almost 9,000 patients, would translate into eliminating 494 unnecessary hospital days, or approximately $1 million in savings. More importantly, this savings comes without risk to patients and would result in an improvement in quality.

The factors mentioned above may not always be amenable to modification. For example, for patients with active gastrointestinal bleeding, postponing colonoscopy by one day for the sake of maintaining a patient on a clear diet may not be feasible. Similarly, performing colonoscopies in the morning is highly dependent on endoscopy suite availability and hospital logistics. Denying opiates to patients experiencing severe pain is not ethical. In many scenarios, however, these variables could be modified, and institutional efforts to support these practices could yield considerable savings. Future prospective studies are needed to verify the real impact of these changes.

Further discussion is needed to contextualize the finding that colonoscopies scheduled in the afternoon are associated with improved bowel preparation quality. Previous research—albeit in the outpatient setting—has demonstrated 11.8 hours as the maximum upper time limit for the time elapsed between the end of bowel preparation to colonoscopy.¹⁴ Another study found an inverse relationship between the quality of bowel preparation and the time after completion of the bowel preparation.¹⁵ This makes sense from a physiological perspective as delaying the time between completion of bowel preparation, and the procedure allows chyme from the small intestine to reaccumulate in the colon. Anecdotally, at our institution as well as at many others, the bowel preparations are ordered to start in the evening to allow the consumption of complete bowel preparation by midnight. As a result of this practice, only patients who have their colonoscopies scheduled before noon fall within the optimal period of 11.8 hours. In the outpatient setting, the use of split preparations has led to the obliteration of the difference in the quality of bowel preparation between morning and afternoon colonoscopies.¹⁶ Prospective trials are needed to evaluate the use of split preparations to improve the quality of afternoon inpatient colonoscopies.

In this large retrospective study evaluating bowel preparation quality in inpatients undergoing colonoscopy, we found that more than half of the patients have IBP and that IBP was associated with an extra day of hospitalization. Our study identifies those patients at highest risk and identifies modifiable risk factors for IBP. Specifically, we found that abstinence from opiates or solid diet before the colonoscopy, along with performing colonoscopies before noon were associated with improved outcomes. Prospective studies are needed to confirm the effects of these interventions on bowel preparation quality.

Disclosures

Carol A Burke, MD has received research funding from Ferring Pharmaceuticals. Other authors have no conflicts of interest to disclose.

Inadequate bowel preparation (IBP) at the time of inpatient colonoscopy is common and associated with increased length of stay and cost of care.¹ The factors that contribute to IBP can be categorized into those that are modifiable and those that are nonmodifiable. While many factors have been associated with IBP, studies have been limited by small sample size or have combined inpatient/outpatient populations, thus limiting generalizability.^1-5 Moreover, most factors associated with IBP, such as socioeconomic status, male gender, increased age, and comorbidities, are nonmodifiable. No studies have explicitly focused on modifiable risk factors, such as medication use, colonoscopy timing, or assessed the potential impact of modifying these factors.

In a large, multihospital system, we examine the frequency of IBP among inpatients undergoing colonoscopy along with factors associated with IBP. We attempted to identify modifiable risk factors that were associated with IBP.

Potential Predictors of IBP

Demographic data such as patient age, gender, ethnicity, body mass index (BMI), and insurance/payor status were obtained from the electronic health record (EHR). International Classification of Disease 9^th and 10^th revision, Clinical Modifications (ICD-9/10-CM) codes were used to obtain patient comorbidities including diabetes, coronary artery disease, heart failure, cirrhosis, gastroparesis, hypothyroidism, inflammatory bowel disease, constipation, stroke, dementia, dysphagia, and nausea/vomiting. Use of opioid medications within three days before colonoscopy was extracted from the medication administration record. These variables were chosen as biologically plausible modifiers of bowel preparation or had previously been assessed in the literature.^1-6 The name and volume, classified as 4 L (GoLytely^®) and < 4 liters (MoviPrep^®) of bowel preparation, time of day when colonoscopy was performed, solid diet the day prior to colonoscopy, type of sedation used (conscious sedation or general anesthesia), and total colonoscopy time (defined as the time from scope insertion to removal) was recorded. Hospitalization-related variables, including the number of hospitalizations in the year before the current hospitalization, the year in which the colonoscopy was performed, and the number of days from admission to colonoscopy, were also obtained from the EHR.

Outcome Measures

An internally validated natural language algorithm, using Structured Queried Language was used to search through colonoscopy reports to identify adequacy of bowel preparation. ProVation^® software allows the gastroenterologist to use some terms to describe bowel preparation in a drop-down menu format. In addition to the Aronchik scale (which allows the gastroenterologist to rate bowel preparation on a five-point scale: “excellent,” “good,” “fair,” “poor,” and “inadequate”) it also allows the provider to use terms such as “adequate” or “adequate to detect polyps >5 mm” as well as “unsatisfactory.”⁷ Mirroring prior literature, bowel preparation quality was classified into “adequate” and “inadequate”; “good” and “excellent” on the Aronchik scale were categorized as adequate as was the term “adequate” in any form; “fair,” “poor,” or “inadequate” on the Aronchik scale were classified as inadequate as was the term “unsatisfactory.” We evaluated the hospital length of stay (LOS) as a secondary outcome measure.

Statistical Analysis

After describing the frequency of IBP, the quality of bowel preparation (adequate vs inadequate) was compared based on the predictors described above. Categorical variables were reported as frequencies with percentages and continuous variables were reported as medians with 25^th-75^th percentile values. The significance of the difference between the proportion or median values of those who had inadequate versus adequate bowel preparation was assessed. Two-sided chi-square analysis was used to assess the significance of differences between categorical variables and the Wilcoxon Rank-Sum test was used to assess the significance of differences between continuous variables.

Multivariate logistic regression analysis was performed to assess factors associated with hospital predictors and outcomes, after adjusting for all the aforementioned factors and clustering the effect based on the endoscopist. To evaluate the potential impact of modifiable factors on IBP, we performed counterfactual analysis, in which the observed distribution was compared to a hypothetical population in which all the modifiable risk factors were optimal.

Overall, 8,819 patients were included in our study population. They had a median age of 64 [53-76] years; 50.5% were female and 51% had an IBP. Patient characteristics and rates of IBP are presented in Table 1.

In unadjusted analyses, with regards to modifiable factors, opiate use within three days of colonoscopy was associated with a higher rate of IBP (55.4% vs 47.3%, P <.001), as was a lower volume (<4L) bowel preparation (55.3% vs 50.4%, P = .003). IBP was less frequent when colonoscopy was performed before noon vs afternoon (50.3% vs 57.4%, P < .001), and when patients were documented to receive a clear liquid diet or nil per os vs a solid diet the day prior to colonoscopy (50.3% vs 57.4%, P < .001). Overall bowel preparation quality improved over time (Figure 1). Median LOS was five [3-11] days. Patients who had IBP on their initial colonoscopy had a LOS one day longer than patients without IBP (six days vs five days, P < .001).

Multivariate Analysis

Table 2 shows the results of the multivariate analysis. The following modifiable factors were associated with IBP: opiate used within three days of the procedure (OR 1.31; 95% CI 1.8, 1.45), having the colonoscopy performed after12:00 pm (OR, 1.25; 95% CI, 1.10, 1.41), and consuming a solid diet the day prior to the colonoscopy (OR, 1.37; 95% CI, 1.18, 1.59). However, the volume of bowel preparation was not associated with IBP. The selected nonmodifiable factors that were found to be associated with IBP included age (increment of five years; OR, 1.04; 95% CI, 1.02, 1.05), male gender (OR, 1.33; 95% CI, 1.23, 1.44), Medicare insurance (OR, 1.17; 95% CI, 1.07, 1.28), Medicaid insurance (OR, 1.34; 95% CI, 1.07, 1.28), gastroparesis (OR, 1.62; 95% CI, 1.16, 2.27), nausea/vomiting (OR 1.21; 95% CI, 1.09, 1.34), and dysphagia (OR, 1.16; 95% CI, 1.01, 1.34).

Potential Impact of Modifiable Variables

We conducted a counterfactual analysis based on a multivariate model to assess the impact of each modifiable risk factor on the IBP rate (Figure 1). In the included study population, 44.9% received an opiate, 39.3% had a colonoscopy after 12:00 pm, and 9.1% received solid food the day prior to the procedure. Holding all other factors constant, if all patients were not prescribed opiates within three days of the procedure a 2.9% reduction in IBP would be expected. Similarly, if all patients underwent colonoscopy before noon, a 2.1% reduction in IBP rate would be expected. A 0.7% reduction would be expected if all patients were maintained on a liquid diet or nil per os. Combined, instituting all these changes (no opiates or solid diet before colonoscopy and performing all colonoscopies before noon) would produce a 5.6% reduction in IBP rate.

In this large, multihospital cohort, IBP was documented in half (51%) of 8,819 inpatient colonoscopies performed. Nonmodifiable patient characteristics independently associated with IBP were age, male gender, white race, Medicare and Medicaid insurance, nausea/vomiting, dysphagia, and gastroparesis. Modifiable factors included not consuming opiates within three days of colonoscopy, avoidance of a solid diet the day prior to colonoscopy and performing the colonoscopy before noon. The volume of bowel preparation consumed was not associated with IBP. In a counterfactual analysis, we found that if all three modifiable factors were optimized, the predicted rate of IBP would drop to 45%.

Many studies, including our analysis, have shown significant differences between the frequency of IBP in inpatient versus outpatient bowel preparations.^8-11 Therefore, it is crucial to study IBP in these settings separately. Three single-institution studies, including a total of 898 patients, have identified risk factors for inpatient IBP. Individual studies ranged in size from 130 to 524 patients with rates of IBP ranging from 22%-57%.^1-3 They found IBP to be associated with increasing age, lower income, ASA Grade >3, diabetes, coronary artery disease (CAD), nausea or vomiting, BMI >25, and chronic constipation. Modifiable factors included opiates, afternoon procedures, and runway times >6 hours.

We also found IBP to be associated with increasing age and male gender. However, we found no association with diabetes, chronic constipation, CAD or BMI. As we were able to adjust for a wider variety of variables, it is possible that we were able to account for residual confounding better than previous studies. For example, we found that having nausea/vomiting, dysphagia, and gastroparesis was associated with IBP. Gastroparesis with associated nausea and vomiting may be the mechanism by which diabetes increases the risk for IBP. Further studies are needed to assess if interventions or alternative bowel cleansing in these patients can result in improved IBP. Finally, in contrast to studies with smaller cohorts which found that lower volume bowel preps improved IBP in the right colon,^4,12 we found no association between IBP based and volume of bowel preparation consumed. Our impact analysis suggests that avoidance of opiates for at least three days before colonoscopy, avoidance of solid diet on the day before colonoscopy and performing all colonoscopies before noon would reduce the rate of IBP by 5.6%. While at first glance this does not appear to be a significant change, from a public health perspective with thousands of inpatient colonoscopies performed every year, it is crucial. We found that IBP was associated with an increased inpatient LOS of approximately one day. Assuming an average cost of one hospital day to be $2,000,¹³ this 5.6% improvement among our almost 9,000 patients, would translate into eliminating 494 unnecessary hospital days, or approximately $1 million in savings. More importantly, this savings comes without risk to patients and would result in an improvement in quality.

The factors mentioned above may not always be amenable to modification. For example, for patients with active gastrointestinal bleeding, postponing colonoscopy by one day for the sake of maintaining a patient on a clear diet may not be feasible. Similarly, performing colonoscopies in the morning is highly dependent on endoscopy suite availability and hospital logistics. Denying opiates to patients experiencing severe pain is not ethical. In many scenarios, however, these variables could be modified, and institutional efforts to support these practices could yield considerable savings. Future prospective studies are needed to verify the real impact of these changes.

Further discussion is needed to contextualize the finding that colonoscopies scheduled in the afternoon are associated with improved bowel preparation quality. Previous research—albeit in the outpatient setting—has demonstrated 11.8 hours as the maximum upper time limit for the time elapsed between the end of bowel preparation to colonoscopy.¹⁴ Another study found an inverse relationship between the quality of bowel preparation and the time after completion of the bowel preparation.¹⁵ This makes sense from a physiological perspective as delaying the time between completion of bowel preparation, and the procedure allows chyme from the small intestine to reaccumulate in the colon. Anecdotally, at our institution as well as at many others, the bowel preparations are ordered to start in the evening to allow the consumption of complete bowel preparation by midnight. As a result of this practice, only patients who have their colonoscopies scheduled before noon fall within the optimal period of 11.8 hours. In the outpatient setting, the use of split preparations has led to the obliteration of the difference in the quality of bowel preparation between morning and afternoon colonoscopies.¹⁶ Prospective trials are needed to evaluate the use of split preparations to improve the quality of afternoon inpatient colonoscopies.

In this large retrospective study evaluating bowel preparation quality in inpatients undergoing colonoscopy, we found that more than half of the patients have IBP and that IBP was associated with an extra day of hospitalization. Our study identifies those patients at highest risk and identifies modifiable risk factors for IBP. Specifically, we found that abstinence from opiates or solid diet before the colonoscopy, along with performing colonoscopies before noon were associated with improved outcomes. Prospective studies are needed to confirm the effects of these interventions on bowel preparation quality.

Disclosures

Carol A Burke, MD has received research funding from Ferring Pharmaceuticals. Other authors have no conflicts of interest to disclose.

Sepsis is both the most expensive condition treated and the most common cause of death in hospitals in the United States.^1-3 Most sepsis patients (as many as 80% to 90%) meet sepsis criteria on hospital arrival, but mortality and costs are higher when meeting criteria after admission.^3-6 Mechanisms of this increased mortality for these distinct populations are not well explored. Patients who present septic in the emergency department (ED) and patients who present as inpatients likely present very different challenges for recognition, treatment, and monitoring.⁷ Yet, how these groups differ by demographic and clinical characteristics, the etiology and severity of infection, and patterns of resuscitation care are not well described. Literature on sepsis epidemiology on hospital wards is particularly limited.⁸

This knowledge gap is important. If hospital-presenting sepsis (HPS) contributes disproportionately to disease burdCHFens, it reflects a high-yield population deserving the focus of quality improvement (QI) initiatives. If specific causes of disparities were identified—eg, poor initial resuscitation— they could be specifically targeted for correction. Given that current treatment guidelines are uniform for the two populations,^9,10 characterizing phenotypic differences could also have implications for both diagnostic and therapeutic recommendations, particularly if the groups display substantially differing clinical presentations. Our prior work has not probed these effects specifically, but suggested ED versus inpatient setting at the time of initial sepsis presentation might be an effect modifier for the association between several elements of fluid resuscitation and patient outcomes.^11,12

We, therefore, conducted a retrospective analysis to ask four sequential questions: (1) Do patients with HPS, compared with EDPS, contribute adverse outcome out of proportion to case prevalence? (2) At the time of initial presentation, how do HPS patients differ from EDPS patients with respect to demographics, comorbidities, infectious etiologies, clinical presentations, and severity of illness (3) If holding observed baseline factors constant, does the physical location of sepsis presentation inherently increase the risk for treatment delays and mortality? (4) To what extent can differences in the likelihood for timely initial treatment between the ED and inpatient settings explain differences in mortality and patient outcomes?

We hypothesized a priori that HPS would reflect chronically sicker patients whom both received less timely resuscitation and who contributed disproportionately frequent bad outcomes. We expected disparities in timely resuscitation care would explain a large proportion of this difference.

We performed a retrospective analysis of the Northwell Sepsis Database, a prospectively captured, multisite, real world, consecutive-sample cohort of all “severe sepsis” and septic shock patients treated at nine tertiary and community hospitals in New York from October 1, 2014, to March 31, 2016. We analyzed all patients from a previously published cohort.¹¹

Database Design and Structure

The Northwell Sepsis Database has previously been described in detail.^11,13,14 Briefly, all patients met clinical sepsis criteria: (1) infection AND (2) ≥2 (SIRS) criteria AND (3) ≥1 acute organ dysfunction criterion. Organ dysfunction criteria were hypotension, acute kidney injury (AKI), coagulopathy, altered gas exchange, elevated bilirubin (≥2.0 mg/dL), or altered mental status (AMS; clarified in Supplemental Table 1). All organ dysfunction was not otherwise explained by patients’ medical histories; eg, patients on warfarin anticoagulation were not documented to have coagulopathy based on international normalized ratio > 1.5. The time of the sepsis episode (and database inclusion) was the time of the first vital sign measurement or laboratory result where a patient simultaneously met all three inclusion criteria: infection, SIRS, and organ dysfunction. The database excludes patients who were <18 years, declined bundle interventions, had advance directives precluding interventions, or were admitted directly to palliative care or hospice. Abstractors assumed comorbidities were absent if not documented within the medical record and that physiologic abnormalities were absent if not measured by the treatment team. There were no missing data for the variables analyzed. We report analysis in adherence with the STROBE statement guidelines for observational research.

Exposure

The primary exposure was whether patients had EDPS versus HPS. We defined EDPS patients as meeting all objective clinical inclusion criteria while physically in the ED. We defined HPS as first meeting sepsis inclusion criteria outside the ED, regardless of the reason for admission, and regardless of whether patients were admitted through the ED or directly to the hospital. All ED patients were admitted to the hospital.

Outcomes

Process outcomes were full 3-hour bundle compliance, time to antibiotic administration, blood cultures before antibiotics, time to fluid initiation, the volume of administered fluid resuscitation, lactate result time, and whether repeat lactate was obtained (Supplemental Table 2). Treatment times were times of administration (rather than order time). The primary patient outcome was hospital mortality. Secondary patient outcomes were mechanical ventilation, ICU admission, ICU days, hospital length of stay (LOS). We discounted HPS patients’ LOS to include only days after meeting the inclusion criteria. Patients were excluded from the analysis of the ICU admission outcome if they were already in the ICU prior to meeting sepsis criteria.

Statistical Analysis

We report continuous variables as means (standard deviation) or medians (interquartile range), and categorical variables as frequencies (proportions), as appropriate. Summative statistics with 95% confidence intervals (CI) describe overall group contributions. We used generalized linear models to determine patient factors associated with EDPS versus HPS, entering random effects for individual study sites to control for intercenter variability.

Next, to generate a propensity-matched cohort, we computed propensity scores adjusted from a priori selected variables: age, sex, tertiary versus community hospital, congestive heart failure (CHF), renal failure, COPD, diabetes, liver failure, immunocompromise, primary source of infection, nosocomial source, temperature, initial lactate, presenting hypotension, altered gas exchange, AMS, AKI, and coagulopathy. We then matched subjects 1:1 without optimization or replacement, imposing a caliper width of 0.01; ie, we required matched pairs to have a <1.0% difference in propensity scores. The macro used to match subjects is publically available.¹⁵

We then compared resuscitation and patient outcomes in the matched cohort using generalized linear models, ie, doubly-robust estimation (DRE).¹⁶ When assessing patient outcomes corrected for resuscitation, we used mixed DRE/multivariable regression. We did this for two reasons: first, DRE has the advantage of only requiring only one approach (propensity vs covariate adjustments) to be correctly specified.¹⁶ Second, computing propensity scores adjusted for resuscitation would be inappropriate given that resuscitation occurs after the exposure allocation (HPS vs EDPS). However, these factors could still impact the outcome and in fact, we hypothesized they were potential mediators of the exposure effect. To interrogate this mediating relationship, we recapitulated the DRE modeling but added covariates for resuscitation factors. Resuscitation-adjusted models controlled for timeliness of antibiotics, fluids, and lactate results; blood cultures before antibiotics; repeat lactate obtained, and fluid volume in the first six hours. Since ICU days and LOS are subject to competing risks bias (LOS could be shorter if patients died earlier), we used proportional hazards models where “the event” was defined as a live discharge to censor for mortality and we report output as inverse hazard ratios. We also tested interaction coefficients for discrete bundle elements and HPS to determine if specific bundle elements were effect modifiers for the association between the presenting location and mortality risk. Finally, we estimated attributable risk differences by comparing adjusted odds ratios of adverse outcome with and without adjustment for resuscitation variables, as described by Sahai et al.¹⁷

As sensitivity analyses, we recomputed propensity scores and generated a new matched cohort that excluded HPS patients who met criteria for sepsis while already in the ICU for another reason (ie, excluding ICU-presenting sepsis). We then recapitulated all analyses as above for this cohort. We performed analyses using SAS version 9.4 (SAS Institute, Cary, North Carolina).

Prevalence and Outcome Contributions

Of the 11,182 sepsis patients in the database, we classified 2,509 (22%) as HPS (Figure 1). HPS contributed 785 (35%) of 2,241 sepsis-related mortalities, 1,241 (38%) mechanical ventilations, and 1,762 (34%) ICU admissions. Of 39,263 total ICU days and 127,178 hospital days, HPS contributed 18,104 (46.1%) and 44,412 (34.9%) days, respectively.

Patient Characteristics

Most HPS presented early in the hospital course, with 1,352 (53.9%) cases meeting study criteria within three days of admission. Median time from admission to meeting study criteria for HPS was two days (interquartile range: one to seven days). We report selected baseline patient characteristics in Table 1 and adjusted associations of baseline variables with HPS versus EDPS in Table 2. The full cohort characterization is available in Supplemental Table 3. Notably, HPS patients more often had CHF (aOR [adjusted odds ratio}: 1.31, CI: 1.18-1.47) or renal failure (aOR: 1.62, CI: 1.38-1.91), gastrointestinal source of infection (aOR: 1.84, CI: 1.48-2.29), hypothermia (aOR: 1.56, CI: 1.28-1.90) hypotension (aOR: 1.85, CI: 1.65-2.08), or altered gas exchange (aOR: 2.46, CI: 1.43-4.24). In contrast, HPS patients less frequently were admitted from skilled nursing facilities (aOR: 0.44, CI: 0.32-0.60), or had COPD (aOR: 0.53, CI: 0.36-0.76), fever (aOR: 0.70, CI: 0.52-0.91), tachypnea (aOR: 0.76, CI: 0.58-0.98), or AKI (aOR: 082, CI: 0.68-0.97). Other baseline variables were similar, including respiratory source, tachycardia, white cell abnormalities, AMS, and coagulopathies. These associations were preserved in the sensitivity analysis excluding ICU-presenting sepsis.

Propensity Matching

Propensity score matching yielded 1,942 matched pairs (n = 3,884, 77% of HPS patients, 22% of EDPS patients). Table 1 and Supplemental Table 3 show patient characteristics after propensity matching. Supplemental Table 4 shows the propensity model. The frequency densities are shown for the cohort as a function of propensity score in Supplemental Figure 1. After matching, frequencies between groups differed by <5% for all categorical variables assessed. In the sensitivity analysis, propensity matching (model in Supplemental Table 5) resulted in 1,233 matched pairs (n = 2,466, 49% of HPS patients, 14% of EDPS patients), with group differences comparable to the primary analysis.

Process Outcomes

We present propensity-matched differences in initial resuscitation in Figure 2A for all HPS patients, as well as non-ICU-presenting HPS, versus EDPS. HPS patients were roughly half as likely to receive fully 3-hour bundle compliant care (17.0% vs 30.3%, aOR: 0.47, CI: 0.40-0.57), to have blood cultures drawn within three hours prior to antibiotics (44.9% vs 67.2%, aOR: 0.40, CI: 0.35-0.46), or to receive fluid resuscitation initiated within two hours (11.1% vs 26.1%, aOR: 0.35, CI: 0.29-0.42). Antibiotic receipt within one hour was comparable (45.3% vs 48.1%, aOR: 0.89, CI: 0.79-1.01). However, differences emerged for antibiotics within three hours (66.2% vs 83.8%, aOR: 0.38, CI: 0.32-0.44) and persisted at six hours (77.0% vs 92.5%, aOR: 0.27, CI: 0.22-33). Excluding ICU-presenting sepsis from propensity matching exaggerated disparities in antibiotic receipt at one hour (43.4% vs 49.1%, aOR: 0.80, CI: 0.68-0.93), three hours (64.2% vs 86.1%, aOR: 0.29, CI: 0.24-0.35), and six hours (75.7% vs 93.0%, aOR: 0.23, CI: 0.18-0.30). HPS patients more frequently had repeat lactate obtained within 24 hours (62.4% vs 54.3%, aOR: 1.40, CI: 1.23-1.59).

Patient Outcomes

HPS patients had higher mortality (31.2% vs19.3%), mechanical ventilation (51.5% vs27.4%), and ICU admission (60.6% vs 46.5%) (Table 1 and Supplemental Table 6). Figure 2b shows propensity-matched and covariate-adjusted differences in patient outcomes before and after adjusting for initial resuscitation. aORs corresponded to approximate relative risk differences¹⁸ of 1.38 (CI: 1.28-1.48), 1.68 (CI: 1.57-1.79), and 1.72 (CI: 1.61-1.84) for mortality, mechanical ventilation, and ICU admission, respectively. HPS was associated with 83% longer mortality-censored ICU stays (five vs nine days, HR^–1: 1.83, CI: 1.65-2.03), and 108% longer hospital stay (eight vs 17 days, HR^–1: 2.08, CI: 1.93-2.24). After adjustment for resuscitation, all effect sizes decreased but persisted. The initial crystalloid volume was a significant negative effect modifier for mortality (Supplemental Table 7). That is, the magnitude of the association between HPS and greater mortality decreased by a factor of 0.89 per 10 mL/kg given (CI: 0.82-0.97). We did not observe significant interaction from other interventions, or overall bundle compliance, meaning these interventions’ association with mortality did not significantly differ between HPS versus EDPS.

The implied attributable risk difference from discrepancies in initial resuscitation was 23.3% for mortality, 35.2% for mechanical ventilation, and 7.6% for ICU admission (Figure 2B). Resuscitation explained 26.5% of longer ICU LOS and 16.7% of longer hospital LOS associated with HPS.

Figure 2C shows sensitivity analysis excluding ICU-presenting sepsis from propensity matching (ie, limiting HPS to hospital ward presentations). Again, HPS was associated with all adverse outcomes, though effect sizes were smaller than in the primary cohort for all outcomes except hospital LOS. In this cohort, resuscitation factors now explained 16.5% of HPS’ association with mortality, and 14.5% of the association with longer ICU LOS. However, they explained a greater proportion (13.0%) of ICU admissions. Attributable risk differences were comparable to the primary cohort for mechanical ventilation (37.6%) and hospital LOS (15.3%).

In this analysis of 11,182 sepsis and septic shock patients, HPS contributed 22% of prevalence but >35% of total sepsis mortalities, ICU utilization, and hospital days. HPS patients had higher comorbidity burdens and had clinical presentations less obviously attributable to infection with more severe organ dysfunction. EDPS received antibiotics within three hours about 1.62 times more often than do HPS patients. EDPS patients also receive fluids initiated within two hours about 1.82 times more often than HPS patients do. HPS had nearly 1.5-fold greater mortality and LOS, and nearly two-fold greater mechanical ventilation and ICU utilization. Resuscitation disparities could partially explain these associations. These patterns persisted when comparing only wards presenting HPS with EDPS.

Our analysis revealed several notable findings. First, these data confirm that HPS represents a potentially high-impact target population that contributes adverse outcomes disproportionately frequently with respect to case prevalence.

Our findings, unsurprisingly, revealed HPS and EDPS reflect dramatically different patient populations. We found that the two groups significantly differed by the majority of the baseline factors we compared. It may be worth asking if and how these substantial differences in illness etiology, chronic health, and acute physiology impact what we consider an optimal approach to management. Significant interaction effects of fluid volume on the association between HPS and mortality suggest differential treatment effects may exist between patients. Indeed, patients who newly arrive from the community and those who are several days into admission likely have different volume status. However, no interactions were noted with other bundle elements, such as timeliness of antibiotics or timeliness of initial fluids.

Another potentially concerning observation was that HPS patients were admitted much less frequently from skilled nursing facilities, as it could imply that this poorer-fairing population had a comparatively higher baseline functional status. The fact that 25% of EDPS cases were admitted from these facilities also underscores the need to engage skilled nursing facility providers in future sepsis initiatives.

We found marked disparities in resuscitation. Timely delivery of interventions, such as antibiotics and initial fluid resuscitation, occurred less than half as often for HPS, especially on hospital wards. While evidence supporting the efficacy of specific 3-hour bundle elements remains unsettled,¹⁹ a wealth of literature demonstrates a correlation between bundle uptake and decreased sepsis mortality, especially for early antibiotic administration.^13,20-26 Some analysis suggests that differing initial resuscitation practices explain different mortality rates in the early goal-directed therapy trials.²⁷ The comparatively poor performance for non-ICU HPS indicates further QI efforts are better focused on inpatient wards, rather than on EDs or ICUs where resuscitation is already delivered with substantially greater fidelity.

While resuscitation differences partially explained outcome discrepancies between groups, they did not account for as much variation as expected. Though resuscitation accounted for >35% of attributable mechanical ventilation risk, it explained only 16.5% of mortality differences for non-ICU HPS vs EDPS. We speculate that several factors may contribute.

First, HPS patients are already hospitalized for another acute insult and may be too physiologically brittle to derive equal benefit from initial resuscitation. Some literature suggests protocolized sepsis resuscitation may paradoxically be more effective in milder/earlier disease.²⁸

Second, clinical information indicating septic organ dysfunction may become available too late in HPS—a possible data limitation where inpatient providers are counterintuitively more likely to miss early signs of patients’ deterioration and a subsequent therapeutic window. Several studies found that fluid resuscitation is associated with improved sepsis outcomes only when it is administered very early.^11,29-31 In inpatient wards, decreased monitoring³² and human factors (eg, hospital workflow, provider-to-patient ratios, electronic documentation burdens)^33,34 may hinder early diagnosis. In contrast, ED environments are explicitly designed to identify acutely ill patients and deliver intervention rapidly. If HPS patients were sicker when they were identified, this would also explain their more severe organ dysfunctions. Our data seems to support this possibility. HPS patients had tachypnea less frequently but more often had impaired gas exchange. This finding may suggest that early tachypnea was either less often detected or documented, or that it had progressed further by the time of detection.

Third, inpatients with sepsis may more often present with greater diagnostic complexity. We observed that HPS patients were more often euthermic and less often tachypneic. Beyond suggesting a greater diagnostic challenge, this also raises questions as to whether differences reflect patient physiology (response to infection) or iatrogenic factors (eg, prior antipyretics). Higher comorbidity and acute physiological burdens also limit the degree to which new organ dysfunction can be clearly attributed to infection. We note differences in the proportion of patients who received antibiotics increased over time, suggesting that HPS patients who received delayed antibiotics did so much later than their EDPS counterparts. This lag could also arise from diagnostic difficulty.

All three possibilities highlight a potential lead time effect, where the same measured three-hour period on the wards, between meeting sepsis criteria and starting treatment, actually reflects a longer period between (as yet unmeasurable) pathobiologic “time zero” and treatment versus the ED. The time of sepsis detection, as distinct from the time of sepsis onset, therefore proves difficult to evaluate and impossible to account for statistically.

Regardless, our findings suggest additional difficulty in both the recognition and resuscitation of inpatient sepsis. Inpatients, especially with infections, may need closer monitoring. How to cost effectively implement this monitoring is a challenge that deserves attention.

A more rational systems approach to HPS likely combines efforts to improve initial resuscitation with other initiatives aimed at both improving monitoring and preventing infection.

To be clear, we do not imply that timely initial resuscitation does not matter on the wards. Rather, resuscitation-focused QI alone does not appear to be sufficient to overcome differences in outcomes for HPS. The 23.3% attributable mortality risk we observed still implies that resuscitation differences could explain nearly one in four excess HPS mortalities. We previously showed that timely resuscitation is strongly associated with better outcomes.^11,13,30 As discussed above, the unclear degree to which better resuscitation is a marker for more obvious presentations is a persistent limitation of prior investigations and the present study.

Taken together, the ultimate question that this study raises but cannot answer is whether the timely recognition of sepsis, rather than any specific treatment, is what truly improves outcomes.

In addition to those above, this study has several limitations. Our study did not differentiate HPS with respect to patients admitted for noninfectious reasons and who subsequently became septic versus nonseptic patients admitted for an infection who subsequently became septic from that infection. Nor could we discriminate between missed ED diagnoses and true delayed presentations. We note distinguishing these entities clinically can be equally challenging. Additionally, this was a propensity-matched retrospective analysis of an existing sepsis cohort, and the many limitations of both retrospective study and propensity matching apply.^35,36 We note that randomizing patients to develop sepsis in the community versus hospital is not feasible and that two of our aims intended to describe overall patterns rather than causal effects. We could not ascertain robust measures of severity of illness (eg, SOFA) because a real world setting precludes required data points—eg, urine output is unreliably recorded. We also note incomplete overlap between inclusion criteria and either Sepsis-2 or -3 definitions,^1,37 because we designed and populated our database prior to publication of Sepsis-3. Further, we could not account for surgical source control, the appropriateness of antimicrobial therapy, mechanical ventilation before sepsis onset, or most treatments given after initial resuscitation.

In conclusion, hospital-presenting sepsis accounted for adverse patient outcomes disproportionately to prevalence. HPS patients had more complex presentations, received timely antibiotics half as often ED-presenting sepsis, and had nearly twice the mortality odds. Resuscitation disparities explained roughly 25% of this difference.

Disclosures

The authors have no conflicts of interest to disclose.

Funding

This investigation was funded in part by a grant from the Center for Medicare and Medicaid Innovation to the High Value Healthcare Collaborative, of which the study sites’ umbrella health system was a part. This grant helped fund the underlying QI program and database in this study.

Sepsis is both the most expensive condition treated and the most common cause of death in hospitals in the United States.^1-3 Most sepsis patients (as many as 80% to 90%) meet sepsis criteria on hospital arrival, but mortality and costs are higher when meeting criteria after admission.^3-6 Mechanisms of this increased mortality for these distinct populations are not well explored. Patients who present septic in the emergency department (ED) and patients who present as inpatients likely present very different challenges for recognition, treatment, and monitoring.⁷ Yet, how these groups differ by demographic and clinical characteristics, the etiology and severity of infection, and patterns of resuscitation care are not well described. Literature on sepsis epidemiology on hospital wards is particularly limited.⁸

This knowledge gap is important. If hospital-presenting sepsis (HPS) contributes disproportionately to disease burdCHFens, it reflects a high-yield population deserving the focus of quality improvement (QI) initiatives. If specific causes of disparities were identified—eg, poor initial resuscitation— they could be specifically targeted for correction. Given that current treatment guidelines are uniform for the two populations,^9,10 characterizing phenotypic differences could also have implications for both diagnostic and therapeutic recommendations, particularly if the groups display substantially differing clinical presentations. Our prior work has not probed these effects specifically, but suggested ED versus inpatient setting at the time of initial sepsis presentation might be an effect modifier for the association between several elements of fluid resuscitation and patient outcomes.^11,12

We, therefore, conducted a retrospective analysis to ask four sequential questions: (1) Do patients with HPS, compared with EDPS, contribute adverse outcome out of proportion to case prevalence? (2) At the time of initial presentation, how do HPS patients differ from EDPS patients with respect to demographics, comorbidities, infectious etiologies, clinical presentations, and severity of illness (3) If holding observed baseline factors constant, does the physical location of sepsis presentation inherently increase the risk for treatment delays and mortality? (4) To what extent can differences in the likelihood for timely initial treatment between the ED and inpatient settings explain differences in mortality and patient outcomes?

We hypothesized a priori that HPS would reflect chronically sicker patients whom both received less timely resuscitation and who contributed disproportionately frequent bad outcomes. We expected disparities in timely resuscitation care would explain a large proportion of this difference.

We performed a retrospective analysis of the Northwell Sepsis Database, a prospectively captured, multisite, real world, consecutive-sample cohort of all “severe sepsis” and septic shock patients treated at nine tertiary and community hospitals in New York from October 1, 2014, to March 31, 2016. We analyzed all patients from a previously published cohort.¹¹

Database Design and Structure

The Northwell Sepsis Database has previously been described in detail.^11,13,14 Briefly, all patients met clinical sepsis criteria: (1) infection AND (2) ≥2 (SIRS) criteria AND (3) ≥1 acute organ dysfunction criterion. Organ dysfunction criteria were hypotension, acute kidney injury (AKI), coagulopathy, altered gas exchange, elevated bilirubin (≥2.0 mg/dL), or altered mental status (AMS; clarified in Supplemental Table 1). All organ dysfunction was not otherwise explained by patients’ medical histories; eg, patients on warfarin anticoagulation were not documented to have coagulopathy based on international normalized ratio > 1.5. The time of the sepsis episode (and database inclusion) was the time of the first vital sign measurement or laboratory result where a patient simultaneously met all three inclusion criteria: infection, SIRS, and organ dysfunction. The database excludes patients who were <18 years, declined bundle interventions, had advance directives precluding interventions, or were admitted directly to palliative care or hospice. Abstractors assumed comorbidities were absent if not documented within the medical record and that physiologic abnormalities were absent if not measured by the treatment team. There were no missing data for the variables analyzed. We report analysis in adherence with the STROBE statement guidelines for observational research.

Exposure

The primary exposure was whether patients had EDPS versus HPS. We defined EDPS patients as meeting all objective clinical inclusion criteria while physically in the ED. We defined HPS as first meeting sepsis inclusion criteria outside the ED, regardless of the reason for admission, and regardless of whether patients were admitted through the ED or directly to the hospital. All ED patients were admitted to the hospital.

Outcomes

Process outcomes were full 3-hour bundle compliance, time to antibiotic administration, blood cultures before antibiotics, time to fluid initiation, the volume of administered fluid resuscitation, lactate result time, and whether repeat lactate was obtained (Supplemental Table 2). Treatment times were times of administration (rather than order time). The primary patient outcome was hospital mortality. Secondary patient outcomes were mechanical ventilation, ICU admission, ICU days, hospital length of stay (LOS). We discounted HPS patients’ LOS to include only days after meeting the inclusion criteria. Patients were excluded from the analysis of the ICU admission outcome if they were already in the ICU prior to meeting sepsis criteria.

Statistical Analysis

We report continuous variables as means (standard deviation) or medians (interquartile range), and categorical variables as frequencies (proportions), as appropriate. Summative statistics with 95% confidence intervals (CI) describe overall group contributions. We used generalized linear models to determine patient factors associated with EDPS versus HPS, entering random effects for individual study sites to control for intercenter variability.

Next, to generate a propensity-matched cohort, we computed propensity scores adjusted from a priori selected variables: age, sex, tertiary versus community hospital, congestive heart failure (CHF), renal failure, COPD, diabetes, liver failure, immunocompromise, primary source of infection, nosocomial source, temperature, initial lactate, presenting hypotension, altered gas exchange, AMS, AKI, and coagulopathy. We then matched subjects 1:1 without optimization or replacement, imposing a caliper width of 0.01; ie, we required matched pairs to have a <1.0% difference in propensity scores. The macro used to match subjects is publically available.¹⁵

We then compared resuscitation and patient outcomes in the matched cohort using generalized linear models, ie, doubly-robust estimation (DRE).¹⁶ When assessing patient outcomes corrected for resuscitation, we used mixed DRE/multivariable regression. We did this for two reasons: first, DRE has the advantage of only requiring only one approach (propensity vs covariate adjustments) to be correctly specified.¹⁶ Second, computing propensity scores adjusted for resuscitation would be inappropriate given that resuscitation occurs after the exposure allocation (HPS vs EDPS). However, these factors could still impact the outcome and in fact, we hypothesized they were potential mediators of the exposure effect. To interrogate this mediating relationship, we recapitulated the DRE modeling but added covariates for resuscitation factors. Resuscitation-adjusted models controlled for timeliness of antibiotics, fluids, and lactate results; blood cultures before antibiotics; repeat lactate obtained, and fluid volume in the first six hours. Since ICU days and LOS are subject to competing risks bias (LOS could be shorter if patients died earlier), we used proportional hazards models where “the event” was defined as a live discharge to censor for mortality and we report output as inverse hazard ratios. We also tested interaction coefficients for discrete bundle elements and HPS to determine if specific bundle elements were effect modifiers for the association between the presenting location and mortality risk. Finally, we estimated attributable risk differences by comparing adjusted odds ratios of adverse outcome with and without adjustment for resuscitation variables, as described by Sahai et al.¹⁷

As sensitivity analyses, we recomputed propensity scores and generated a new matched cohort that excluded HPS patients who met criteria for sepsis while already in the ICU for another reason (ie, excluding ICU-presenting sepsis). We then recapitulated all analyses as above for this cohort. We performed analyses using SAS version 9.4 (SAS Institute, Cary, North Carolina).

Prevalence and Outcome Contributions

Of the 11,182 sepsis patients in the database, we classified 2,509 (22%) as HPS (Figure 1). HPS contributed 785 (35%) of 2,241 sepsis-related mortalities, 1,241 (38%) mechanical ventilations, and 1,762 (34%) ICU admissions. Of 39,263 total ICU days and 127,178 hospital days, HPS contributed 18,104 (46.1%) and 44,412 (34.9%) days, respectively.

Patient Characteristics

Most HPS presented early in the hospital course, with 1,352 (53.9%) cases meeting study criteria within three days of admission. Median time from admission to meeting study criteria for HPS was two days (interquartile range: one to seven days). We report selected baseline patient characteristics in Table 1 and adjusted associations of baseline variables with HPS versus EDPS in Table 2. The full cohort characterization is available in Supplemental Table 3. Notably, HPS patients more often had CHF (aOR [adjusted odds ratio}: 1.31, CI: 1.18-1.47) or renal failure (aOR: 1.62, CI: 1.38-1.91), gastrointestinal source of infection (aOR: 1.84, CI: 1.48-2.29), hypothermia (aOR: 1.56, CI: 1.28-1.90) hypotension (aOR: 1.85, CI: 1.65-2.08), or altered gas exchange (aOR: 2.46, CI: 1.43-4.24). In contrast, HPS patients less frequently were admitted from skilled nursing facilities (aOR: 0.44, CI: 0.32-0.60), or had COPD (aOR: 0.53, CI: 0.36-0.76), fever (aOR: 0.70, CI: 0.52-0.91), tachypnea (aOR: 0.76, CI: 0.58-0.98), or AKI (aOR: 082, CI: 0.68-0.97). Other baseline variables were similar, including respiratory source, tachycardia, white cell abnormalities, AMS, and coagulopathies. These associations were preserved in the sensitivity analysis excluding ICU-presenting sepsis.

Propensity Matching

Propensity score matching yielded 1,942 matched pairs (n = 3,884, 77% of HPS patients, 22% of EDPS patients). Table 1 and Supplemental Table 3 show patient characteristics after propensity matching. Supplemental Table 4 shows the propensity model. The frequency densities are shown for the cohort as a function of propensity score in Supplemental Figure 1. After matching, frequencies between groups differed by <5% for all categorical variables assessed. In the sensitivity analysis, propensity matching (model in Supplemental Table 5) resulted in 1,233 matched pairs (n = 2,466, 49% of HPS patients, 14% of EDPS patients), with group differences comparable to the primary analysis.

Process Outcomes

We present propensity-matched differences in initial resuscitation in Figure 2A for all HPS patients, as well as non-ICU-presenting HPS, versus EDPS. HPS patients were roughly half as likely to receive fully 3-hour bundle compliant care (17.0% vs 30.3%, aOR: 0.47, CI: 0.40-0.57), to have blood cultures drawn within three hours prior to antibiotics (44.9% vs 67.2%, aOR: 0.40, CI: 0.35-0.46), or to receive fluid resuscitation initiated within two hours (11.1% vs 26.1%, aOR: 0.35, CI: 0.29-0.42). Antibiotic receipt within one hour was comparable (45.3% vs 48.1%, aOR: 0.89, CI: 0.79-1.01). However, differences emerged for antibiotics within three hours (66.2% vs 83.8%, aOR: 0.38, CI: 0.32-0.44) and persisted at six hours (77.0% vs 92.5%, aOR: 0.27, CI: 0.22-33). Excluding ICU-presenting sepsis from propensity matching exaggerated disparities in antibiotic receipt at one hour (43.4% vs 49.1%, aOR: 0.80, CI: 0.68-0.93), three hours (64.2% vs 86.1%, aOR: 0.29, CI: 0.24-0.35), and six hours (75.7% vs 93.0%, aOR: 0.23, CI: 0.18-0.30). HPS patients more frequently had repeat lactate obtained within 24 hours (62.4% vs 54.3%, aOR: 1.40, CI: 1.23-1.59).

Patient Outcomes

HPS patients had higher mortality (31.2% vs19.3%), mechanical ventilation (51.5% vs27.4%), and ICU admission (60.6% vs 46.5%) (Table 1 and Supplemental Table 6). Figure 2b shows propensity-matched and covariate-adjusted differences in patient outcomes before and after adjusting for initial resuscitation. aORs corresponded to approximate relative risk differences¹⁸ of 1.38 (CI: 1.28-1.48), 1.68 (CI: 1.57-1.79), and 1.72 (CI: 1.61-1.84) for mortality, mechanical ventilation, and ICU admission, respectively. HPS was associated with 83% longer mortality-censored ICU stays (five vs nine days, HR^–1: 1.83, CI: 1.65-2.03), and 108% longer hospital stay (eight vs 17 days, HR^–1: 2.08, CI: 1.93-2.24). After adjustment for resuscitation, all effect sizes decreased but persisted. The initial crystalloid volume was a significant negative effect modifier for mortality (Supplemental Table 7). That is, the magnitude of the association between HPS and greater mortality decreased by a factor of 0.89 per 10 mL/kg given (CI: 0.82-0.97). We did not observe significant interaction from other interventions, or overall bundle compliance, meaning these interventions’ association with mortality did not significantly differ between HPS versus EDPS.

The implied attributable risk difference from discrepancies in initial resuscitation was 23.3% for mortality, 35.2% for mechanical ventilation, and 7.6% for ICU admission (Figure 2B). Resuscitation explained 26.5% of longer ICU LOS and 16.7% of longer hospital LOS associated with HPS.

Figure 2C shows sensitivity analysis excluding ICU-presenting sepsis from propensity matching (ie, limiting HPS to hospital ward presentations). Again, HPS was associated with all adverse outcomes, though effect sizes were smaller than in the primary cohort for all outcomes except hospital LOS. In this cohort, resuscitation factors now explained 16.5% of HPS’ association with mortality, and 14.5% of the association with longer ICU LOS. However, they explained a greater proportion (13.0%) of ICU admissions. Attributable risk differences were comparable to the primary cohort for mechanical ventilation (37.6%) and hospital LOS (15.3%).

In this analysis of 11,182 sepsis and septic shock patients, HPS contributed 22% of prevalence but >35% of total sepsis mortalities, ICU utilization, and hospital days. HPS patients had higher comorbidity burdens and had clinical presentations less obviously attributable to infection with more severe organ dysfunction. EDPS received antibiotics within three hours about 1.62 times more often than do HPS patients. EDPS patients also receive fluids initiated within two hours about 1.82 times more often than HPS patients do. HPS had nearly 1.5-fold greater mortality and LOS, and nearly two-fold greater mechanical ventilation and ICU utilization. Resuscitation disparities could partially explain these associations. These patterns persisted when comparing only wards presenting HPS with EDPS.

Our analysis revealed several notable findings. First, these data confirm that HPS represents a potentially high-impact target population that contributes adverse outcomes disproportionately frequently with respect to case prevalence.

Our findings, unsurprisingly, revealed HPS and EDPS reflect dramatically different patient populations. We found that the two groups significantly differed by the majority of the baseline factors we compared. It may be worth asking if and how these substantial differences in illness etiology, chronic health, and acute physiology impact what we consider an optimal approach to management. Significant interaction effects of fluid volume on the association between HPS and mortality suggest differential treatment effects may exist between patients. Indeed, patients who newly arrive from the community and those who are several days into admission likely have different volume status. However, no interactions were noted with other bundle elements, such as timeliness of antibiotics or timeliness of initial fluids.

Another potentially concerning observation was that HPS patients were admitted much less frequently from skilled nursing facilities, as it could imply that this poorer-fairing population had a comparatively higher baseline functional status. The fact that 25% of EDPS cases were admitted from these facilities also underscores the need to engage skilled nursing facility providers in future sepsis initiatives.

We found marked disparities in resuscitation. Timely delivery of interventions, such as antibiotics and initial fluid resuscitation, occurred less than half as often for HPS, especially on hospital wards. While evidence supporting the efficacy of specific 3-hour bundle elements remains unsettled,¹⁹ a wealth of literature demonstrates a correlation between bundle uptake and decreased sepsis mortality, especially for early antibiotic administration.^13,20-26 Some analysis suggests that differing initial resuscitation practices explain different mortality rates in the early goal-directed therapy trials.²⁷ The comparatively poor performance for non-ICU HPS indicates further QI efforts are better focused on inpatient wards, rather than on EDs or ICUs where resuscitation is already delivered with substantially greater fidelity.

While resuscitation differences partially explained outcome discrepancies between groups, they did not account for as much variation as expected. Though resuscitation accounted for >35% of attributable mechanical ventilation risk, it explained only 16.5% of mortality differences for non-ICU HPS vs EDPS. We speculate that several factors may contribute.

First, HPS patients are already hospitalized for another acute insult and may be too physiologically brittle to derive equal benefit from initial resuscitation. Some literature suggests protocolized sepsis resuscitation may paradoxically be more effective in milder/earlier disease.²⁸

Second, clinical information indicating septic organ dysfunction may become available too late in HPS—a possible data limitation where inpatient providers are counterintuitively more likely to miss early signs of patients’ deterioration and a subsequent therapeutic window. Several studies found that fluid resuscitation is associated with improved sepsis outcomes only when it is administered very early.^11,29-31 In inpatient wards, decreased monitoring³² and human factors (eg, hospital workflow, provider-to-patient ratios, electronic documentation burdens)^33,34 may hinder early diagnosis. In contrast, ED environments are explicitly designed to identify acutely ill patients and deliver intervention rapidly. If HPS patients were sicker when they were identified, this would also explain their more severe organ dysfunctions. Our data seems to support this possibility. HPS patients had tachypnea less frequently but more often had impaired gas exchange. This finding may suggest that early tachypnea was either less often detected or documented, or that it had progressed further by the time of detection.

Third, inpatients with sepsis may more often present with greater diagnostic complexity. We observed that HPS patients were more often euthermic and less often tachypneic. Beyond suggesting a greater diagnostic challenge, this also raises questions as to whether differences reflect patient physiology (response to infection) or iatrogenic factors (eg, prior antipyretics). Higher comorbidity and acute physiological burdens also limit the degree to which new organ dysfunction can be clearly attributed to infection. We note differences in the proportion of patients who received antibiotics increased over time, suggesting that HPS patients who received delayed antibiotics did so much later than their EDPS counterparts. This lag could also arise from diagnostic difficulty.

All three possibilities highlight a potential lead time effect, where the same measured three-hour period on the wards, between meeting sepsis criteria and starting treatment, actually reflects a longer period between (as yet unmeasurable) pathobiologic “time zero” and treatment versus the ED. The time of sepsis detection, as distinct from the time of sepsis onset, therefore proves difficult to evaluate and impossible to account for statistically.

Regardless, our findings suggest additional difficulty in both the recognition and resuscitation of inpatient sepsis. Inpatients, especially with infections, may need closer monitoring. How to cost effectively implement this monitoring is a challenge that deserves attention.

A more rational systems approach to HPS likely combines efforts to improve initial resuscitation with other initiatives aimed at both improving monitoring and preventing infection.

To be clear, we do not imply that timely initial resuscitation does not matter on the wards. Rather, resuscitation-focused QI alone does not appear to be sufficient to overcome differences in outcomes for HPS. The 23.3% attributable mortality risk we observed still implies that resuscitation differences could explain nearly one in four excess HPS mortalities. We previously showed that timely resuscitation is strongly associated with better outcomes.^11,13,30 As discussed above, the unclear degree to which better resuscitation is a marker for more obvious presentations is a persistent limitation of prior investigations and the present study.

Taken together, the ultimate question that this study raises but cannot answer is whether the timely recognition of sepsis, rather than any specific treatment, is what truly improves outcomes.

In addition to those above, this study has several limitations. Our study did not differentiate HPS with respect to patients admitted for noninfectious reasons and who subsequently became septic versus nonseptic patients admitted for an infection who subsequently became septic from that infection. Nor could we discriminate between missed ED diagnoses and true delayed presentations. We note distinguishing these entities clinically can be equally challenging. Additionally, this was a propensity-matched retrospective analysis of an existing sepsis cohort, and the many limitations of both retrospective study and propensity matching apply.^35,36 We note that randomizing patients to develop sepsis in the community versus hospital is not feasible and that two of our aims intended to describe overall patterns rather than causal effects. We could not ascertain robust measures of severity of illness (eg, SOFA) because a real world setting precludes required data points—eg, urine output is unreliably recorded. We also note incomplete overlap between inclusion criteria and either Sepsis-2 or -3 definitions,^1,37 because we designed and populated our database prior to publication of Sepsis-3. Further, we could not account for surgical source control, the appropriateness of antimicrobial therapy, mechanical ventilation before sepsis onset, or most treatments given after initial resuscitation.

In conclusion, hospital-presenting sepsis accounted for adverse patient outcomes disproportionately to prevalence. HPS patients had more complex presentations, received timely antibiotics half as often ED-presenting sepsis, and had nearly twice the mortality odds. Resuscitation disparities explained roughly 25% of this difference.

Disclosures

The authors have no conflicts of interest to disclose.

Funding

This investigation was funded in part by a grant from the Center for Medicare and Medicaid Innovation to the High Value Healthcare Collaborative, of which the study sites’ umbrella health system was a part. This grant helped fund the underlying QI program and database in this study.

Sepsis is both the most expensive condition treated and the most common cause of death in hospitals in the United States.^1-3 Most sepsis patients (as many as 80% to 90%) meet sepsis criteria on hospital arrival, but mortality and costs are higher when meeting criteria after admission.^3-6 Mechanisms of this increased mortality for these distinct populations are not well explored. Patients who present septic in the emergency department (ED) and patients who present as inpatients likely present very different challenges for recognition, treatment, and monitoring.⁷ Yet, how these groups differ by demographic and clinical characteristics, the etiology and severity of infection, and patterns of resuscitation care are not well described. Literature on sepsis epidemiology on hospital wards is particularly limited.⁸

This knowledge gap is important. If hospital-presenting sepsis (HPS) contributes disproportionately to disease burdCHFens, it reflects a high-yield population deserving the focus of quality improvement (QI) initiatives. If specific causes of disparities were identified—eg, poor initial resuscitation— they could be specifically targeted for correction. Given that current treatment guidelines are uniform for the two populations,^9,10 characterizing phenotypic differences could also have implications for both diagnostic and therapeutic recommendations, particularly if the groups display substantially differing clinical presentations. Our prior work has not probed these effects specifically, but suggested ED versus inpatient setting at the time of initial sepsis presentation might be an effect modifier for the association between several elements of fluid resuscitation and patient outcomes.^11,12

We, therefore, conducted a retrospective analysis to ask four sequential questions: (1) Do patients with HPS, compared with EDPS, contribute adverse outcome out of proportion to case prevalence? (2) At the time of initial presentation, how do HPS patients differ from EDPS patients with respect to demographics, comorbidities, infectious etiologies, clinical presentations, and severity of illness (3) If holding observed baseline factors constant, does the physical location of sepsis presentation inherently increase the risk for treatment delays and mortality? (4) To what extent can differences in the likelihood for timely initial treatment between the ED and inpatient settings explain differences in mortality and patient outcomes?

We hypothesized a priori that HPS would reflect chronically sicker patients whom both received less timely resuscitation and who contributed disproportionately frequent bad outcomes. We expected disparities in timely resuscitation care would explain a large proportion of this difference.

We performed a retrospective analysis of the Northwell Sepsis Database, a prospectively captured, multisite, real world, consecutive-sample cohort of all “severe sepsis” and septic shock patients treated at nine tertiary and community hospitals in New York from October 1, 2014, to March 31, 2016. We analyzed all patients from a previously published cohort.¹¹

Database Design and Structure

The Northwell Sepsis Database has previously been described in detail.^11,13,14 Briefly, all patients met clinical sepsis criteria: (1) infection AND (2) ≥2 (SIRS) criteria AND (3) ≥1 acute organ dysfunction criterion. Organ dysfunction criteria were hypotension, acute kidney injury (AKI), coagulopathy, altered gas exchange, elevated bilirubin (≥2.0 mg/dL), or altered mental status (AMS; clarified in Supplemental Table 1). All organ dysfunction was not otherwise explained by patients’ medical histories; eg, patients on warfarin anticoagulation were not documented to have coagulopathy based on international normalized ratio > 1.5. The time of the sepsis episode (and database inclusion) was the time of the first vital sign measurement or laboratory result where a patient simultaneously met all three inclusion criteria: infection, SIRS, and organ dysfunction. The database excludes patients who were <18 years, declined bundle interventions, had advance directives precluding interventions, or were admitted directly to palliative care or hospice. Abstractors assumed comorbidities were absent if not documented within the medical record and that physiologic abnormalities were absent if not measured by the treatment team. There were no missing data for the variables analyzed. We report analysis in adherence with the STROBE statement guidelines for observational research.

Exposure

The primary exposure was whether patients had EDPS versus HPS. We defined EDPS patients as meeting all objective clinical inclusion criteria while physically in the ED. We defined HPS as first meeting sepsis inclusion criteria outside the ED, regardless of the reason for admission, and regardless of whether patients were admitted through the ED or directly to the hospital. All ED patients were admitted to the hospital.

Outcomes

Process outcomes were full 3-hour bundle compliance, time to antibiotic administration, blood cultures before antibiotics, time to fluid initiation, the volume of administered fluid resuscitation, lactate result time, and whether repeat lactate was obtained (Supplemental Table 2). Treatment times were times of administration (rather than order time). The primary patient outcome was hospital mortality. Secondary patient outcomes were mechanical ventilation, ICU admission, ICU days, hospital length of stay (LOS). We discounted HPS patients’ LOS to include only days after meeting the inclusion criteria. Patients were excluded from the analysis of the ICU admission outcome if they were already in the ICU prior to meeting sepsis criteria.

Statistical Analysis

We report continuous variables as means (standard deviation) or medians (interquartile range), and categorical variables as frequencies (proportions), as appropriate. Summative statistics with 95% confidence intervals (CI) describe overall group contributions. We used generalized linear models to determine patient factors associated with EDPS versus HPS, entering random effects for individual study sites to control for intercenter variability.

Next, to generate a propensity-matched cohort, we computed propensity scores adjusted from a priori selected variables: age, sex, tertiary versus community hospital, congestive heart failure (CHF), renal failure, COPD, diabetes, liver failure, immunocompromise, primary source of infection, nosocomial source, temperature, initial lactate, presenting hypotension, altered gas exchange, AMS, AKI, and coagulopathy. We then matched subjects 1:1 without optimization or replacement, imposing a caliper width of 0.01; ie, we required matched pairs to have a <1.0% difference in propensity scores. The macro used to match subjects is publically available.¹⁵

We then compared resuscitation and patient outcomes in the matched cohort using generalized linear models, ie, doubly-robust estimation (DRE).¹⁶ When assessing patient outcomes corrected for resuscitation, we used mixed DRE/multivariable regression. We did this for two reasons: first, DRE has the advantage of only requiring only one approach (propensity vs covariate adjustments) to be correctly specified.¹⁶ Second, computing propensity scores adjusted for resuscitation would be inappropriate given that resuscitation occurs after the exposure allocation (HPS vs EDPS). However, these factors could still impact the outcome and in fact, we hypothesized they were potential mediators of the exposure effect. To interrogate this mediating relationship, we recapitulated the DRE modeling but added covariates for resuscitation factors. Resuscitation-adjusted models controlled for timeliness of antibiotics, fluids, and lactate results; blood cultures before antibiotics; repeat lactate obtained, and fluid volume in the first six hours. Since ICU days and LOS are subject to competing risks bias (LOS could be shorter if patients died earlier), we used proportional hazards models where “the event” was defined as a live discharge to censor for mortality and we report output as inverse hazard ratios. We also tested interaction coefficients for discrete bundle elements and HPS to determine if specific bundle elements were effect modifiers for the association between the presenting location and mortality risk. Finally, we estimated attributable risk differences by comparing adjusted odds ratios of adverse outcome with and without adjustment for resuscitation variables, as described by Sahai et al.¹⁷

As sensitivity analyses, we recomputed propensity scores and generated a new matched cohort that excluded HPS patients who met criteria for sepsis while already in the ICU for another reason (ie, excluding ICU-presenting sepsis). We then recapitulated all analyses as above for this cohort. We performed analyses using SAS version 9.4 (SAS Institute, Cary, North Carolina).

Prevalence and Outcome Contributions

Of the 11,182 sepsis patients in the database, we classified 2,509 (22%) as HPS (Figure 1). HPS contributed 785 (35%) of 2,241 sepsis-related mortalities, 1,241 (38%) mechanical ventilations, and 1,762 (34%) ICU admissions. Of 39,263 total ICU days and 127,178 hospital days, HPS contributed 18,104 (46.1%) and 44,412 (34.9%) days, respectively.

Patient Characteristics

Most HPS presented early in the hospital course, with 1,352 (53.9%) cases meeting study criteria within three days of admission. Median time from admission to meeting study criteria for HPS was two days (interquartile range: one to seven days). We report selected baseline patient characteristics in Table 1 and adjusted associations of baseline variables with HPS versus EDPS in Table 2. The full cohort characterization is available in Supplemental Table 3. Notably, HPS patients more often had CHF (aOR [adjusted odds ratio}: 1.31, CI: 1.18-1.47) or renal failure (aOR: 1.62, CI: 1.38-1.91), gastrointestinal source of infection (aOR: 1.84, CI: 1.48-2.29), hypothermia (aOR: 1.56, CI: 1.28-1.90) hypotension (aOR: 1.85, CI: 1.65-2.08), or altered gas exchange (aOR: 2.46, CI: 1.43-4.24). In contrast, HPS patients less frequently were admitted from skilled nursing facilities (aOR: 0.44, CI: 0.32-0.60), or had COPD (aOR: 0.53, CI: 0.36-0.76), fever (aOR: 0.70, CI: 0.52-0.91), tachypnea (aOR: 0.76, CI: 0.58-0.98), or AKI (aOR: 082, CI: 0.68-0.97). Other baseline variables were similar, including respiratory source, tachycardia, white cell abnormalities, AMS, and coagulopathies. These associations were preserved in the sensitivity analysis excluding ICU-presenting sepsis.

Propensity Matching

Propensity score matching yielded 1,942 matched pairs (n = 3,884, 77% of HPS patients, 22% of EDPS patients). Table 1 and Supplemental Table 3 show patient characteristics after propensity matching. Supplemental Table 4 shows the propensity model. The frequency densities are shown for the cohort as a function of propensity score in Supplemental Figure 1. After matching, frequencies between groups differed by <5% for all categorical variables assessed. In the sensitivity analysis, propensity matching (model in Supplemental Table 5) resulted in 1,233 matched pairs (n = 2,466, 49% of HPS patients, 14% of EDPS patients), with group differences comparable to the primary analysis.

Process Outcomes

We present propensity-matched differences in initial resuscitation in Figure 2A for all HPS patients, as well as non-ICU-presenting HPS, versus EDPS. HPS patients were roughly half as likely to receive fully 3-hour bundle compliant care (17.0% vs 30.3%, aOR: 0.47, CI: 0.40-0.57), to have blood cultures drawn within three hours prior to antibiotics (44.9% vs 67.2%, aOR: 0.40, CI: 0.35-0.46), or to receive fluid resuscitation initiated within two hours (11.1% vs 26.1%, aOR: 0.35, CI: 0.29-0.42). Antibiotic receipt within one hour was comparable (45.3% vs 48.1%, aOR: 0.89, CI: 0.79-1.01). However, differences emerged for antibiotics within three hours (66.2% vs 83.8%, aOR: 0.38, CI: 0.32-0.44) and persisted at six hours (77.0% vs 92.5%, aOR: 0.27, CI: 0.22-33). Excluding ICU-presenting sepsis from propensity matching exaggerated disparities in antibiotic receipt at one hour (43.4% vs 49.1%, aOR: 0.80, CI: 0.68-0.93), three hours (64.2% vs 86.1%, aOR: 0.29, CI: 0.24-0.35), and six hours (75.7% vs 93.0%, aOR: 0.23, CI: 0.18-0.30). HPS patients more frequently had repeat lactate obtained within 24 hours (62.4% vs 54.3%, aOR: 1.40, CI: 1.23-1.59).

Patient Outcomes

HPS patients had higher mortality (31.2% vs19.3%), mechanical ventilation (51.5% vs27.4%), and ICU admission (60.6% vs 46.5%) (Table 1 and Supplemental Table 6). Figure 2b shows propensity-matched and covariate-adjusted differences in patient outcomes before and after adjusting for initial resuscitation. aORs corresponded to approximate relative risk differences¹⁸ of 1.38 (CI: 1.28-1.48), 1.68 (CI: 1.57-1.79), and 1.72 (CI: 1.61-1.84) for mortality, mechanical ventilation, and ICU admission, respectively. HPS was associated with 83% longer mortality-censored ICU stays (five vs nine days, HR^–1: 1.83, CI: 1.65-2.03), and 108% longer hospital stay (eight vs 17 days, HR^–1: 2.08, CI: 1.93-2.24). After adjustment for resuscitation, all effect sizes decreased but persisted. The initial crystalloid volume was a significant negative effect modifier for mortality (Supplemental Table 7). That is, the magnitude of the association between HPS and greater mortality decreased by a factor of 0.89 per 10 mL/kg given (CI: 0.82-0.97). We did not observe significant interaction from other interventions, or overall bundle compliance, meaning these interventions’ association with mortality did not significantly differ between HPS versus EDPS.

The implied attributable risk difference from discrepancies in initial resuscitation was 23.3% for mortality, 35.2% for mechanical ventilation, and 7.6% for ICU admission (Figure 2B). Resuscitation explained 26.5% of longer ICU LOS and 16.7% of longer hospital LOS associated with HPS.

Figure 2C shows sensitivity analysis excluding ICU-presenting sepsis from propensity matching (ie, limiting HPS to hospital ward presentations). Again, HPS was associated with all adverse outcomes, though effect sizes were smaller than in the primary cohort for all outcomes except hospital LOS. In this cohort, resuscitation factors now explained 16.5% of HPS’ association with mortality, and 14.5% of the association with longer ICU LOS. However, they explained a greater proportion (13.0%) of ICU admissions. Attributable risk differences were comparable to the primary cohort for mechanical ventilation (37.6%) and hospital LOS (15.3%).

In this analysis of 11,182 sepsis and septic shock patients, HPS contributed 22% of prevalence but >35% of total sepsis mortalities, ICU utilization, and hospital days. HPS patients had higher comorbidity burdens and had clinical presentations less obviously attributable to infection with more severe organ dysfunction. EDPS received antibiotics within three hours about 1.62 times more often than do HPS patients. EDPS patients also receive fluids initiated within two hours about 1.82 times more often than HPS patients do. HPS had nearly 1.5-fold greater mortality and LOS, and nearly two-fold greater mechanical ventilation and ICU utilization. Resuscitation disparities could partially explain these associations. These patterns persisted when comparing only wards presenting HPS with EDPS.

Our analysis revealed several notable findings. First, these data confirm that HPS represents a potentially high-impact target population that contributes adverse outcomes disproportionately frequently with respect to case prevalence.

Our findings, unsurprisingly, revealed HPS and EDPS reflect dramatically different patient populations. We found that the two groups significantly differed by the majority of the baseline factors we compared. It may be worth asking if and how these substantial differences in illness etiology, chronic health, and acute physiology impact what we consider an optimal approach to management. Significant interaction effects of fluid volume on the association between HPS and mortality suggest differential treatment effects may exist between patients. Indeed, patients who newly arrive from the community and those who are several days into admission likely have different volume status. However, no interactions were noted with other bundle elements, such as timeliness of antibiotics or timeliness of initial fluids.

Another potentially concerning observation was that HPS patients were admitted much less frequently from skilled nursing facilities, as it could imply that this poorer-fairing population had a comparatively higher baseline functional status. The fact that 25% of EDPS cases were admitted from these facilities also underscores the need to engage skilled nursing facility providers in future sepsis initiatives.

We found marked disparities in resuscitation. Timely delivery of interventions, such as antibiotics and initial fluid resuscitation, occurred less than half as often for HPS, especially on hospital wards. While evidence supporting the efficacy of specific 3-hour bundle elements remains unsettled,¹⁹ a wealth of literature demonstrates a correlation between bundle uptake and decreased sepsis mortality, especially for early antibiotic administration.^13,20-26 Some analysis suggests that differing initial resuscitation practices explain different mortality rates in the early goal-directed therapy trials.²⁷ The comparatively poor performance for non-ICU HPS indicates further QI efforts are better focused on inpatient wards, rather than on EDs or ICUs where resuscitation is already delivered with substantially greater fidelity.

While resuscitation differences partially explained outcome discrepancies between groups, they did not account for as much variation as expected. Though resuscitation accounted for >35% of attributable mechanical ventilation risk, it explained only 16.5% of mortality differences for non-ICU HPS vs EDPS. We speculate that several factors may contribute.

First, HPS patients are already hospitalized for another acute insult and may be too physiologically brittle to derive equal benefit from initial resuscitation. Some literature suggests protocolized sepsis resuscitation may paradoxically be more effective in milder/earlier disease.²⁸

Second, clinical information indicating septic organ dysfunction may become available too late in HPS—a possible data limitation where inpatient providers are counterintuitively more likely to miss early signs of patients’ deterioration and a subsequent therapeutic window. Several studies found that fluid resuscitation is associated with improved sepsis outcomes only when it is administered very early.^11,29-31 In inpatient wards, decreased monitoring³² and human factors (eg, hospital workflow, provider-to-patient ratios, electronic documentation burdens)^33,34 may hinder early diagnosis. In contrast, ED environments are explicitly designed to identify acutely ill patients and deliver intervention rapidly. If HPS patients were sicker when they were identified, this would also explain their more severe organ dysfunctions. Our data seems to support this possibility. HPS patients had tachypnea less frequently but more often had impaired gas exchange. This finding may suggest that early tachypnea was either less often detected or documented, or that it had progressed further by the time of detection.

Third, inpatients with sepsis may more often present with greater diagnostic complexity. We observed that HPS patients were more often euthermic and less often tachypneic. Beyond suggesting a greater diagnostic challenge, this also raises questions as to whether differences reflect patient physiology (response to infection) or iatrogenic factors (eg, prior antipyretics). Higher comorbidity and acute physiological burdens also limit the degree to which new organ dysfunction can be clearly attributed to infection. We note differences in the proportion of patients who received antibiotics increased over time, suggesting that HPS patients who received delayed antibiotics did so much later than their EDPS counterparts. This lag could also arise from diagnostic difficulty.

All three possibilities highlight a potential lead time effect, where the same measured three-hour period on the wards, between meeting sepsis criteria and starting treatment, actually reflects a longer period between (as yet unmeasurable) pathobiologic “time zero” and treatment versus the ED. The time of sepsis detection, as distinct from the time of sepsis onset, therefore proves difficult to evaluate and impossible to account for statistically.

Regardless, our findings suggest additional difficulty in both the recognition and resuscitation of inpatient sepsis. Inpatients, especially with infections, may need closer monitoring. How to cost effectively implement this monitoring is a challenge that deserves attention.

A more rational systems approach to HPS likely combines efforts to improve initial resuscitation with other initiatives aimed at both improving monitoring and preventing infection.

To be clear, we do not imply that timely initial resuscitation does not matter on the wards. Rather, resuscitation-focused QI alone does not appear to be sufficient to overcome differences in outcomes for HPS. The 23.3% attributable mortality risk we observed still implies that resuscitation differences could explain nearly one in four excess HPS mortalities. We previously showed that timely resuscitation is strongly associated with better outcomes.^11,13,30 As discussed above, the unclear degree to which better resuscitation is a marker for more obvious presentations is a persistent limitation of prior investigations and the present study.

Taken together, the ultimate question that this study raises but cannot answer is whether the timely recognition of sepsis, rather than any specific treatment, is what truly improves outcomes.

In addition to those above, this study has several limitations. Our study did not differentiate HPS with respect to patients admitted for noninfectious reasons and who subsequently became septic versus nonseptic patients admitted for an infection who subsequently became septic from that infection. Nor could we discriminate between missed ED diagnoses and true delayed presentations. We note distinguishing these entities clinically can be equally challenging. Additionally, this was a propensity-matched retrospective analysis of an existing sepsis cohort, and the many limitations of both retrospective study and propensity matching apply.^35,36 We note that randomizing patients to develop sepsis in the community versus hospital is not feasible and that two of our aims intended to describe overall patterns rather than causal effects. We could not ascertain robust measures of severity of illness (eg, SOFA) because a real world setting precludes required data points—eg, urine output is unreliably recorded. We also note incomplete overlap between inclusion criteria and either Sepsis-2 or -3 definitions,^1,37 because we designed and populated our database prior to publication of Sepsis-3. Further, we could not account for surgical source control, the appropriateness of antimicrobial therapy, mechanical ventilation before sepsis onset, or most treatments given after initial resuscitation.

In conclusion, hospital-presenting sepsis accounted for adverse patient outcomes disproportionately to prevalence. HPS patients had more complex presentations, received timely antibiotics half as often ED-presenting sepsis, and had nearly twice the mortality odds. Resuscitation disparities explained roughly 25% of this difference.

Disclosures

The authors have no conflicts of interest to disclose.

Funding

This investigation was funded in part by a grant from the Center for Medicare and Medicaid Innovation to the High Value Healthcare Collaborative, of which the study sites’ umbrella health system was a part. This grant helped fund the underlying QI program and database in this study.

Anticoagulation is commonly used in the management of atrial fibrillation to reduce the risk of ischemic stroke. Warfarin and other anticoagulants increase the risk of hemorrhagic complications, including upper gastrointestinal bleeding (UGIB). Following UGIB, management of anticoagulation is highly variable. Many patients permanently discontinue anticoagulation, while others continue without interruption.^1-4 Among patients who resume warfarin, different cohorts have measured median times to resumption ranging from four days to 50 days.^1-3 Outcomes data are sparse, and clinical guidelines offer little direction.⁵

Following UGIB, the balance between the risks and benefits of anticoagulation changes over time. Rebleeding risk is highest immediately after the event and declines quickly; therefore, rapid resumption of anticoagulation causes patient harm.³ Meanwhile, the risk of stroke remains constant, and delay in resumption of anticoagulation is associated with increased risk of stroke and death.¹ At some point in time following the initial UGIB, the expected harm from bleeding would equal the expected harm from stroke. This time point would represent the optimal time to restart anticoagulation.

Trial data are unlikely to identify the optimal time for restarting anticoagulation. A randomized trial comparing discrete reinitiation times (eg, two weeks vs six weeks) may easily miss the optimal timing. Moreover, because the daily probability of thromboembolic events is low, large numbers of patients would be required to power such a study. In addition, a number of oral anticoagulants are now approved for prevention of thromboembolic stroke in atrial fibrillation, and each drug may have different optimal timing.

In contrast to randomized trials that would be impracticable for addressing this clinical issue, microsimulation modeling can provide granular information regarding the optimal time to restart anticoagulation. Herein, we set out to estimate the expected benefit of reinitiation of warfarin, the most commonly used oral anticoagulant,⁶ or apixaban, the direct oral anticoagulant with the most favorable risk profile,⁷ as a function of days after UGIB.

We previously described a microsimulation model of anticoagulation among patients with nonvalvular atrial fibrillation (NVAF; hereafter, we refer to this model as the Personalized Anticoagulation Decision-Making Assistance model, or PADMA).^8,9 For this study, we extended this model to incorporate the probability of rebleeding following UGIB and include apixaban as an alternative to warfarin. This model begins with a synthetic population following UGIB, the members of which are at varying risk for thromboembolism, recurrent UGIB, and other hemorrhages. For each patient, the model simulates a number of possible events (eg, thromboembolic stroke, intracranial hemorrhage, rebleeding, and other major extracranial hemorrhages) on each day of an acute period of 90 days after hemostasis. Patients who survive until the end of the acute period enter a simulation with annual, rather than daily, cycles. Our model then estimates total quality-adjusted life-years (QALYs) for each patient, discounted to the present. We report the average discounted QALYs produced by the model for the same population if all individuals in our input population were to resume either warfarin or apixaban on a specific day. Input parameters and ranges are summarized in Table 1, a simplified schematic of our model is shown in the Supplemental Appendix, and additional details regarding model structure and assumptions can be found in earlier work.^8,9 We simulated from a health system perspective over a lifelong time horizon. All analyses were performed in version 14 of Stata (StataCorp, LLC, College Station, Texas).

Synthetic Population

To generate a population reflective of the comorbidities and age distribution of the US population with NVAF, we merged relevant variables from the National Health and Nutrition Examination Survey (NHANES; 2011-2012), using multiple imputation to correct for missing variables.¹⁰ We then bootstrapped to national population estimates by age and sex to arrive at a hypothetical population of the United States.¹¹ Because NHANES does not include atrial fibrillation, we applied sex- and age-specific prevalence rates from the AnTicoagulation and Risk Factors In Atrial Fibrillation study.¹² We then calculated commonly used risk scores (CHA₂DS₂-Vasc and HAS-BLED) for each patient and limited the population to patients with a CHA₂DS₂-Vasc score of one or greater.^13,14 The population resuming apixaban was further limited to patients whose creatinine clearance was 25 mL/min or greater in keeping with the entry criteria in the phase 3 clinical trial on which the medication’s approval was based.¹⁵

To estimate patient-specific probability of rebleeding, we generated a Rockall score for each patient.¹⁶ Although the discrimination of the Rockall score is limited for individual patients, as with all other tools used to predict rebleeding following UGIB, the Rockall score has demonstrated reasonable calibration across a threefold risk gradient.^17-19 International consensus guidelines recommend the Rockall score as one of two risk prediction tools for clinical use in the management of patients with UGIB.²⁰ In addition, because the Rockall score includes some demographic components (five of a possible 11 points), our estimates of rebleeding risk are covariant with other patient-specific risks. We assumed that the endoscopic components of the Rockall score were present in our cohort at the same frequency as in the original derivation and are independent of known patient risk factors.¹⁶ For example, 441 out of 4,025 patients in the original Rockall derivation cohort presented with a systolic blood pressure less than 100 mm Hg. We assumed that an independent and random 10.96% of the cohort would present with shock, which confers two points in the Rockall score.

The population was replicated 60 times, with identical copies of the population resuming anticoagulation on each of days 1-60 (where day zero represents hemostasis). Intermediate data regarding our simulated population can be found in the Supplemental Appendix and in prior work.

Event Type, Severity, and Mortality

Each patient in our simulation could sustain several discrete and independent events: ischemic stroke, intracranial hemorrhage, recurrent UGIB, or extracranial major hemorrhage other than recurrent UGIB. As in prior analyses using the PADMA model, we did not consider minor hemorrhagic events.⁸

The probability of each event was conditional on the corresponding risk scoring system. Patient-specific probability of ischemic stroke was conditional on CHA₂DS₂-Vasc score.^21,22 Patient-specific probability of intracranial hemorrhage was conditional on HAS-BLED score, with the proportions of intracranial hemorrhage of each considered subtype (intracerebral, subarachnoid, or subdural) bootstrapped from previously-published data.^21-24 Patient-specific probability of rebleeding was conditional on Rockall score from the combined Rockall and Vreeburg validation cohorts.¹⁷ Patient-specific probability of extracranial major hemorrhage was conditional on HAS-BLED score.²¹ To avoid double-counting of UGIB, we subtracted the baseline risk of UGIB from the overall rate of extracranial major hemorrhages using previously-published data regarding relative frequency and a bootstrapping approach.²⁵

Probability of Rebleeding Over Time

To estimate the decrease in rebleeding risk over time, we searched the Medline database for systematic reviews of recurrent bleeding following UGIB using the strategy detailed in the Supplemental Appendix. Using the interval rates of rebleeding we identified, we calculated implied daily rates of rebleeding at the midpoint of each interval. For example, 39.5% of rebleeding events occurred within three days of hemostasis, implying a daily rate of approximately 13.2% on day two (32 of 81 events over a three-day period). We repeated this process to estimate daily rates at the midpoint of each reported time interval and fitted an exponential decay function.²⁶ Our exponential fitted these datapoints quite well, but we lacked sufficient data to test other survival functions (eg, Gompertz, lognormal, etc.). Our fitted exponential can be expressed as:

P _rebleeding = b ₀ *exp(b ₁ *day)

where b₀ = 0.1843 (SE: 0.0136) and b₁ = –0.1563 (SE: 0.0188). For example, a mean of 3.9% of rebleeding episodes will occur on day 10 (0.1843 *exp(–0.1563 *10)).

Relative Risks of Events with Anticoagulation

For patients resuming warfarin, the probabilities of each event were adjusted based on patient-specific daily INR. All INRs were assumed to be 1.0 until the day of warfarin reinitiation, after which interpolated trajectories of postinitiation INR measurements were sampled for each patient from an earlier study of clinical warfarin initiation.²⁷ Relative risks of ischemic stroke and hemorrhagic events were calculated based on each day’s INR.

For patients taking apixaban, we assumed that the medication would reach full therapeutic effect one day after reinitiation. Based on available evidence, we applied the relative risks of each event with apixaban compared with warfarin.²⁵

Future Disability and Mortality

Each event in our simulation resulted in hospitalization. Length of stay was sampled for each diagnosis.²⁸ The disutility of hospitalization was estimated based on length of stay.⁸ Inpatient mortality and future disability were predicted for each event as previously described.⁸ We assumed that recurrent episodes of UGIB conferred morbidity and mortality identical to extracranial major hemorrhages more broadly.^29,30

Disutilities

We used a multiplicative model for disutility with baseline utilities conditional on age and sex.³¹ Each day after resumption of anticoagulation carried a disutility of 0.012 for warfarin or 0.002 for apixaban, which we assumed to be equivalent to aspirin in disutility.³² Long-term disutility and life expectancy were conditional on modified Rankin Score (mRS).^33,34 We discounted all QALYs to day zero using standard exponential discounting and a discount rate centered at 3%. We then computed the average discounted QALYs among the cohort of patients that resumed anticoagulation on each day following the index UGIB.

Sensitivity Analyses and Metamodel

To assess sensitivity to continuously varying input parameters, such as discount rate, the proportion of extracranial major hemorrhages that are upper GI bleeds, and inpatient mortality from extracranial major hemorrhage, we constructed a metamodel (a regression model of our microsimulation results).³⁵ We tested for interactions among input parameters and dropped parameters that were not statistically significant predictors of discounted QALYs from our metamodel. We then tested for interactions between each parameter and day resuming anticoagulation to determine which factors may impact the optimal day of reinitiation. Finally, we used predicted marginal effects from our metamodel to assess the change in optimal day across the ranges of each input parameter when other parameters were held at their medians.

Resuming warfarin on day zero produced the fewest QALYs. With delay in reinitiation of anticoagulation, expected QALYs increased, peaked, and then declined for all scenarios. In our base-case simulation of warfarin, peak utility was achieved by resumption 41 days after the index UGIB. Resumption between days 32 and 51 produced greater than 99.9% of peak utility. In our base-case simulation of apixaban, peak utility was achieved by resumption 32 days after the index UGIB. Resumption between days 21 and 47 produced greater than 99.9% of peak utility. Results for warfarin and apixaban are shown in Figures 1 and 2, respectively.

The optimal day of warfarin reinitiation was most sensitive to CHA₂DS₂-Vasc scores and varied by around 11 days between a CHA₂DS₂-Vasc score of one and a CHA₂DS₂-Vasc score of six (the 5^th and 95^th percentiles, respectively) when all other parameters are held at their medians. Results were comparatively insensitive to rebleeding risk. Varying Rockall score from two to seven (the 5^th and 95^th percentiles, respectively) added three days to optimal warfarin resumption. Varying other parameters from the 5^th to the 95^th percentile (including HAS-BLED score, sex, age, and discount rate) changed expected QALYs but did not change the optimal day of reinitiation of warfarin. Optimal day of reinitiation for warfarin stratified by CHA₂DS₂-Vasc score is shown in Table 2.

Anticoagulation is frequently prescribed for patients with NVAF, and hemorrhagic complications are common. Although anticoagulants are withheld following hemorrhages, scant evidence to inform the optimal timing of reinitiation is available. In this microsimulation analysis, we found that the optimal time to reinitiate anticoagulation following UGIB is around 41 days for warfarin and around 32 days for apixaban. We have further demonstrated that the optimal timing of reinitiation can vary by nearly two weeks, depending on a patient’s underlying risk of stroke, and that early reinitiation is more sensitive to rebleeding risk than late reinitiation.

Prior work has shown that early reinitiation of anticoagulation leads to higher rates of recurrent hemorrhage while failure to reinitiate anticoagulation is associated with higher rates of stroke and mortality.^1-4,36 Our results add to the literature in a number of important ways. First, our model not only confirms that anticoagulation should be restarted but also suggests when this action should be taken. The competing risks of bleeding and stroke have left clinicians with little guidance; we have quantified the clinical reasoning required for the decision to resume anticoagulation. Second, by including the disutility of hospitalization and long-term disability, our model more accurately represents the complex tradeoffs between recurrent hemorrhage and (potentially disabling) stroke than would a comparison of event rates. Third, our model is conditional upon patient risk factors, allowing clinicians to personalize the timing of anticoagulation resumption. Theory would suggest that patients at higher risk of ischemic stroke benefit from earlier resumption of anticoagulation, while patients at higher risk of hemorrhage benefit from delayed reinitiation. We have quantified the extent to which patient-specific risks should change timing. Fourth, we offer a means of improving expected health outcomes that requires little more than appropriate scheduling. Current practice regarding resuming anticoagulation is widely variable. Many patients never resume warfarin, and those that do resume do so after highly varied periods of time.^1-5,36 We offer a means of standardizing clinical practice and improving expected patient outcomes.

Disclosures

The authors have nothing to disclose.

Funding

The authors received no specific funding for this work.

Anticoagulation is commonly used in the management of atrial fibrillation to reduce the risk of ischemic stroke. Warfarin and other anticoagulants increase the risk of hemorrhagic complications, including upper gastrointestinal bleeding (UGIB). Following UGIB, management of anticoagulation is highly variable. Many patients permanently discontinue anticoagulation, while others continue without interruption.^1-4 Among patients who resume warfarin, different cohorts have measured median times to resumption ranging from four days to 50 days.^1-3 Outcomes data are sparse, and clinical guidelines offer little direction.⁵

Following UGIB, the balance between the risks and benefits of anticoagulation changes over time. Rebleeding risk is highest immediately after the event and declines quickly; therefore, rapid resumption of anticoagulation causes patient harm.³ Meanwhile, the risk of stroke remains constant, and delay in resumption of anticoagulation is associated with increased risk of stroke and death.¹ At some point in time following the initial UGIB, the expected harm from bleeding would equal the expected harm from stroke. This time point would represent the optimal time to restart anticoagulation.

Trial data are unlikely to identify the optimal time for restarting anticoagulation. A randomized trial comparing discrete reinitiation times (eg, two weeks vs six weeks) may easily miss the optimal timing. Moreover, because the daily probability of thromboembolic events is low, large numbers of patients would be required to power such a study. In addition, a number of oral anticoagulants are now approved for prevention of thromboembolic stroke in atrial fibrillation, and each drug may have different optimal timing.

In contrast to randomized trials that would be impracticable for addressing this clinical issue, microsimulation modeling can provide granular information regarding the optimal time to restart anticoagulation. Herein, we set out to estimate the expected benefit of reinitiation of warfarin, the most commonly used oral anticoagulant,⁶ or apixaban, the direct oral anticoagulant with the most favorable risk profile,⁷ as a function of days after UGIB.

We previously described a microsimulation model of anticoagulation among patients with nonvalvular atrial fibrillation (NVAF; hereafter, we refer to this model as the Personalized Anticoagulation Decision-Making Assistance model, or PADMA).^8,9 For this study, we extended this model to incorporate the probability of rebleeding following UGIB and include apixaban as an alternative to warfarin. This model begins with a synthetic population following UGIB, the members of which are at varying risk for thromboembolism, recurrent UGIB, and other hemorrhages. For each patient, the model simulates a number of possible events (eg, thromboembolic stroke, intracranial hemorrhage, rebleeding, and other major extracranial hemorrhages) on each day of an acute period of 90 days after hemostasis. Patients who survive until the end of the acute period enter a simulation with annual, rather than daily, cycles. Our model then estimates total quality-adjusted life-years (QALYs) for each patient, discounted to the present. We report the average discounted QALYs produced by the model for the same population if all individuals in our input population were to resume either warfarin or apixaban on a specific day. Input parameters and ranges are summarized in Table 1, a simplified schematic of our model is shown in the Supplemental Appendix, and additional details regarding model structure and assumptions can be found in earlier work.^8,9 We simulated from a health system perspective over a lifelong time horizon. All analyses were performed in version 14 of Stata (StataCorp, LLC, College Station, Texas).

Synthetic Population

To generate a population reflective of the comorbidities and age distribution of the US population with NVAF, we merged relevant variables from the National Health and Nutrition Examination Survey (NHANES; 2011-2012), using multiple imputation to correct for missing variables.¹⁰ We then bootstrapped to national population estimates by age and sex to arrive at a hypothetical population of the United States.¹¹ Because NHANES does not include atrial fibrillation, we applied sex- and age-specific prevalence rates from the AnTicoagulation and Risk Factors In Atrial Fibrillation study.¹² We then calculated commonly used risk scores (CHA₂DS₂-Vasc and HAS-BLED) for each patient and limited the population to patients with a CHA₂DS₂-Vasc score of one or greater.^13,14 The population resuming apixaban was further limited to patients whose creatinine clearance was 25 mL/min or greater in keeping with the entry criteria in the phase 3 clinical trial on which the medication’s approval was based.¹⁵

To estimate patient-specific probability of rebleeding, we generated a Rockall score for each patient.¹⁶ Although the discrimination of the Rockall score is limited for individual patients, as with all other tools used to predict rebleeding following UGIB, the Rockall score has demonstrated reasonable calibration across a threefold risk gradient.^17-19 International consensus guidelines recommend the Rockall score as one of two risk prediction tools for clinical use in the management of patients with UGIB.²⁰ In addition, because the Rockall score includes some demographic components (five of a possible 11 points), our estimates of rebleeding risk are covariant with other patient-specific risks. We assumed that the endoscopic components of the Rockall score were present in our cohort at the same frequency as in the original derivation and are independent of known patient risk factors.¹⁶ For example, 441 out of 4,025 patients in the original Rockall derivation cohort presented with a systolic blood pressure less than 100 mm Hg. We assumed that an independent and random 10.96% of the cohort would present with shock, which confers two points in the Rockall score.

The population was replicated 60 times, with identical copies of the population resuming anticoagulation on each of days 1-60 (where day zero represents hemostasis). Intermediate data regarding our simulated population can be found in the Supplemental Appendix and in prior work.

Event Type, Severity, and Mortality

Each patient in our simulation could sustain several discrete and independent events: ischemic stroke, intracranial hemorrhage, recurrent UGIB, or extracranial major hemorrhage other than recurrent UGIB. As in prior analyses using the PADMA model, we did not consider minor hemorrhagic events.⁸

The probability of each event was conditional on the corresponding risk scoring system. Patient-specific probability of ischemic stroke was conditional on CHA₂DS₂-Vasc score.^21,22 Patient-specific probability of intracranial hemorrhage was conditional on HAS-BLED score, with the proportions of intracranial hemorrhage of each considered subtype (intracerebral, subarachnoid, or subdural) bootstrapped from previously-published data.^21-24 Patient-specific probability of rebleeding was conditional on Rockall score from the combined Rockall and Vreeburg validation cohorts.¹⁷ Patient-specific probability of extracranial major hemorrhage was conditional on HAS-BLED score.²¹ To avoid double-counting of UGIB, we subtracted the baseline risk of UGIB from the overall rate of extracranial major hemorrhages using previously-published data regarding relative frequency and a bootstrapping approach.²⁵

Probability of Rebleeding Over Time

To estimate the decrease in rebleeding risk over time, we searched the Medline database for systematic reviews of recurrent bleeding following UGIB using the strategy detailed in the Supplemental Appendix. Using the interval rates of rebleeding we identified, we calculated implied daily rates of rebleeding at the midpoint of each interval. For example, 39.5% of rebleeding events occurred within three days of hemostasis, implying a daily rate of approximately 13.2% on day two (32 of 81 events over a three-day period). We repeated this process to estimate daily rates at the midpoint of each reported time interval and fitted an exponential decay function.²⁶ Our exponential fitted these datapoints quite well, but we lacked sufficient data to test other survival functions (eg, Gompertz, lognormal, etc.). Our fitted exponential can be expressed as:

P _rebleeding = b ₀ *exp(b ₁ *day)

where b₀ = 0.1843 (SE: 0.0136) and b₁ = –0.1563 (SE: 0.0188). For example, a mean of 3.9% of rebleeding episodes will occur on day 10 (0.1843 *exp(–0.1563 *10)).

Relative Risks of Events with Anticoagulation

For patients resuming warfarin, the probabilities of each event were adjusted based on patient-specific daily INR. All INRs were assumed to be 1.0 until the day of warfarin reinitiation, after which interpolated trajectories of postinitiation INR measurements were sampled for each patient from an earlier study of clinical warfarin initiation.²⁷ Relative risks of ischemic stroke and hemorrhagic events were calculated based on each day’s INR.

For patients taking apixaban, we assumed that the medication would reach full therapeutic effect one day after reinitiation. Based on available evidence, we applied the relative risks of each event with apixaban compared with warfarin.²⁵

Future Disability and Mortality

Each event in our simulation resulted in hospitalization. Length of stay was sampled for each diagnosis.²⁸ The disutility of hospitalization was estimated based on length of stay.⁸ Inpatient mortality and future disability were predicted for each event as previously described.⁸ We assumed that recurrent episodes of UGIB conferred morbidity and mortality identical to extracranial major hemorrhages more broadly.^29,30

Disutilities

We used a multiplicative model for disutility with baseline utilities conditional on age and sex.³¹ Each day after resumption of anticoagulation carried a disutility of 0.012 for warfarin or 0.002 for apixaban, which we assumed to be equivalent to aspirin in disutility.³² Long-term disutility and life expectancy were conditional on modified Rankin Score (mRS).^33,34 We discounted all QALYs to day zero using standard exponential discounting and a discount rate centered at 3%. We then computed the average discounted QALYs among the cohort of patients that resumed anticoagulation on each day following the index UGIB.

Sensitivity Analyses and Metamodel

To assess sensitivity to continuously varying input parameters, such as discount rate, the proportion of extracranial major hemorrhages that are upper GI bleeds, and inpatient mortality from extracranial major hemorrhage, we constructed a metamodel (a regression model of our microsimulation results).³⁵ We tested for interactions among input parameters and dropped parameters that were not statistically significant predictors of discounted QALYs from our metamodel. We then tested for interactions between each parameter and day resuming anticoagulation to determine which factors may impact the optimal day of reinitiation. Finally, we used predicted marginal effects from our metamodel to assess the change in optimal day across the ranges of each input parameter when other parameters were held at their medians.

Resuming warfarin on day zero produced the fewest QALYs. With delay in reinitiation of anticoagulation, expected QALYs increased, peaked, and then declined for all scenarios. In our base-case simulation of warfarin, peak utility was achieved by resumption 41 days after the index UGIB. Resumption between days 32 and 51 produced greater than 99.9% of peak utility. In our base-case simulation of apixaban, peak utility was achieved by resumption 32 days after the index UGIB. Resumption between days 21 and 47 produced greater than 99.9% of peak utility. Results for warfarin and apixaban are shown in Figures 1 and 2, respectively.

The optimal day of warfarin reinitiation was most sensitive to CHA₂DS₂-Vasc scores and varied by around 11 days between a CHA₂DS₂-Vasc score of one and a CHA₂DS₂-Vasc score of six (the 5^th and 95^th percentiles, respectively) when all other parameters are held at their medians. Results were comparatively insensitive to rebleeding risk. Varying Rockall score from two to seven (the 5^th and 95^th percentiles, respectively) added three days to optimal warfarin resumption. Varying other parameters from the 5^th to the 95^th percentile (including HAS-BLED score, sex, age, and discount rate) changed expected QALYs but did not change the optimal day of reinitiation of warfarin. Optimal day of reinitiation for warfarin stratified by CHA₂DS₂-Vasc score is shown in Table 2.

Anticoagulation is frequently prescribed for patients with NVAF, and hemorrhagic complications are common. Although anticoagulants are withheld following hemorrhages, scant evidence to inform the optimal timing of reinitiation is available. In this microsimulation analysis, we found that the optimal time to reinitiate anticoagulation following UGIB is around 41 days for warfarin and around 32 days for apixaban. We have further demonstrated that the optimal timing of reinitiation can vary by nearly two weeks, depending on a patient’s underlying risk of stroke, and that early reinitiation is more sensitive to rebleeding risk than late reinitiation.

Prior work has shown that early reinitiation of anticoagulation leads to higher rates of recurrent hemorrhage while failure to reinitiate anticoagulation is associated with higher rates of stroke and mortality.^1-4,36 Our results add to the literature in a number of important ways. First, our model not only confirms that anticoagulation should be restarted but also suggests when this action should be taken. The competing risks of bleeding and stroke have left clinicians with little guidance; we have quantified the clinical reasoning required for the decision to resume anticoagulation. Second, by including the disutility of hospitalization and long-term disability, our model more accurately represents the complex tradeoffs between recurrent hemorrhage and (potentially disabling) stroke than would a comparison of event rates. Third, our model is conditional upon patient risk factors, allowing clinicians to personalize the timing of anticoagulation resumption. Theory would suggest that patients at higher risk of ischemic stroke benefit from earlier resumption of anticoagulation, while patients at higher risk of hemorrhage benefit from delayed reinitiation. We have quantified the extent to which patient-specific risks should change timing. Fourth, we offer a means of improving expected health outcomes that requires little more than appropriate scheduling. Current practice regarding resuming anticoagulation is widely variable. Many patients never resume warfarin, and those that do resume do so after highly varied periods of time.^1-5,36 We offer a means of standardizing clinical practice and improving expected patient outcomes.

Disclosures

The authors have nothing to disclose.

Funding

The authors received no specific funding for this work.

Anticoagulation is commonly used in the management of atrial fibrillation to reduce the risk of ischemic stroke. Warfarin and other anticoagulants increase the risk of hemorrhagic complications, including upper gastrointestinal bleeding (UGIB). Following UGIB, management of anticoagulation is highly variable. Many patients permanently discontinue anticoagulation, while others continue without interruption.^1-4 Among patients who resume warfarin, different cohorts have measured median times to resumption ranging from four days to 50 days.^1-3 Outcomes data are sparse, and clinical guidelines offer little direction.⁵

Following UGIB, the balance between the risks and benefits of anticoagulation changes over time. Rebleeding risk is highest immediately after the event and declines quickly; therefore, rapid resumption of anticoagulation causes patient harm.³ Meanwhile, the risk of stroke remains constant, and delay in resumption of anticoagulation is associated with increased risk of stroke and death.¹ At some point in time following the initial UGIB, the expected harm from bleeding would equal the expected harm from stroke. This time point would represent the optimal time to restart anticoagulation.

Trial data are unlikely to identify the optimal time for restarting anticoagulation. A randomized trial comparing discrete reinitiation times (eg, two weeks vs six weeks) may easily miss the optimal timing. Moreover, because the daily probability of thromboembolic events is low, large numbers of patients would be required to power such a study. In addition, a number of oral anticoagulants are now approved for prevention of thromboembolic stroke in atrial fibrillation, and each drug may have different optimal timing.

In contrast to randomized trials that would be impracticable for addressing this clinical issue, microsimulation modeling can provide granular information regarding the optimal time to restart anticoagulation. Herein, we set out to estimate the expected benefit of reinitiation of warfarin, the most commonly used oral anticoagulant,⁶ or apixaban, the direct oral anticoagulant with the most favorable risk profile,⁷ as a function of days after UGIB.

We previously described a microsimulation model of anticoagulation among patients with nonvalvular atrial fibrillation (NVAF; hereafter, we refer to this model as the Personalized Anticoagulation Decision-Making Assistance model, or PADMA).^8,9 For this study, we extended this model to incorporate the probability of rebleeding following UGIB and include apixaban as an alternative to warfarin. This model begins with a synthetic population following UGIB, the members of which are at varying risk for thromboembolism, recurrent UGIB, and other hemorrhages. For each patient, the model simulates a number of possible events (eg, thromboembolic stroke, intracranial hemorrhage, rebleeding, and other major extracranial hemorrhages) on each day of an acute period of 90 days after hemostasis. Patients who survive until the end of the acute period enter a simulation with annual, rather than daily, cycles. Our model then estimates total quality-adjusted life-years (QALYs) for each patient, discounted to the present. We report the average discounted QALYs produced by the model for the same population if all individuals in our input population were to resume either warfarin or apixaban on a specific day. Input parameters and ranges are summarized in Table 1, a simplified schematic of our model is shown in the Supplemental Appendix, and additional details regarding model structure and assumptions can be found in earlier work.^8,9 We simulated from a health system perspective over a lifelong time horizon. All analyses were performed in version 14 of Stata (StataCorp, LLC, College Station, Texas).

Synthetic Population

To generate a population reflective of the comorbidities and age distribution of the US population with NVAF, we merged relevant variables from the National Health and Nutrition Examination Survey (NHANES; 2011-2012), using multiple imputation to correct for missing variables.¹⁰ We then bootstrapped to national population estimates by age and sex to arrive at a hypothetical population of the United States.¹¹ Because NHANES does not include atrial fibrillation, we applied sex- and age-specific prevalence rates from the AnTicoagulation and Risk Factors In Atrial Fibrillation study.¹² We then calculated commonly used risk scores (CHA₂DS₂-Vasc and HAS-BLED) for each patient and limited the population to patients with a CHA₂DS₂-Vasc score of one or greater.^13,14 The population resuming apixaban was further limited to patients whose creatinine clearance was 25 mL/min or greater in keeping with the entry criteria in the phase 3 clinical trial on which the medication’s approval was based.¹⁵

To estimate patient-specific probability of rebleeding, we generated a Rockall score for each patient.¹⁶ Although the discrimination of the Rockall score is limited for individual patients, as with all other tools used to predict rebleeding following UGIB, the Rockall score has demonstrated reasonable calibration across a threefold risk gradient.^17-19 International consensus guidelines recommend the Rockall score as one of two risk prediction tools for clinical use in the management of patients with UGIB.²⁰ In addition, because the Rockall score includes some demographic components (five of a possible 11 points), our estimates of rebleeding risk are covariant with other patient-specific risks. We assumed that the endoscopic components of the Rockall score were present in our cohort at the same frequency as in the original derivation and are independent of known patient risk factors.¹⁶ For example, 441 out of 4,025 patients in the original Rockall derivation cohort presented with a systolic blood pressure less than 100 mm Hg. We assumed that an independent and random 10.96% of the cohort would present with shock, which confers two points in the Rockall score.

The population was replicated 60 times, with identical copies of the population resuming anticoagulation on each of days 1-60 (where day zero represents hemostasis). Intermediate data regarding our simulated population can be found in the Supplemental Appendix and in prior work.

Event Type, Severity, and Mortality

Each patient in our simulation could sustain several discrete and independent events: ischemic stroke, intracranial hemorrhage, recurrent UGIB, or extracranial major hemorrhage other than recurrent UGIB. As in prior analyses using the PADMA model, we did not consider minor hemorrhagic events.⁸

The probability of each event was conditional on the corresponding risk scoring system. Patient-specific probability of ischemic stroke was conditional on CHA₂DS₂-Vasc score.^21,22 Patient-specific probability of intracranial hemorrhage was conditional on HAS-BLED score, with the proportions of intracranial hemorrhage of each considered subtype (intracerebral, subarachnoid, or subdural) bootstrapped from previously-published data.^21-24 Patient-specific probability of rebleeding was conditional on Rockall score from the combined Rockall and Vreeburg validation cohorts.¹⁷ Patient-specific probability of extracranial major hemorrhage was conditional on HAS-BLED score.²¹ To avoid double-counting of UGIB, we subtracted the baseline risk of UGIB from the overall rate of extracranial major hemorrhages using previously-published data regarding relative frequency and a bootstrapping approach.²⁵

Probability of Rebleeding Over Time

To estimate the decrease in rebleeding risk over time, we searched the Medline database for systematic reviews of recurrent bleeding following UGIB using the strategy detailed in the Supplemental Appendix. Using the interval rates of rebleeding we identified, we calculated implied daily rates of rebleeding at the midpoint of each interval. For example, 39.5% of rebleeding events occurred within three days of hemostasis, implying a daily rate of approximately 13.2% on day two (32 of 81 events over a three-day period). We repeated this process to estimate daily rates at the midpoint of each reported time interval and fitted an exponential decay function.²⁶ Our exponential fitted these datapoints quite well, but we lacked sufficient data to test other survival functions (eg, Gompertz, lognormal, etc.). Our fitted exponential can be expressed as:

P _rebleeding = b ₀ *exp(b ₁ *day)

where b₀ = 0.1843 (SE: 0.0136) and b₁ = –0.1563 (SE: 0.0188). For example, a mean of 3.9% of rebleeding episodes will occur on day 10 (0.1843 *exp(–0.1563 *10)).

Relative Risks of Events with Anticoagulation

For patients resuming warfarin, the probabilities of each event were adjusted based on patient-specific daily INR. All INRs were assumed to be 1.0 until the day of warfarin reinitiation, after which interpolated trajectories of postinitiation INR measurements were sampled for each patient from an earlier study of clinical warfarin initiation.²⁷ Relative risks of ischemic stroke and hemorrhagic events were calculated based on each day’s INR.

For patients taking apixaban, we assumed that the medication would reach full therapeutic effect one day after reinitiation. Based on available evidence, we applied the relative risks of each event with apixaban compared with warfarin.²⁵

Future Disability and Mortality

Each event in our simulation resulted in hospitalization. Length of stay was sampled for each diagnosis.²⁸ The disutility of hospitalization was estimated based on length of stay.⁸ Inpatient mortality and future disability were predicted for each event as previously described.⁸ We assumed that recurrent episodes of UGIB conferred morbidity and mortality identical to extracranial major hemorrhages more broadly.^29,30

Disutilities

We used a multiplicative model for disutility with baseline utilities conditional on age and sex.³¹ Each day after resumption of anticoagulation carried a disutility of 0.012 for warfarin or 0.002 for apixaban, which we assumed to be equivalent to aspirin in disutility.³² Long-term disutility and life expectancy were conditional on modified Rankin Score (mRS).^33,34 We discounted all QALYs to day zero using standard exponential discounting and a discount rate centered at 3%. We then computed the average discounted QALYs among the cohort of patients that resumed anticoagulation on each day following the index UGIB.

Sensitivity Analyses and Metamodel

To assess sensitivity to continuously varying input parameters, such as discount rate, the proportion of extracranial major hemorrhages that are upper GI bleeds, and inpatient mortality from extracranial major hemorrhage, we constructed a metamodel (a regression model of our microsimulation results).³⁵ We tested for interactions among input parameters and dropped parameters that were not statistically significant predictors of discounted QALYs from our metamodel. We then tested for interactions between each parameter and day resuming anticoagulation to determine which factors may impact the optimal day of reinitiation. Finally, we used predicted marginal effects from our metamodel to assess the change in optimal day across the ranges of each input parameter when other parameters were held at their medians.

Resuming warfarin on day zero produced the fewest QALYs. With delay in reinitiation of anticoagulation, expected QALYs increased, peaked, and then declined for all scenarios. In our base-case simulation of warfarin, peak utility was achieved by resumption 41 days after the index UGIB. Resumption between days 32 and 51 produced greater than 99.9% of peak utility. In our base-case simulation of apixaban, peak utility was achieved by resumption 32 days after the index UGIB. Resumption between days 21 and 47 produced greater than 99.9% of peak utility. Results for warfarin and apixaban are shown in Figures 1 and 2, respectively.

The optimal day of warfarin reinitiation was most sensitive to CHA₂DS₂-Vasc scores and varied by around 11 days between a CHA₂DS₂-Vasc score of one and a CHA₂DS₂-Vasc score of six (the 5^th and 95^th percentiles, respectively) when all other parameters are held at their medians. Results were comparatively insensitive to rebleeding risk. Varying Rockall score from two to seven (the 5^th and 95^th percentiles, respectively) added three days to optimal warfarin resumption. Varying other parameters from the 5^th to the 95^th percentile (including HAS-BLED score, sex, age, and discount rate) changed expected QALYs but did not change the optimal day of reinitiation of warfarin. Optimal day of reinitiation for warfarin stratified by CHA₂DS₂-Vasc score is shown in Table 2.

Anticoagulation is frequently prescribed for patients with NVAF, and hemorrhagic complications are common. Although anticoagulants are withheld following hemorrhages, scant evidence to inform the optimal timing of reinitiation is available. In this microsimulation analysis, we found that the optimal time to reinitiate anticoagulation following UGIB is around 41 days for warfarin and around 32 days for apixaban. We have further demonstrated that the optimal timing of reinitiation can vary by nearly two weeks, depending on a patient’s underlying risk of stroke, and that early reinitiation is more sensitive to rebleeding risk than late reinitiation.

Prior work has shown that early reinitiation of anticoagulation leads to higher rates of recurrent hemorrhage while failure to reinitiate anticoagulation is associated with higher rates of stroke and mortality.^1-4,36 Our results add to the literature in a number of important ways. First, our model not only confirms that anticoagulation should be restarted but also suggests when this action should be taken. The competing risks of bleeding and stroke have left clinicians with little guidance; we have quantified the clinical reasoning required for the decision to resume anticoagulation. Second, by including the disutility of hospitalization and long-term disability, our model more accurately represents the complex tradeoffs between recurrent hemorrhage and (potentially disabling) stroke than would a comparison of event rates. Third, our model is conditional upon patient risk factors, allowing clinicians to personalize the timing of anticoagulation resumption. Theory would suggest that patients at higher risk of ischemic stroke benefit from earlier resumption of anticoagulation, while patients at higher risk of hemorrhage benefit from delayed reinitiation. We have quantified the extent to which patient-specific risks should change timing. Fourth, we offer a means of improving expected health outcomes that requires little more than appropriate scheduling. Current practice regarding resuming anticoagulation is widely variable. Many patients never resume warfarin, and those that do resume do so after highly varied periods of time.^1-5,36 We offer a means of standardizing clinical practice and improving expected patient outcomes.

Disclosures

The authors have nothing to disclose.

Funding

The authors received no specific funding for this work.

Historically, academic medicine faculty were predominantly physician-scientists.¹ During the past decade, the number of clinician-educators and nontenured clinicians has grown.² Many academically oriented clinical faculty at our institution would like to participate in and learn how to conduct quality scholarship. While institutional requirements vary, scholarly work is often required for promotion,³ and faculty may also desire to support the scholarly work of residents. Moreover, a core program component of the Accreditation Council of Graduate Medical Education standards requires faculty to “maintain an environment of inquiry and scholarship with an active research component.”⁴ Yet clinical faculty often find academic projects to be challenging. Similar to residents, clinical academic faculty frequently lack formal training in health services research or quality improvement science, have insufficient mentorship, and typically have limited uncommitted time and resources.⁵

One approach to this problem has been to pair junior clinicians with traditional physician scientists as mentors.^6,7 This type of mentorship for clinical faculty is increasingly difficult to access because of growing pressure on physician-scientist faculty to conduct their own research, seek extramural funding, meet clinical expectations, and mentor fellows and faculty in their own disciplines.⁸ Moreover, senior research faculty may not be prepared or have the time to teach junior faculty how to deal with common stumbling blocks (eg, institutional review board [IRB] applications, statistically testable hypothesis development, and statistical analysis).^8,9 Seminars or works-in-progress sessions are another strategy to bolster scholarly work, but the experience at our institution is that such sessions are often not relevant at the time of delivery and can be intimidating to clinical faculty who lack extensive knowledge about research methods and prior research experience.

Another approach to supporting the research efforts of academic clinicians is to fund a consulting statistician. However, without sufficient content expertise, statisticians may be frustrated in their efforts to assist clinicians who struggle to formulate a testable question or to work directly with data collected. Statisticians may be inexperienced in writing IRB applications or implementing protocols in a clinical or educational setting. Furthermore, statistical consultations are often limited in scope¹⁰ and, in our setting, rarely produce a durable improvement in the research skills of the faculty member or the enduring partnership required to complete a longer-term project. Because of these shortcomings, we have found that purely statistical support resources are often underutilized and ineffective.

Other models to facilitate scholarship have been employed, but few focus on facilitating scholarship of clinical faculty. One strategy involved supporting hospitalist’s academic productivity by reducing hospitalists’ full-time equivalent (FTE) and providing mentorship.¹¹ For many, this approach is likely cost-prohibitive. Others have focused primarily on resident and fellow scholarships.^5,6

In this report, we describe an educational innovation to educate and support the scholarly work of academic hospitalists and internists by using an academic research coach. We recruited a health researcher with extensive experience in research methods and strong interpersonal skills with the ability to explain and teach research concepts in an accessible manner. We sought an individual who would provide high-yield single consultations, join project teams to provide ongoing mentorship from conception to completion, and consequently, bolster scholarly productivity and learning among nonresearch clinicians in our Division. We anticipated that providing support for multiple aspects of a project would be more likely to help faculty overcome barriers to research and disseminate their project results as scholarly output.

The coach initiative was implemented in the Division of General Internal Medicine at the University of Washington. The Division has over 200 members (60 hospitalists), including clinical instructors and acting instructors, who have not yet been appointed to the regular faculty (clinician-educators and physician scientists), and full-time clinical faculty. Division members staff clinical services at four area hospitals and 10 affiliated internal medicine and specialty clinics. Eligible clients were all Division members, although the focus of the initial program targeted hospitalists at our three primary teaching hospitals. Fellows, residents, students, and faculty from within and outside the Division were welcome to participate in a project involving coaching as long as a Division faculty member was engaged in the project.

Program Description

The overall goal of the coach initiative was to support the scholarly work of primarily clinical Division members. Given our focus was on clinical faculty with little training on research methodology, we did not expect the coach to secure grant funding for the position. Instead, we aimed to increase the quality and quantity of scholarship through publications, abstracts, and small grants. We defined scholarly work broadly: clinical research, quality improvement, medical education research, and other forms of scientific inquiry or synthesis. The coach was established as a 0.50 FTE position with a 12-month annually renewable appointment. The role was deemed that of a coach instead of a mentor because the coach was available to all Division members and involved task-oriented consultations with check-ins to facilitate projects, rather than a deeper more developmental relationship that typically exists with mentoring. The Division leadership identified support for scholarly activity as a high priority and mentorship as an unmet need based on faculty feedback. Clinical revenue supported the position.

Necessary qualifications, determined prior to hiring, included a PhD in health services or related field (eg, epidemiology) or a master’s degree with five years of experience in project management, clinical research, and study design. The position also called for expertise in articulating research questions, selecting study designs, navigating the IRB approval process, collecting/managing data, analyzing statistics, and mentoring and teaching clinical faculty in their scholarly endeavors. A track record in generating academic output (manuscripts and abstracts at regional/national meetings) was required. We circulated a description of the position to Division faculty and to leadership in our School of Public Health.

Based on these criteria, an inaugural coach was hired (author C.M.M.). The coach had a PhD in epidemiology, 10 years of research experience, 16 publications, and had recently finished a National Institutes of Health (NIH) career development award. At the time of hiring, she was a Clinical Assistant Professor in the School of Dentistry, which provided additional FTE. She had no extramural funding but was applying for NIH-level grants and had received several small grants.

To ensure uptake of the coach’s services, we realized that it was necessary to delineate the scope of services available, clarify availability of the coach, and define expectations regarding authorship. We used an iterative process that took into consideration the coach’s expertise, services most needed by the Division’s clinicians, and discussions with Division leadership and faculty at faculty meetings across hospitals and clinics. A range of services and authorship expectations were defined. Consensus was reached that the coach should be invited to coauthor projects where design, analysis, and/or substantial intellectual content was provided and for which authorship criteria were met.¹² Collegial reviews by the coach of already developed manuscripts and time-limited, low-intensity consultations that did not involve substantial intellectual contributions did not warrant authorship.¹² On this basis, we created and distributed a flyer to publicize these guidelines and invite Division members to contact the coach (Figure 1).

The coach attended Division, section, and clinical group meetings to publicize the initiative. The coach also individually met with faculty throughout the Division, explained her role, described services available, and answered questions. The marketing effort was continuous and calibrated with more or less exposure depending on existing projects and the coach’s availability. In addition, the coach coordinated with the director of the Division’s faculty development program to cohost works-in-progress seminars, identify coach clients to present at these meetings, and provide brief presentations on a basic research skill at meetings. Faculty built rapport with the coach through these activities and became more comfortable reaching out for assistance. Because of the large size of the Division, it was decided to roll out the initiative in a stepwise fashion, starting with hospitalists before expanding to the rest of the Division.

Most faculty contacted the coach by e-mail to request a consultation, at which time the coach requested that they complete a preconsultation handout (Figure 2). Initial coaching appointments lasted one hour and were in-person. Coaching entailed an in-depth analysis of the project plan and advice on how to move the project forward. The coach provided tailored scholarly project advice and expertise in research methods. After initial consultations, she would review grant proposals, IRB applications, manuscripts, case report forms, abstracts, and other products. Her efforts typically focused on improving the methods and scientific and technical writing. Assistance with statistical analysis was provided on a case-by-case basis to maintain broad availability. To address statistically complex questions, the coach had five hours of monthly access to a PhD biostatistician via an on-campus consulting service. Follow-up appointments were encouraged and provided as needed by e-mail, phone, or in-person. The coach conducted regular reach outs to facilitate projects. However, execution of the research was generally the responsibility of the faculty member.

Program Evaluation

To characterize the reach and scope of the program, the coach tracked the number of faculty supported, types of services provided, status of initiated projects, numbers of grants generated, and the dissemination of scholarly products including papers and abstracts. We used these metrics to create summary reports to identify successes and areas for improvement. Monthly meetings between the coach and Division leadership were used to fine-tune the approach.

We surveyed coach clients anonymously to assess their satisfaction with the coach initiative. Using Likert scale questions where 1 = completely disagree and 5 = completely agree, we asked (1) if they would recommend the coach to colleagues, (2) if their work was higher quality because of the coach, (3) if they were overall satisfied with the coach, (4) whether the Division should continue to support the coach, and (5) if the coach’s lack of clinical training negatively affected their experience. This work was considered a quality improvement initiative for which IRB approval was not required.

Over 18 months, the coach supported a 49 Division members including 30 hospitalists and 63 projects. Projects included a wide range of scholarship: medical education research, qualitative research, clinical quality improvement projects, observational studies, and a randomized clinical trial. Many clients (n = 16) used the coach for more than one project. The scope of work included limited support projects (identifying research resource and brainstorming project feasibility) lasting one to two sessions (n = 25), projects with a limited scope (collegial reviews of manuscripts and assistance with IRB submissions) but requiring more than two consultations (n = 24), and ongoing in-depth support projects (contributions on design, data collection, analysis, and manuscript writing) that required three consultations or more (n = 14). The majority of Division members (75%) supported did not have master’s level training in a health services-related area, six had NIH or other national-level funding, and two had small grants funded by local sources prior to providing support. The number of Division faculty on a given project ranged from one to four.

The coach directly supported 13 manuscripts with coach authorship, seven manuscripts without authorship, 11 abstracts, and four grant submissions (Appendix). The coach was a coauthor on all the abstracts and a coinvestigator on the grant applications. Of the 13 publications the coach coauthored, 11 publications have been accepted to peer-reviewed journals and two are currently in the submission process. The types of articles published included one medical evaluation report, one qualitative study, one randomized clinical trial, three quality assessment/improvement reports, and five epidemiologic studies. The types of abstracts included one qualitative report, one systematic review, one randomized clinical trial, two quality improvement projects, two epidemiologic studies, and four medical education projects. Three of four small grants submitted to local and national funders were funded.

The coach’s influence extended beyond the Division. Forty-eight university faculty, fellows, or students not affiliated with general internal medicine benefited from coach coaching: 26 were authors on papers and/or abstracts coauthored by the coach, 17 on manuscripts the coach reviewed without authorship, and five participated in consultations.

The coach found the experience rewarding. She enjoyed working on the methodologic aspects of projects and benefited from being included as coauthor on papers.

Twenty-nine of the 43 faculty (67%) still at the institution responded to the program assessment survey. Faculty strongly agreed that they would recommend the coach to colleagues (average ± standard deviation [SD]: 4.7 ± 0.5), that it improved the quality of their work (4.5 ± 0.9), that they were overall satisfied with the coaching (4.6 ± 0.7), and that the Division should continue to support the coach (4.9 ± 0.4). Faculty did not agree that the lack of clinical training of the coach was a barrier (2.0 ± 1.3).

The coach program was highly utilized, well regarded, and delivered substantial, tangible, and academic output. We anticipate the coach initiative will continue to be a valuable resource for our Division and could prove to be a valuable model for other institutions seeking to bolster the scholarly work of clinical academicians.

Several lessons emerged through the course of this project. First, we realized it is essential to select a coach who is both knowledgeable and approachable. We found that after meeting the coach, many faculty sought her help who otherwise would not have. An explicit, ongoing marketing strategy with regular contact with faculty at meetings was a key to receiving consult requests.

Second, the lack of a clinical background did not seem to hinder the coach’s ability to coach clinicians. The coach acknowledged her lack of clinical experience and relied on clients to explain the clinical context of projects. We also learned that the coach’s substantial experience with the logistics of research was invaluable. For example, the coach had substantial experience with the IRB process and her pre-reviews of IRB applications made for a short and relatively seamless experience navigating the IRB process. The coach also facilitated collaborations and leveraged existing resources at our institution. For example, for a qualitative research project, the coach helped identify a health services faculty member with this specific expertise, which led to a successful collaboration and publication. Although a more junior coach with less established qualifications may be helpful with research methods and with the research process, our endeavor suggests that having a more highly trained and experienced researcher was extremely valuable. Finally, we learned that for a Division of our size, the 0.50 FTE allotted to the coach is a minimum requirement. The coach spent approximately four hours a week on marketing, attending faculty meetings and conducting brief didactics, two hours per week on administration, and 14 hours per week on consultations. Faculty generally received support soon after their requests, but there were occasional wait times, which may have delayed some projects.

Academic leaders at our institution have noted the success of our coach initiative and have created a demand for coach services. We are exploring funding models that would allow for the expansion of coach services to other departments and divisions. We are in the initial stages of creating an Academic Scholarship Support Core under the supervision of the coach. Within this Core, we envision that various research support services will be triaged to staff with appropriate expertise; for example, a regulatory coordinator would review IRB applications while a master’s level statistician would conduct statistical analyses.

We have also transitioned to a new coach and have continued to experience success with the program. Our initial coach (author C.M.M.) obtained an NIH R01, a foundation grant, and took over a summer program that trains dental faculty in clinical research methods leaving insufficient time for coaching. Our new coach also has a PhD in epidemiology with NIH R01 funding but has more available FTE. Both of our coaches are graduates of our School of Public Health and institutions with such schools may have good access to the expertise needed. Nonclinical PhDs are often almost entirely reliant on grants, and some nongrant support is often attractive to these researchers. Additionally, PhDs who are junior or mid-career faculty that have the needed training are relatively affordable, particularly when the resource is made available to large number of faculty. For example, our first coach cost $48,000 a year for 50% FTE.

A limitation to our assessment of the coach initiative was the lack of pre- and postintervention metrics of scholarly productivity. We cannot definitively say that the Division’s scholarly output has increased because of the coach. Nevertheless, we are confident that the coach’s coaching has enhanced the scholarly work of individual clinicians and provided value to the Division as a whole. The coach program has been a success in our Division. Other institutions facing the challenge of supporting the research efforts of academic clinicians may consider this model as a worthy investment.

Disclosures

The authors have nothing to disclose.

Historically, academic medicine faculty were predominantly physician-scientists.¹ During the past decade, the number of clinician-educators and nontenured clinicians has grown.² Many academically oriented clinical faculty at our institution would like to participate in and learn how to conduct quality scholarship. While institutional requirements vary, scholarly work is often required for promotion,³ and faculty may also desire to support the scholarly work of residents. Moreover, a core program component of the Accreditation Council of Graduate Medical Education standards requires faculty to “maintain an environment of inquiry and scholarship with an active research component.”⁴ Yet clinical faculty often find academic projects to be challenging. Similar to residents, clinical academic faculty frequently lack formal training in health services research or quality improvement science, have insufficient mentorship, and typically have limited uncommitted time and resources.⁵

One approach to this problem has been to pair junior clinicians with traditional physician scientists as mentors.^6,7 This type of mentorship for clinical faculty is increasingly difficult to access because of growing pressure on physician-scientist faculty to conduct their own research, seek extramural funding, meet clinical expectations, and mentor fellows and faculty in their own disciplines.⁸ Moreover, senior research faculty may not be prepared or have the time to teach junior faculty how to deal with common stumbling blocks (eg, institutional review board [IRB] applications, statistically testable hypothesis development, and statistical analysis).^8,9 Seminars or works-in-progress sessions are another strategy to bolster scholarly work, but the experience at our institution is that such sessions are often not relevant at the time of delivery and can be intimidating to clinical faculty who lack extensive knowledge about research methods and prior research experience.

Another approach to supporting the research efforts of academic clinicians is to fund a consulting statistician. However, without sufficient content expertise, statisticians may be frustrated in their efforts to assist clinicians who struggle to formulate a testable question or to work directly with data collected. Statisticians may be inexperienced in writing IRB applications or implementing protocols in a clinical or educational setting. Furthermore, statistical consultations are often limited in scope¹⁰ and, in our setting, rarely produce a durable improvement in the research skills of the faculty member or the enduring partnership required to complete a longer-term project. Because of these shortcomings, we have found that purely statistical support resources are often underutilized and ineffective.

Other models to facilitate scholarship have been employed, but few focus on facilitating scholarship of clinical faculty. One strategy involved supporting hospitalist’s academic productivity by reducing hospitalists’ full-time equivalent (FTE) and providing mentorship.¹¹ For many, this approach is likely cost-prohibitive. Others have focused primarily on resident and fellow scholarships.^5,6

In this report, we describe an educational innovation to educate and support the scholarly work of academic hospitalists and internists by using an academic research coach. We recruited a health researcher with extensive experience in research methods and strong interpersonal skills with the ability to explain and teach research concepts in an accessible manner. We sought an individual who would provide high-yield single consultations, join project teams to provide ongoing mentorship from conception to completion, and consequently, bolster scholarly productivity and learning among nonresearch clinicians in our Division. We anticipated that providing support for multiple aspects of a project would be more likely to help faculty overcome barriers to research and disseminate their project results as scholarly output.

The coach initiative was implemented in the Division of General Internal Medicine at the University of Washington. The Division has over 200 members (60 hospitalists), including clinical instructors and acting instructors, who have not yet been appointed to the regular faculty (clinician-educators and physician scientists), and full-time clinical faculty. Division members staff clinical services at four area hospitals and 10 affiliated internal medicine and specialty clinics. Eligible clients were all Division members, although the focus of the initial program targeted hospitalists at our three primary teaching hospitals. Fellows, residents, students, and faculty from within and outside the Division were welcome to participate in a project involving coaching as long as a Division faculty member was engaged in the project.

Program Description

The overall goal of the coach initiative was to support the scholarly work of primarily clinical Division members. Given our focus was on clinical faculty with little training on research methodology, we did not expect the coach to secure grant funding for the position. Instead, we aimed to increase the quality and quantity of scholarship through publications, abstracts, and small grants. We defined scholarly work broadly: clinical research, quality improvement, medical education research, and other forms of scientific inquiry or synthesis. The coach was established as a 0.50 FTE position with a 12-month annually renewable appointment. The role was deemed that of a coach instead of a mentor because the coach was available to all Division members and involved task-oriented consultations with check-ins to facilitate projects, rather than a deeper more developmental relationship that typically exists with mentoring. The Division leadership identified support for scholarly activity as a high priority and mentorship as an unmet need based on faculty feedback. Clinical revenue supported the position.

Necessary qualifications, determined prior to hiring, included a PhD in health services or related field (eg, epidemiology) or a master’s degree with five years of experience in project management, clinical research, and study design. The position also called for expertise in articulating research questions, selecting study designs, navigating the IRB approval process, collecting/managing data, analyzing statistics, and mentoring and teaching clinical faculty in their scholarly endeavors. A track record in generating academic output (manuscripts and abstracts at regional/national meetings) was required. We circulated a description of the position to Division faculty and to leadership in our School of Public Health.

Based on these criteria, an inaugural coach was hired (author C.M.M.). The coach had a PhD in epidemiology, 10 years of research experience, 16 publications, and had recently finished a National Institutes of Health (NIH) career development award. At the time of hiring, she was a Clinical Assistant Professor in the School of Dentistry, which provided additional FTE. She had no extramural funding but was applying for NIH-level grants and had received several small grants.

To ensure uptake of the coach’s services, we realized that it was necessary to delineate the scope of services available, clarify availability of the coach, and define expectations regarding authorship. We used an iterative process that took into consideration the coach’s expertise, services most needed by the Division’s clinicians, and discussions with Division leadership and faculty at faculty meetings across hospitals and clinics. A range of services and authorship expectations were defined. Consensus was reached that the coach should be invited to coauthor projects where design, analysis, and/or substantial intellectual content was provided and for which authorship criteria were met.¹² Collegial reviews by the coach of already developed manuscripts and time-limited, low-intensity consultations that did not involve substantial intellectual contributions did not warrant authorship.¹² On this basis, we created and distributed a flyer to publicize these guidelines and invite Division members to contact the coach (Figure 1).

The coach attended Division, section, and clinical group meetings to publicize the initiative. The coach also individually met with faculty throughout the Division, explained her role, described services available, and answered questions. The marketing effort was continuous and calibrated with more or less exposure depending on existing projects and the coach’s availability. In addition, the coach coordinated with the director of the Division’s faculty development program to cohost works-in-progress seminars, identify coach clients to present at these meetings, and provide brief presentations on a basic research skill at meetings. Faculty built rapport with the coach through these activities and became more comfortable reaching out for assistance. Because of the large size of the Division, it was decided to roll out the initiative in a stepwise fashion, starting with hospitalists before expanding to the rest of the Division.

Most faculty contacted the coach by e-mail to request a consultation, at which time the coach requested that they complete a preconsultation handout (Figure 2). Initial coaching appointments lasted one hour and were in-person. Coaching entailed an in-depth analysis of the project plan and advice on how to move the project forward. The coach provided tailored scholarly project advice and expertise in research methods. After initial consultations, she would review grant proposals, IRB applications, manuscripts, case report forms, abstracts, and other products. Her efforts typically focused on improving the methods and scientific and technical writing. Assistance with statistical analysis was provided on a case-by-case basis to maintain broad availability. To address statistically complex questions, the coach had five hours of monthly access to a PhD biostatistician via an on-campus consulting service. Follow-up appointments were encouraged and provided as needed by e-mail, phone, or in-person. The coach conducted regular reach outs to facilitate projects. However, execution of the research was generally the responsibility of the faculty member.

Program Evaluation

To characterize the reach and scope of the program, the coach tracked the number of faculty supported, types of services provided, status of initiated projects, numbers of grants generated, and the dissemination of scholarly products including papers and abstracts. We used these metrics to create summary reports to identify successes and areas for improvement. Monthly meetings between the coach and Division leadership were used to fine-tune the approach.

We surveyed coach clients anonymously to assess their satisfaction with the coach initiative. Using Likert scale questions where 1 = completely disagree and 5 = completely agree, we asked (1) if they would recommend the coach to colleagues, (2) if their work was higher quality because of the coach, (3) if they were overall satisfied with the coach, (4) whether the Division should continue to support the coach, and (5) if the coach’s lack of clinical training negatively affected their experience. This work was considered a quality improvement initiative for which IRB approval was not required.

Over 18 months, the coach supported a 49 Division members including 30 hospitalists and 63 projects. Projects included a wide range of scholarship: medical education research, qualitative research, clinical quality improvement projects, observational studies, and a randomized clinical trial. Many clients (n = 16) used the coach for more than one project. The scope of work included limited support projects (identifying research resource and brainstorming project feasibility) lasting one to two sessions (n = 25), projects with a limited scope (collegial reviews of manuscripts and assistance with IRB submissions) but requiring more than two consultations (n = 24), and ongoing in-depth support projects (contributions on design, data collection, analysis, and manuscript writing) that required three consultations or more (n = 14). The majority of Division members (75%) supported did not have master’s level training in a health services-related area, six had NIH or other national-level funding, and two had small grants funded by local sources prior to providing support. The number of Division faculty on a given project ranged from one to four.

The coach directly supported 13 manuscripts with coach authorship, seven manuscripts without authorship, 11 abstracts, and four grant submissions (Appendix). The coach was a coauthor on all the abstracts and a coinvestigator on the grant applications. Of the 13 publications the coach coauthored, 11 publications have been accepted to peer-reviewed journals and two are currently in the submission process. The types of articles published included one medical evaluation report, one qualitative study, one randomized clinical trial, three quality assessment/improvement reports, and five epidemiologic studies. The types of abstracts included one qualitative report, one systematic review, one randomized clinical trial, two quality improvement projects, two epidemiologic studies, and four medical education projects. Three of four small grants submitted to local and national funders were funded.

The coach’s influence extended beyond the Division. Forty-eight university faculty, fellows, or students not affiliated with general internal medicine benefited from coach coaching: 26 were authors on papers and/or abstracts coauthored by the coach, 17 on manuscripts the coach reviewed without authorship, and five participated in consultations.

The coach found the experience rewarding. She enjoyed working on the methodologic aspects of projects and benefited from being included as coauthor on papers.

Twenty-nine of the 43 faculty (67%) still at the institution responded to the program assessment survey. Faculty strongly agreed that they would recommend the coach to colleagues (average ± standard deviation [SD]: 4.7 ± 0.5), that it improved the quality of their work (4.5 ± 0.9), that they were overall satisfied with the coaching (4.6 ± 0.7), and that the Division should continue to support the coach (4.9 ± 0.4). Faculty did not agree that the lack of clinical training of the coach was a barrier (2.0 ± 1.3).

The coach program was highly utilized, well regarded, and delivered substantial, tangible, and academic output. We anticipate the coach initiative will continue to be a valuable resource for our Division and could prove to be a valuable model for other institutions seeking to bolster the scholarly work of clinical academicians.

Several lessons emerged through the course of this project. First, we realized it is essential to select a coach who is both knowledgeable and approachable. We found that after meeting the coach, many faculty sought her help who otherwise would not have. An explicit, ongoing marketing strategy with regular contact with faculty at meetings was a key to receiving consult requests.

Second, the lack of a clinical background did not seem to hinder the coach’s ability to coach clinicians. The coach acknowledged her lack of clinical experience and relied on clients to explain the clinical context of projects. We also learned that the coach’s substantial experience with the logistics of research was invaluable. For example, the coach had substantial experience with the IRB process and her pre-reviews of IRB applications made for a short and relatively seamless experience navigating the IRB process. The coach also facilitated collaborations and leveraged existing resources at our institution. For example, for a qualitative research project, the coach helped identify a health services faculty member with this specific expertise, which led to a successful collaboration and publication. Although a more junior coach with less established qualifications may be helpful with research methods and with the research process, our endeavor suggests that having a more highly trained and experienced researcher was extremely valuable. Finally, we learned that for a Division of our size, the 0.50 FTE allotted to the coach is a minimum requirement. The coach spent approximately four hours a week on marketing, attending faculty meetings and conducting brief didactics, two hours per week on administration, and 14 hours per week on consultations. Faculty generally received support soon after their requests, but there were occasional wait times, which may have delayed some projects.

Academic leaders at our institution have noted the success of our coach initiative and have created a demand for coach services. We are exploring funding models that would allow for the expansion of coach services to other departments and divisions. We are in the initial stages of creating an Academic Scholarship Support Core under the supervision of the coach. Within this Core, we envision that various research support services will be triaged to staff with appropriate expertise; for example, a regulatory coordinator would review IRB applications while a master’s level statistician would conduct statistical analyses.

We have also transitioned to a new coach and have continued to experience success with the program. Our initial coach (author C.M.M.) obtained an NIH R01, a foundation grant, and took over a summer program that trains dental faculty in clinical research methods leaving insufficient time for coaching. Our new coach also has a PhD in epidemiology with NIH R01 funding but has more available FTE. Both of our coaches are graduates of our School of Public Health and institutions with such schools may have good access to the expertise needed. Nonclinical PhDs are often almost entirely reliant on grants, and some nongrant support is often attractive to these researchers. Additionally, PhDs who are junior or mid-career faculty that have the needed training are relatively affordable, particularly when the resource is made available to large number of faculty. For example, our first coach cost $48,000 a year for 50% FTE.

A limitation to our assessment of the coach initiative was the lack of pre- and postintervention metrics of scholarly productivity. We cannot definitively say that the Division’s scholarly output has increased because of the coach. Nevertheless, we are confident that the coach’s coaching has enhanced the scholarly work of individual clinicians and provided value to the Division as a whole. The coach program has been a success in our Division. Other institutions facing the challenge of supporting the research efforts of academic clinicians may consider this model as a worthy investment.

Disclosures

The authors have nothing to disclose.

Historically, academic medicine faculty were predominantly physician-scientists.¹ During the past decade, the number of clinician-educators and nontenured clinicians has grown.² Many academically oriented clinical faculty at our institution would like to participate in and learn how to conduct quality scholarship. While institutional requirements vary, scholarly work is often required for promotion,³ and faculty may also desire to support the scholarly work of residents. Moreover, a core program component of the Accreditation Council of Graduate Medical Education standards requires faculty to “maintain an environment of inquiry and scholarship with an active research component.”⁴ Yet clinical faculty often find academic projects to be challenging. Similar to residents, clinical academic faculty frequently lack formal training in health services research or quality improvement science, have insufficient mentorship, and typically have limited uncommitted time and resources.⁵

One approach to this problem has been to pair junior clinicians with traditional physician scientists as mentors.^6,7 This type of mentorship for clinical faculty is increasingly difficult to access because of growing pressure on physician-scientist faculty to conduct their own research, seek extramural funding, meet clinical expectations, and mentor fellows and faculty in their own disciplines.⁸ Moreover, senior research faculty may not be prepared or have the time to teach junior faculty how to deal with common stumbling blocks (eg, institutional review board [IRB] applications, statistically testable hypothesis development, and statistical analysis).^8,9 Seminars or works-in-progress sessions are another strategy to bolster scholarly work, but the experience at our institution is that such sessions are often not relevant at the time of delivery and can be intimidating to clinical faculty who lack extensive knowledge about research methods and prior research experience.

Another approach to supporting the research efforts of academic clinicians is to fund a consulting statistician. However, without sufficient content expertise, statisticians may be frustrated in their efforts to assist clinicians who struggle to formulate a testable question or to work directly with data collected. Statisticians may be inexperienced in writing IRB applications or implementing protocols in a clinical or educational setting. Furthermore, statistical consultations are often limited in scope¹⁰ and, in our setting, rarely produce a durable improvement in the research skills of the faculty member or the enduring partnership required to complete a longer-term project. Because of these shortcomings, we have found that purely statistical support resources are often underutilized and ineffective.

Other models to facilitate scholarship have been employed, but few focus on facilitating scholarship of clinical faculty. One strategy involved supporting hospitalist’s academic productivity by reducing hospitalists’ full-time equivalent (FTE) and providing mentorship.¹¹ For many, this approach is likely cost-prohibitive. Others have focused primarily on resident and fellow scholarships.^5,6

In this report, we describe an educational innovation to educate and support the scholarly work of academic hospitalists and internists by using an academic research coach. We recruited a health researcher with extensive experience in research methods and strong interpersonal skills with the ability to explain and teach research concepts in an accessible manner. We sought an individual who would provide high-yield single consultations, join project teams to provide ongoing mentorship from conception to completion, and consequently, bolster scholarly productivity and learning among nonresearch clinicians in our Division. We anticipated that providing support for multiple aspects of a project would be more likely to help faculty overcome barriers to research and disseminate their project results as scholarly output.

The coach initiative was implemented in the Division of General Internal Medicine at the University of Washington. The Division has over 200 members (60 hospitalists), including clinical instructors and acting instructors, who have not yet been appointed to the regular faculty (clinician-educators and physician scientists), and full-time clinical faculty. Division members staff clinical services at four area hospitals and 10 affiliated internal medicine and specialty clinics. Eligible clients were all Division members, although the focus of the initial program targeted hospitalists at our three primary teaching hospitals. Fellows, residents, students, and faculty from within and outside the Division were welcome to participate in a project involving coaching as long as a Division faculty member was engaged in the project.

Program Description

The overall goal of the coach initiative was to support the scholarly work of primarily clinical Division members. Given our focus was on clinical faculty with little training on research methodology, we did not expect the coach to secure grant funding for the position. Instead, we aimed to increase the quality and quantity of scholarship through publications, abstracts, and small grants. We defined scholarly work broadly: clinical research, quality improvement, medical education research, and other forms of scientific inquiry or synthesis. The coach was established as a 0.50 FTE position with a 12-month annually renewable appointment. The role was deemed that of a coach instead of a mentor because the coach was available to all Division members and involved task-oriented consultations with check-ins to facilitate projects, rather than a deeper more developmental relationship that typically exists with mentoring. The Division leadership identified support for scholarly activity as a high priority and mentorship as an unmet need based on faculty feedback. Clinical revenue supported the position.

Necessary qualifications, determined prior to hiring, included a PhD in health services or related field (eg, epidemiology) or a master’s degree with five years of experience in project management, clinical research, and study design. The position also called for expertise in articulating research questions, selecting study designs, navigating the IRB approval process, collecting/managing data, analyzing statistics, and mentoring and teaching clinical faculty in their scholarly endeavors. A track record in generating academic output (manuscripts and abstracts at regional/national meetings) was required. We circulated a description of the position to Division faculty and to leadership in our School of Public Health.

Based on these criteria, an inaugural coach was hired (author C.M.M.). The coach had a PhD in epidemiology, 10 years of research experience, 16 publications, and had recently finished a National Institutes of Health (NIH) career development award. At the time of hiring, she was a Clinical Assistant Professor in the School of Dentistry, which provided additional FTE. She had no extramural funding but was applying for NIH-level grants and had received several small grants.

To ensure uptake of the coach’s services, we realized that it was necessary to delineate the scope of services available, clarify availability of the coach, and define expectations regarding authorship. We used an iterative process that took into consideration the coach’s expertise, services most needed by the Division’s clinicians, and discussions with Division leadership and faculty at faculty meetings across hospitals and clinics. A range of services and authorship expectations were defined. Consensus was reached that the coach should be invited to coauthor projects where design, analysis, and/or substantial intellectual content was provided and for which authorship criteria were met.¹² Collegial reviews by the coach of already developed manuscripts and time-limited, low-intensity consultations that did not involve substantial intellectual contributions did not warrant authorship.¹² On this basis, we created and distributed a flyer to publicize these guidelines and invite Division members to contact the coach (Figure 1).

The coach attended Division, section, and clinical group meetings to publicize the initiative. The coach also individually met with faculty throughout the Division, explained her role, described services available, and answered questions. The marketing effort was continuous and calibrated with more or less exposure depending on existing projects and the coach’s availability. In addition, the coach coordinated with the director of the Division’s faculty development program to cohost works-in-progress seminars, identify coach clients to present at these meetings, and provide brief presentations on a basic research skill at meetings. Faculty built rapport with the coach through these activities and became more comfortable reaching out for assistance. Because of the large size of the Division, it was decided to roll out the initiative in a stepwise fashion, starting with hospitalists before expanding to the rest of the Division.

Most faculty contacted the coach by e-mail to request a consultation, at which time the coach requested that they complete a preconsultation handout (Figure 2). Initial coaching appointments lasted one hour and were in-person. Coaching entailed an in-depth analysis of the project plan and advice on how to move the project forward. The coach provided tailored scholarly project advice and expertise in research methods. After initial consultations, she would review grant proposals, IRB applications, manuscripts, case report forms, abstracts, and other products. Her efforts typically focused on improving the methods and scientific and technical writing. Assistance with statistical analysis was provided on a case-by-case basis to maintain broad availability. To address statistically complex questions, the coach had five hours of monthly access to a PhD biostatistician via an on-campus consulting service. Follow-up appointments were encouraged and provided as needed by e-mail, phone, or in-person. The coach conducted regular reach outs to facilitate projects. However, execution of the research was generally the responsibility of the faculty member.

Program Evaluation

To characterize the reach and scope of the program, the coach tracked the number of faculty supported, types of services provided, status of initiated projects, numbers of grants generated, and the dissemination of scholarly products including papers and abstracts. We used these metrics to create summary reports to identify successes and areas for improvement. Monthly meetings between the coach and Division leadership were used to fine-tune the approach.

We surveyed coach clients anonymously to assess their satisfaction with the coach initiative. Using Likert scale questions where 1 = completely disagree and 5 = completely agree, we asked (1) if they would recommend the coach to colleagues, (2) if their work was higher quality because of the coach, (3) if they were overall satisfied with the coach, (4) whether the Division should continue to support the coach, and (5) if the coach’s lack of clinical training negatively affected their experience. This work was considered a quality improvement initiative for which IRB approval was not required.

Over 18 months, the coach supported a 49 Division members including 30 hospitalists and 63 projects. Projects included a wide range of scholarship: medical education research, qualitative research, clinical quality improvement projects, observational studies, and a randomized clinical trial. Many clients (n = 16) used the coach for more than one project. The scope of work included limited support projects (identifying research resource and brainstorming project feasibility) lasting one to two sessions (n = 25), projects with a limited scope (collegial reviews of manuscripts and assistance with IRB submissions) but requiring more than two consultations (n = 24), and ongoing in-depth support projects (contributions on design, data collection, analysis, and manuscript writing) that required three consultations or more (n = 14). The majority of Division members (75%) supported did not have master’s level training in a health services-related area, six had NIH or other national-level funding, and two had small grants funded by local sources prior to providing support. The number of Division faculty on a given project ranged from one to four.

The coach directly supported 13 manuscripts with coach authorship, seven manuscripts without authorship, 11 abstracts, and four grant submissions (Appendix). The coach was a coauthor on all the abstracts and a coinvestigator on the grant applications. Of the 13 publications the coach coauthored, 11 publications have been accepted to peer-reviewed journals and two are currently in the submission process. The types of articles published included one medical evaluation report, one qualitative study, one randomized clinical trial, three quality assessment/improvement reports, and five epidemiologic studies. The types of abstracts included one qualitative report, one systematic review, one randomized clinical trial, two quality improvement projects, two epidemiologic studies, and four medical education projects. Three of four small grants submitted to local and national funders were funded.

The coach’s influence extended beyond the Division. Forty-eight university faculty, fellows, or students not affiliated with general internal medicine benefited from coach coaching: 26 were authors on papers and/or abstracts coauthored by the coach, 17 on manuscripts the coach reviewed without authorship, and five participated in consultations.

The coach found the experience rewarding. She enjoyed working on the methodologic aspects of projects and benefited from being included as coauthor on papers.

Twenty-nine of the 43 faculty (67%) still at the institution responded to the program assessment survey. Faculty strongly agreed that they would recommend the coach to colleagues (average ± standard deviation [SD]: 4.7 ± 0.5), that it improved the quality of their work (4.5 ± 0.9), that they were overall satisfied with the coaching (4.6 ± 0.7), and that the Division should continue to support the coach (4.9 ± 0.4). Faculty did not agree that the lack of clinical training of the coach was a barrier (2.0 ± 1.3).

The coach program was highly utilized, well regarded, and delivered substantial, tangible, and academic output. We anticipate the coach initiative will continue to be a valuable resource for our Division and could prove to be a valuable model for other institutions seeking to bolster the scholarly work of clinical academicians.

Several lessons emerged through the course of this project. First, we realized it is essential to select a coach who is both knowledgeable and approachable. We found that after meeting the coach, many faculty sought her help who otherwise would not have. An explicit, ongoing marketing strategy with regular contact with faculty at meetings was a key to receiving consult requests.

Second, the lack of a clinical background did not seem to hinder the coach’s ability to coach clinicians. The coach acknowledged her lack of clinical experience and relied on clients to explain the clinical context of projects. We also learned that the coach’s substantial experience with the logistics of research was invaluable. For example, the coach had substantial experience with the IRB process and her pre-reviews of IRB applications made for a short and relatively seamless experience navigating the IRB process. The coach also facilitated collaborations and leveraged existing resources at our institution. For example, for a qualitative research project, the coach helped identify a health services faculty member with this specific expertise, which led to a successful collaboration and publication. Although a more junior coach with less established qualifications may be helpful with research methods and with the research process, our endeavor suggests that having a more highly trained and experienced researcher was extremely valuable. Finally, we learned that for a Division of our size, the 0.50 FTE allotted to the coach is a minimum requirement. The coach spent approximately four hours a week on marketing, attending faculty meetings and conducting brief didactics, two hours per week on administration, and 14 hours per week on consultations. Faculty generally received support soon after their requests, but there were occasional wait times, which may have delayed some projects.

Academic leaders at our institution have noted the success of our coach initiative and have created a demand for coach services. We are exploring funding models that would allow for the expansion of coach services to other departments and divisions. We are in the initial stages of creating an Academic Scholarship Support Core under the supervision of the coach. Within this Core, we envision that various research support services will be triaged to staff with appropriate expertise; for example, a regulatory coordinator would review IRB applications while a master’s level statistician would conduct statistical analyses.

We have also transitioned to a new coach and have continued to experience success with the program. Our initial coach (author C.M.M.) obtained an NIH R01, a foundation grant, and took over a summer program that trains dental faculty in clinical research methods leaving insufficient time for coaching. Our new coach also has a PhD in epidemiology with NIH R01 funding but has more available FTE. Both of our coaches are graduates of our School of Public Health and institutions with such schools may have good access to the expertise needed. Nonclinical PhDs are often almost entirely reliant on grants, and some nongrant support is often attractive to these researchers. Additionally, PhDs who are junior or mid-career faculty that have the needed training are relatively affordable, particularly when the resource is made available to large number of faculty. For example, our first coach cost $48,000 a year for 50% FTE.

A limitation to our assessment of the coach initiative was the lack of pre- and postintervention metrics of scholarly productivity. We cannot definitively say that the Division’s scholarly output has increased because of the coach. Nevertheless, we are confident that the coach’s coaching has enhanced the scholarly work of individual clinicians and provided value to the Division as a whole. The coach program has been a success in our Division. Other institutions facing the challenge of supporting the research efforts of academic clinicians may consider this model as a worthy investment.

Disclosures

The authors have nothing to disclose.

Acute kidney injury (AKI) is increasingly common in the hospitalized patient^1,2 with recent adult and pediatric multinational studies reporting AKI rates of 57% and 27%, respectively.^3,4 The development of AKI is associated with significant adverse outcomes including an increased risk of mortality.^5-7 For those that survive, the history of AKI may contribute to a lifetime of impaired health with chronic kidney disease.^8,9 This is particularly concerning for pediatric patients as AKI may impact morbidity for many decades, influence available therapies for these morbidities, and ultimately contribute to a shortened lifespan.¹⁰

AKI in the hospitalized patient is no longer accepted as an unfortunate and unavoidable consequence of illness or the indicated therapy. Currently, there is strong interest in this hospital-acquired condition with global initiatives aimed at increased prevention and early detection and treatment of AKI.^11,12 To this objective, risk stratification tools or prediction models could assist clinicians in decision making. Numerous studies have tested AKI prediction models either in particular high-risk populations or based on associated comorbidities, biomarkers, and critical illness scores. These studies are predominantly in adult populations, and few have been externally validated.¹³ While associations between certain medications and AKI are well known, an AKI prediction model that is applicable to pediatric or adult populations and is based on medication exposure is difficult. However, there is a growing recognition of the potential to develop such a model using the electronic health record (EHR).¹⁴

In 2013, Seattle Children’s Hospital (SCH) implemented a nephrotoxin and AKI detection system to assist in clinical decision making within the EHR. This system instituted the automatic ordering of serum creatinines to screen for AKI when the provider ordered three or more medications that were suspected to be nephrotoxic. Other clinical factors such as the diagnoses or preexisting conditions were not considered in the decision-tool algorithm. This original algorithm (Algorithm 1) was later modified and the list of suspected nephrotoxins was expanded (Table 1) in order to align with a national pediatric AKI collaborative (Algorithm 2). However, it was unclear whether the algorithm modification would improve AKI detection.

Context

SCH is a tertiary care academic teaching hospital affiliated with the University of Washington School of Medicine, Harborview Medical Center, and the Seattle Cancer Care Alliance. The hospital has 371 licensed beds and approximately 18 medical subspecialty services.

Study Population

This was a retrospective cohort study examining all patients ages 0-21 years admitted to SCH between December 1, 2013 and November 30, 2015. The detection system was modified to align with the national pediatric AKI collaborative, Nephrotoxic Injury Negated by Just-in-Time Action (NINJA) in November 2014. Both acute care and intensive care patients were included (data not separated by location). Patients who had end-stage kidney disease and were receiving dialysis and patients who were evaluated in the emergency department without being admitted or admitted as observation status were excluded from analysis. Patients were also excluded if they did not have a baseline serum creatinine as defined below.

Study Measures

AKI is defined at SCH using the Kidney Disease: Improving Global Outcomes Stage 1 criteria as a guideline. The diagnosis of AKI is based on an increase in the baseline serum creatinine by 0.3 mg/dL or an increase in the serum creatinine by >1.5 times the baseline assuming the incoming creatinine is 0.5 mg/dL or higher. For our definition, the increase in serum creatinine needs to have occurred within a one-week timeframe and urine output is not a diagnostic criterion.¹⁵ Baseline serum creatinine is defined as the lowest serum creatinine in the previous six months. Forty medications were classified as nephrotoxins based on previous analysis¹⁶ and adapted for our institutional formulary.

Statistical Analysis

To evaluate the efficacy of our systems in detecting nephrotoxin-related AKI, the sensitivity and the specificity using both our original algorithm (Algorithm 1) and the modified algorithm (Algorithm 2) were generated on our complete data set. To test sensitivity, the proportion of AKI patients who would trigger alert using Algorithm 1 and then with Algorithm 2 was identified. Similarly, to test specificity, the proportion of non-AKI patients who did not trigger an alert by the surveillance systems was identified. The differences in sensitivity and specificity between the two algorithms were evaluated using two-sample tests of proportion.

The statistical method of Combinatorial Inference has been utilized in studies of cancer biology¹⁷ and in genomics.¹⁸ A variation of this approach was used in this study to identify the specific medication combinations most associated with AKI. First, all of the nephrotoxic medications and medication combinations that were prescribed during our study period were identified from a data set (ie, a training set) containing 75% of all encounters selected at random without replacement. Using this training set, the prevalence of each medication combination and the rate of AKI associated with each combination were identified. The predicted overall AKI risk of an individual medication is the average of all the AKI rates associated with each combination containing that specific medication. Also incorporated into the determination of the predicted AKI risk was the prevalence of that medication combination.

To test our model’s predictive capability, the algorithm was applied to the remaining 25% of the total patient data (ie, the test set). The predicted AKI risk was compared with the actual AKI rate in the test data set. Our model’s predictive capability was represented in a receiver operator characteristic (ROC) analysis. The goal was to achieve an area under the ROC curve (AUC) approaching one as this would reflect 100% sensitivity and 100% specificity, whereas an AUC of 0.5 would represent a random guess (50% chance of being correct).

Lastly, our final step was to use our model’s ROC curve to determine an optimal threshold of AKI risk for which to trigger an alert. This predicted risk threshold was based on our goal to increase our surveillance system’s sensitivity balanced with maintaining an acceptable specificity.

An a priori threshold of P = .05 was used to determine statistical significance of all results. Analyses were conducted in Stata 12.1 (StataCorp LP, College Station, Texas) and R 3.3.2 (R Foundation for Statistical Computing, Vienna, Austria). A sample data set containing replication code for our model can be found in an online repository (https://dataverse.harvard.edu/dataverse/chuan). This study was approved by the Seattle Children’s Institutional Review Board.

Sensitivity and Specificity

Of the patient encounters, 14,779 were eligible during the study period. The sensitivity of the system’s ability to identify nephrotoxin-related AKI decreased from 46.9% using Algorithm 1 to 43.3% using Algorithm 2, a change of 3.6% (P = .22). The specificity increased from 73.6% to 89.3%, a change of 15.7% (P < .001; Table 2).

Improvement of Our Nephrotoxin-Related AKI Detection System Using a Novel AKI Prediction Strategy

A total of 838 medication combinations were identified in our training set and the predicted AKI risk for every medication combination was determined. By comparing the predicted risk of AKI to the actual AKI occurrence, an ROC curve with an AUC of 0.756 (Figure) was generated. An increase in system sensitivity was prioritized when determining the optimal AKI risk at which the model would trigger an alert. Setting an alert threshold at a predicted AKI risk of >8%, our model performed with a sensitivity of 74% while decreasing the specificity to 70%.

Identification of High-Risk Nephrotoxic Medications and Medication Combinations

Approximately 200 medication combinations were associated with >8% AKI risk, our new AKI prediction model’s alert threshold. Medication combinations consisting of up to 11 concomitantly prescribed medications were present in our data set. However, many of these combinations were infrequently prescribed. Further analysis, conducted in order to increase the clinical relevance of our findings, identified 10 medications or medication combinations that were both associated with a predicted AKI risk of >8% and that were prescribed on average greater than twice a month (Table 3).

The nephrotoxin-related AKI detection system at SCH automatically places orders for serum creatinines on patients who have met criteria for concomitant nephrotoxin exposure. This has given us a robust database from which to develop our clinical decision-making tool. Both our original and updated systems were based on the absolute number of concomitant nephrotoxic medications prescribed.¹⁶ This is a reasonable approach given the complexity of building a surveillance system¹⁹ and resource limitations. However, a system based on observed rather than theoretical or in vitro data, adaptable to the institution and designed for ongoing refinement, would be more valuable.

The interest in AKI prediction tools continues to be high. Bedford et al. employed numerous variables and diagnostic codes to predict the development of AKI in adults during hospitalization. They were able to produce a prediction model with a reasonable fit (AUC 0.72) to identify patients at higher risk for AKI but were less successful in their attempts to predict progression to severe AKI.²⁰ Hodgson et al. recently developed an adult AKI prediction score (AUC 0.65-0.72) also based on numerous clinical factors that was able to positively impact inpatient mortality.²¹ To our knowledge, our model is unique in that it focuses on nephrotoxins using a predicted AKI risk algorithm based on observed AKI rates of previously ordered medications/medication combinations (two to 11 medications). Having a decision tool targeting medications gives the clinician guidance that can be used to make a specific intervention rather than identifying a patient at risk due to a diagnosis code or other difficult to modify factors.

There are abundant case studies and reports using logistic regression models identifying specific medications associated with AKI. Our choice of methodology was based on our assessment that logistic regression models would be inadequate for the development of a real-time clinical decision-making tool for several reasons. Using logistic regression to explore every medication combination based on our medication list would be challenging as there are approximately 5.5 × 10¹⁰ potential medication combinations. Additionally, logistic regression ignores any potential interactions between the medications. This is an important point as medication interactions can be synergistic, neutral, or antagonist. Consequently, the outcome generated from a set of combined variables may be different from one generated from the sum of each variable taken independently. Logistic regression also does not account for the potential prescribing trends among providers as it assumes that all medications or medication combinations are equally available at the same time. However, in practice, depending on numerous factors, such as hospital culture (eg, the presence of clinical standard work pathways), local bacterial resistance patterns, or medication shortages; certain medication combinations may occur more frequently while others not at all. Finally, logistic regression cannot account for the possibility of a medication combination occurring; therefore, logistic regression may identify a combination strongly associated with AKI that is rarely prescribed.

We theorized that AKI detection would improve with the Algorithm 2 modifications, including the expanded nephrotoxin list, which accompanied alignment with the national pediatric AKI collaborative, NINJA. The finding that our surveillance sensitivity did not improve with this system update supported our subsequent objective to develop a novel nephrotoxin-related AKI decision tool or detection system using our EHR data to identify which specific medications and/or medication combinations were associated with a higher rate of AKI. However, it should be noted that two factors related to measurement bias introduce limitations to our sensitivity and specificity analyses. First, regarding the presence of the alert system, our system will order serum creatinines on patients when they have been exposed to nephrotoxins. Consequently, the proportion of patients with creatinines measured will increase in the nephrotoxin-exposed patients. Unexposed patients may have AKI that is not detected because creatinines may not be ordered. Therefore, there is the potential for a relative increase in AKI detection among nephrotoxin-exposed patients as compared with unexposed patients, which would then affect the measured sensitivity and specificity of the alert. Second, the automated alerts require a baseline creatinine in order to trigger therefore are unable to identify AKI among patients who do not have a baseline serum creatinine measurement.

Our new nephrotoxin-related AKI detection model performed best when an alert was triggered for those medications or medication combinations with a predicted AKI risk of >8%. Forty-six medication combinations consisting of exactly two medications were determined to have a predicted AKI risk of >8% therefore would trigger an alert in our new model system. These medication combinations would not have triggered an alert using either of the previous system algorithms as both algorithms are based on the presence of three or more concomitant nephrotoxic medications.

From the list of suspected nephrotoxins, we identified 11 unique medications in 10 different combinations with a predicted AKI risk of >8% that were prescribed frequently (at least twice a month on average; Table 3). Notably, six out of 10 medication combinations involved vancomycin. Piperacillin-tazobactam was also represented in several combinations. These findings support the concern that others have reported regarding these two medications particularly when prescribed together.^22,23

Acknowledgments

Dr. Karyn E. Yonekawa conceptualized and designed the study, directed the data analysis, interpreted the data, drafted, revised and gave final approval of the manuscript. Dr. Chuan Zhou contributed to the study design, acquired data, conducted the data analysis, critically reviewed, and gave final approval of the manuscript. Ms. Wren L. Haaland contributed to the study design, acquired data, conducted the data analysis, critically reviewed, and gave final approval of the manuscript. Dr. Davene R. Wright contributed to the study design, data analysis, critically reviewed, revised, and gave final approval of the manuscript.

The authors would like to thank Holly Clifton and Suzanne Spencer for their assistance with data acquisition and Drs. Derya Caglar, Corrie McDaniel, and Thida Ong for their writing support.

All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

Disclosures

The authors have no conflicts of interest to report.

Acute kidney injury (AKI) is increasingly common in the hospitalized patient^1,2 with recent adult and pediatric multinational studies reporting AKI rates of 57% and 27%, respectively.^3,4 The development of AKI is associated with significant adverse outcomes including an increased risk of mortality.^5-7 For those that survive, the history of AKI may contribute to a lifetime of impaired health with chronic kidney disease.^8,9 This is particularly concerning for pediatric patients as AKI may impact morbidity for many decades, influence available therapies for these morbidities, and ultimately contribute to a shortened lifespan.¹⁰

AKI in the hospitalized patient is no longer accepted as an unfortunate and unavoidable consequence of illness or the indicated therapy. Currently, there is strong interest in this hospital-acquired condition with global initiatives aimed at increased prevention and early detection and treatment of AKI.^11,12 To this objective, risk stratification tools or prediction models could assist clinicians in decision making. Numerous studies have tested AKI prediction models either in particular high-risk populations or based on associated comorbidities, biomarkers, and critical illness scores. These studies are predominantly in adult populations, and few have been externally validated.¹³ While associations between certain medications and AKI are well known, an AKI prediction model that is applicable to pediatric or adult populations and is based on medication exposure is difficult. However, there is a growing recognition of the potential to develop such a model using the electronic health record (EHR).¹⁴

In 2013, Seattle Children’s Hospital (SCH) implemented a nephrotoxin and AKI detection system to assist in clinical decision making within the EHR. This system instituted the automatic ordering of serum creatinines to screen for AKI when the provider ordered three or more medications that were suspected to be nephrotoxic. Other clinical factors such as the diagnoses or preexisting conditions were not considered in the decision-tool algorithm. This original algorithm (Algorithm 1) was later modified and the list of suspected nephrotoxins was expanded (Table 1) in order to align with a national pediatric AKI collaborative (Algorithm 2). However, it was unclear whether the algorithm modification would improve AKI detection.

Context

SCH is a tertiary care academic teaching hospital affiliated with the University of Washington School of Medicine, Harborview Medical Center, and the Seattle Cancer Care Alliance. The hospital has 371 licensed beds and approximately 18 medical subspecialty services.

Study Population

This was a retrospective cohort study examining all patients ages 0-21 years admitted to SCH between December 1, 2013 and November 30, 2015. The detection system was modified to align with the national pediatric AKI collaborative, Nephrotoxic Injury Negated by Just-in-Time Action (NINJA) in November 2014. Both acute care and intensive care patients were included (data not separated by location). Patients who had end-stage kidney disease and were receiving dialysis and patients who were evaluated in the emergency department without being admitted or admitted as observation status were excluded from analysis. Patients were also excluded if they did not have a baseline serum creatinine as defined below.

Study Measures

AKI is defined at SCH using the Kidney Disease: Improving Global Outcomes Stage 1 criteria as a guideline. The diagnosis of AKI is based on an increase in the baseline serum creatinine by 0.3 mg/dL or an increase in the serum creatinine by >1.5 times the baseline assuming the incoming creatinine is 0.5 mg/dL or higher. For our definition, the increase in serum creatinine needs to have occurred within a one-week timeframe and urine output is not a diagnostic criterion.¹⁵ Baseline serum creatinine is defined as the lowest serum creatinine in the previous six months. Forty medications were classified as nephrotoxins based on previous analysis¹⁶ and adapted for our institutional formulary.

Statistical Analysis

To evaluate the efficacy of our systems in detecting nephrotoxin-related AKI, the sensitivity and the specificity using both our original algorithm (Algorithm 1) and the modified algorithm (Algorithm 2) were generated on our complete data set. To test sensitivity, the proportion of AKI patients who would trigger alert using Algorithm 1 and then with Algorithm 2 was identified. Similarly, to test specificity, the proportion of non-AKI patients who did not trigger an alert by the surveillance systems was identified. The differences in sensitivity and specificity between the two algorithms were evaluated using two-sample tests of proportion.

The statistical method of Combinatorial Inference has been utilized in studies of cancer biology¹⁷ and in genomics.¹⁸ A variation of this approach was used in this study to identify the specific medication combinations most associated with AKI. First, all of the nephrotoxic medications and medication combinations that were prescribed during our study period were identified from a data set (ie, a training set) containing 75% of all encounters selected at random without replacement. Using this training set, the prevalence of each medication combination and the rate of AKI associated with each combination were identified. The predicted overall AKI risk of an individual medication is the average of all the AKI rates associated with each combination containing that specific medication. Also incorporated into the determination of the predicted AKI risk was the prevalence of that medication combination.

To test our model’s predictive capability, the algorithm was applied to the remaining 25% of the total patient data (ie, the test set). The predicted AKI risk was compared with the actual AKI rate in the test data set. Our model’s predictive capability was represented in a receiver operator characteristic (ROC) analysis. The goal was to achieve an area under the ROC curve (AUC) approaching one as this would reflect 100% sensitivity and 100% specificity, whereas an AUC of 0.5 would represent a random guess (50% chance of being correct).

Lastly, our final step was to use our model’s ROC curve to determine an optimal threshold of AKI risk for which to trigger an alert. This predicted risk threshold was based on our goal to increase our surveillance system’s sensitivity balanced with maintaining an acceptable specificity.

An a priori threshold of P = .05 was used to determine statistical significance of all results. Analyses were conducted in Stata 12.1 (StataCorp LP, College Station, Texas) and R 3.3.2 (R Foundation for Statistical Computing, Vienna, Austria). A sample data set containing replication code for our model can be found in an online repository (https://dataverse.harvard.edu/dataverse/chuan). This study was approved by the Seattle Children’s Institutional Review Board.

Sensitivity and Specificity

Of the patient encounters, 14,779 were eligible during the study period. The sensitivity of the system’s ability to identify nephrotoxin-related AKI decreased from 46.9% using Algorithm 1 to 43.3% using Algorithm 2, a change of 3.6% (P = .22). The specificity increased from 73.6% to 89.3%, a change of 15.7% (P < .001; Table 2).

Improvement of Our Nephrotoxin-Related AKI Detection System Using a Novel AKI Prediction Strategy

A total of 838 medication combinations were identified in our training set and the predicted AKI risk for every medication combination was determined. By comparing the predicted risk of AKI to the actual AKI occurrence, an ROC curve with an AUC of 0.756 (Figure) was generated. An increase in system sensitivity was prioritized when determining the optimal AKI risk at which the model would trigger an alert. Setting an alert threshold at a predicted AKI risk of >8%, our model performed with a sensitivity of 74% while decreasing the specificity to 70%.

Identification of High-Risk Nephrotoxic Medications and Medication Combinations

Approximately 200 medication combinations were associated with >8% AKI risk, our new AKI prediction model’s alert threshold. Medication combinations consisting of up to 11 concomitantly prescribed medications were present in our data set. However, many of these combinations were infrequently prescribed. Further analysis, conducted in order to increase the clinical relevance of our findings, identified 10 medications or medication combinations that were both associated with a predicted AKI risk of >8% and that were prescribed on average greater than twice a month (Table 3).

The nephrotoxin-related AKI detection system at SCH automatically places orders for serum creatinines on patients who have met criteria for concomitant nephrotoxin exposure. This has given us a robust database from which to develop our clinical decision-making tool. Both our original and updated systems were based on the absolute number of concomitant nephrotoxic medications prescribed.¹⁶ This is a reasonable approach given the complexity of building a surveillance system¹⁹ and resource limitations. However, a system based on observed rather than theoretical or in vitro data, adaptable to the institution and designed for ongoing refinement, would be more valuable.

The interest in AKI prediction tools continues to be high. Bedford et al. employed numerous variables and diagnostic codes to predict the development of AKI in adults during hospitalization. They were able to produce a prediction model with a reasonable fit (AUC 0.72) to identify patients at higher risk for AKI but were less successful in their attempts to predict progression to severe AKI.²⁰ Hodgson et al. recently developed an adult AKI prediction score (AUC 0.65-0.72) also based on numerous clinical factors that was able to positively impact inpatient mortality.²¹ To our knowledge, our model is unique in that it focuses on nephrotoxins using a predicted AKI risk algorithm based on observed AKI rates of previously ordered medications/medication combinations (two to 11 medications). Having a decision tool targeting medications gives the clinician guidance that can be used to make a specific intervention rather than identifying a patient at risk due to a diagnosis code or other difficult to modify factors.

There are abundant case studies and reports using logistic regression models identifying specific medications associated with AKI. Our choice of methodology was based on our assessment that logistic regression models would be inadequate for the development of a real-time clinical decision-making tool for several reasons. Using logistic regression to explore every medication combination based on our medication list would be challenging as there are approximately 5.5 × 10¹⁰ potential medication combinations. Additionally, logistic regression ignores any potential interactions between the medications. This is an important point as medication interactions can be synergistic, neutral, or antagonist. Consequently, the outcome generated from a set of combined variables may be different from one generated from the sum of each variable taken independently. Logistic regression also does not account for the potential prescribing trends among providers as it assumes that all medications or medication combinations are equally available at the same time. However, in practice, depending on numerous factors, such as hospital culture (eg, the presence of clinical standard work pathways), local bacterial resistance patterns, or medication shortages; certain medication combinations may occur more frequently while others not at all. Finally, logistic regression cannot account for the possibility of a medication combination occurring; therefore, logistic regression may identify a combination strongly associated with AKI that is rarely prescribed.

We theorized that AKI detection would improve with the Algorithm 2 modifications, including the expanded nephrotoxin list, which accompanied alignment with the national pediatric AKI collaborative, NINJA. The finding that our surveillance sensitivity did not improve with this system update supported our subsequent objective to develop a novel nephrotoxin-related AKI decision tool or detection system using our EHR data to identify which specific medications and/or medication combinations were associated with a higher rate of AKI. However, it should be noted that two factors related to measurement bias introduce limitations to our sensitivity and specificity analyses. First, regarding the presence of the alert system, our system will order serum creatinines on patients when they have been exposed to nephrotoxins. Consequently, the proportion of patients with creatinines measured will increase in the nephrotoxin-exposed patients. Unexposed patients may have AKI that is not detected because creatinines may not be ordered. Therefore, there is the potential for a relative increase in AKI detection among nephrotoxin-exposed patients as compared with unexposed patients, which would then affect the measured sensitivity and specificity of the alert. Second, the automated alerts require a baseline creatinine in order to trigger therefore are unable to identify AKI among patients who do not have a baseline serum creatinine measurement.

Our new nephrotoxin-related AKI detection model performed best when an alert was triggered for those medications or medication combinations with a predicted AKI risk of >8%. Forty-six medication combinations consisting of exactly two medications were determined to have a predicted AKI risk of >8% therefore would trigger an alert in our new model system. These medication combinations would not have triggered an alert using either of the previous system algorithms as both algorithms are based on the presence of three or more concomitant nephrotoxic medications.

From the list of suspected nephrotoxins, we identified 11 unique medications in 10 different combinations with a predicted AKI risk of >8% that were prescribed frequently (at least twice a month on average; Table 3). Notably, six out of 10 medication combinations involved vancomycin. Piperacillin-tazobactam was also represented in several combinations. These findings support the concern that others have reported regarding these two medications particularly when prescribed together.^22,23

Acknowledgments

Dr. Karyn E. Yonekawa conceptualized and designed the study, directed the data analysis, interpreted the data, drafted, revised and gave final approval of the manuscript. Dr. Chuan Zhou contributed to the study design, acquired data, conducted the data analysis, critically reviewed, and gave final approval of the manuscript. Ms. Wren L. Haaland contributed to the study design, acquired data, conducted the data analysis, critically reviewed, and gave final approval of the manuscript. Dr. Davene R. Wright contributed to the study design, data analysis, critically reviewed, revised, and gave final approval of the manuscript.

The authors would like to thank Holly Clifton and Suzanne Spencer for their assistance with data acquisition and Drs. Derya Caglar, Corrie McDaniel, and Thida Ong for their writing support.

All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

Disclosures

The authors have no conflicts of interest to report.

Acute kidney injury (AKI) is increasingly common in the hospitalized patient^1,2 with recent adult and pediatric multinational studies reporting AKI rates of 57% and 27%, respectively.^3,4 The development of AKI is associated with significant adverse outcomes including an increased risk of mortality.^5-7 For those that survive, the history of AKI may contribute to a lifetime of impaired health with chronic kidney disease.^8,9 This is particularly concerning for pediatric patients as AKI may impact morbidity for many decades, influence available therapies for these morbidities, and ultimately contribute to a shortened lifespan.¹⁰

AKI in the hospitalized patient is no longer accepted as an unfortunate and unavoidable consequence of illness or the indicated therapy. Currently, there is strong interest in this hospital-acquired condition with global initiatives aimed at increased prevention and early detection and treatment of AKI.^11,12 To this objective, risk stratification tools or prediction models could assist clinicians in decision making. Numerous studies have tested AKI prediction models either in particular high-risk populations or based on associated comorbidities, biomarkers, and critical illness scores. These studies are predominantly in adult populations, and few have been externally validated.¹³ While associations between certain medications and AKI are well known, an AKI prediction model that is applicable to pediatric or adult populations and is based on medication exposure is difficult. However, there is a growing recognition of the potential to develop such a model using the electronic health record (EHR).¹⁴

In 2013, Seattle Children’s Hospital (SCH) implemented a nephrotoxin and AKI detection system to assist in clinical decision making within the EHR. This system instituted the automatic ordering of serum creatinines to screen for AKI when the provider ordered three or more medications that were suspected to be nephrotoxic. Other clinical factors such as the diagnoses or preexisting conditions were not considered in the decision-tool algorithm. This original algorithm (Algorithm 1) was later modified and the list of suspected nephrotoxins was expanded (Table 1) in order to align with a national pediatric AKI collaborative (Algorithm 2). However, it was unclear whether the algorithm modification would improve AKI detection.

Context

SCH is a tertiary care academic teaching hospital affiliated with the University of Washington School of Medicine, Harborview Medical Center, and the Seattle Cancer Care Alliance. The hospital has 371 licensed beds and approximately 18 medical subspecialty services.

Study Population

This was a retrospective cohort study examining all patients ages 0-21 years admitted to SCH between December 1, 2013 and November 30, 2015. The detection system was modified to align with the national pediatric AKI collaborative, Nephrotoxic Injury Negated by Just-in-Time Action (NINJA) in November 2014. Both acute care and intensive care patients were included (data not separated by location). Patients who had end-stage kidney disease and were receiving dialysis and patients who were evaluated in the emergency department without being admitted or admitted as observation status were excluded from analysis. Patients were also excluded if they did not have a baseline serum creatinine as defined below.

Study Measures

AKI is defined at SCH using the Kidney Disease: Improving Global Outcomes Stage 1 criteria as a guideline. The diagnosis of AKI is based on an increase in the baseline serum creatinine by 0.3 mg/dL or an increase in the serum creatinine by >1.5 times the baseline assuming the incoming creatinine is 0.5 mg/dL or higher. For our definition, the increase in serum creatinine needs to have occurred within a one-week timeframe and urine output is not a diagnostic criterion.¹⁵ Baseline serum creatinine is defined as the lowest serum creatinine in the previous six months. Forty medications were classified as nephrotoxins based on previous analysis¹⁶ and adapted for our institutional formulary.

Statistical Analysis

To evaluate the efficacy of our systems in detecting nephrotoxin-related AKI, the sensitivity and the specificity using both our original algorithm (Algorithm 1) and the modified algorithm (Algorithm 2) were generated on our complete data set. To test sensitivity, the proportion of AKI patients who would trigger alert using Algorithm 1 and then with Algorithm 2 was identified. Similarly, to test specificity, the proportion of non-AKI patients who did not trigger an alert by the surveillance systems was identified. The differences in sensitivity and specificity between the two algorithms were evaluated using two-sample tests of proportion.

The statistical method of Combinatorial Inference has been utilized in studies of cancer biology¹⁷ and in genomics.¹⁸ A variation of this approach was used in this study to identify the specific medication combinations most associated with AKI. First, all of the nephrotoxic medications and medication combinations that were prescribed during our study period were identified from a data set (ie, a training set) containing 75% of all encounters selected at random without replacement. Using this training set, the prevalence of each medication combination and the rate of AKI associated with each combination were identified. The predicted overall AKI risk of an individual medication is the average of all the AKI rates associated with each combination containing that specific medication. Also incorporated into the determination of the predicted AKI risk was the prevalence of that medication combination.

To test our model’s predictive capability, the algorithm was applied to the remaining 25% of the total patient data (ie, the test set). The predicted AKI risk was compared with the actual AKI rate in the test data set. Our model’s predictive capability was represented in a receiver operator characteristic (ROC) analysis. The goal was to achieve an area under the ROC curve (AUC) approaching one as this would reflect 100% sensitivity and 100% specificity, whereas an AUC of 0.5 would represent a random guess (50% chance of being correct).

Lastly, our final step was to use our model’s ROC curve to determine an optimal threshold of AKI risk for which to trigger an alert. This predicted risk threshold was based on our goal to increase our surveillance system’s sensitivity balanced with maintaining an acceptable specificity.

An a priori threshold of P = .05 was used to determine statistical significance of all results. Analyses were conducted in Stata 12.1 (StataCorp LP, College Station, Texas) and R 3.3.2 (R Foundation for Statistical Computing, Vienna, Austria). A sample data set containing replication code for our model can be found in an online repository (https://dataverse.harvard.edu/dataverse/chuan). This study was approved by the Seattle Children’s Institutional Review Board.

Sensitivity and Specificity

Of the patient encounters, 14,779 were eligible during the study period. The sensitivity of the system’s ability to identify nephrotoxin-related AKI decreased from 46.9% using Algorithm 1 to 43.3% using Algorithm 2, a change of 3.6% (P = .22). The specificity increased from 73.6% to 89.3%, a change of 15.7% (P < .001; Table 2).

Improvement of Our Nephrotoxin-Related AKI Detection System Using a Novel AKI Prediction Strategy

A total of 838 medication combinations were identified in our training set and the predicted AKI risk for every medication combination was determined. By comparing the predicted risk of AKI to the actual AKI occurrence, an ROC curve with an AUC of 0.756 (Figure) was generated. An increase in system sensitivity was prioritized when determining the optimal AKI risk at which the model would trigger an alert. Setting an alert threshold at a predicted AKI risk of >8%, our model performed with a sensitivity of 74% while decreasing the specificity to 70%.

Identification of High-Risk Nephrotoxic Medications and Medication Combinations

Approximately 200 medication combinations were associated with >8% AKI risk, our new AKI prediction model’s alert threshold. Medication combinations consisting of up to 11 concomitantly prescribed medications were present in our data set. However, many of these combinations were infrequently prescribed. Further analysis, conducted in order to increase the clinical relevance of our findings, identified 10 medications or medication combinations that were both associated with a predicted AKI risk of >8% and that were prescribed on average greater than twice a month (Table 3).

The nephrotoxin-related AKI detection system at SCH automatically places orders for serum creatinines on patients who have met criteria for concomitant nephrotoxin exposure. This has given us a robust database from which to develop our clinical decision-making tool. Both our original and updated systems were based on the absolute number of concomitant nephrotoxic medications prescribed.¹⁶ This is a reasonable approach given the complexity of building a surveillance system¹⁹ and resource limitations. However, a system based on observed rather than theoretical or in vitro data, adaptable to the institution and designed for ongoing refinement, would be more valuable.

The interest in AKI prediction tools continues to be high. Bedford et al. employed numerous variables and diagnostic codes to predict the development of AKI in adults during hospitalization. They were able to produce a prediction model with a reasonable fit (AUC 0.72) to identify patients at higher risk for AKI but were less successful in their attempts to predict progression to severe AKI.²⁰ Hodgson et al. recently developed an adult AKI prediction score (AUC 0.65-0.72) also based on numerous clinical factors that was able to positively impact inpatient mortality.²¹ To our knowledge, our model is unique in that it focuses on nephrotoxins using a predicted AKI risk algorithm based on observed AKI rates of previously ordered medications/medication combinations (two to 11 medications). Having a decision tool targeting medications gives the clinician guidance that can be used to make a specific intervention rather than identifying a patient at risk due to a diagnosis code or other difficult to modify factors.

There are abundant case studies and reports using logistic regression models identifying specific medications associated with AKI. Our choice of methodology was based on our assessment that logistic regression models would be inadequate for the development of a real-time clinical decision-making tool for several reasons. Using logistic regression to explore every medication combination based on our medication list would be challenging as there are approximately 5.5 × 10¹⁰ potential medication combinations. Additionally, logistic regression ignores any potential interactions between the medications. This is an important point as medication interactions can be synergistic, neutral, or antagonist. Consequently, the outcome generated from a set of combined variables may be different from one generated from the sum of each variable taken independently. Logistic regression also does not account for the potential prescribing trends among providers as it assumes that all medications or medication combinations are equally available at the same time. However, in practice, depending on numerous factors, such as hospital culture (eg, the presence of clinical standard work pathways), local bacterial resistance patterns, or medication shortages; certain medication combinations may occur more frequently while others not at all. Finally, logistic regression cannot account for the possibility of a medication combination occurring; therefore, logistic regression may identify a combination strongly associated with AKI that is rarely prescribed.

We theorized that AKI detection would improve with the Algorithm 2 modifications, including the expanded nephrotoxin list, which accompanied alignment with the national pediatric AKI collaborative, NINJA. The finding that our surveillance sensitivity did not improve with this system update supported our subsequent objective to develop a novel nephrotoxin-related AKI decision tool or detection system using our EHR data to identify which specific medications and/or medication combinations were associated with a higher rate of AKI. However, it should be noted that two factors related to measurement bias introduce limitations to our sensitivity and specificity analyses. First, regarding the presence of the alert system, our system will order serum creatinines on patients when they have been exposed to nephrotoxins. Consequently, the proportion of patients with creatinines measured will increase in the nephrotoxin-exposed patients. Unexposed patients may have AKI that is not detected because creatinines may not be ordered. Therefore, there is the potential for a relative increase in AKI detection among nephrotoxin-exposed patients as compared with unexposed patients, which would then affect the measured sensitivity and specificity of the alert. Second, the automated alerts require a baseline creatinine in order to trigger therefore are unable to identify AKI among patients who do not have a baseline serum creatinine measurement.

Our new nephrotoxin-related AKI detection model performed best when an alert was triggered for those medications or medication combinations with a predicted AKI risk of >8%. Forty-six medication combinations consisting of exactly two medications were determined to have a predicted AKI risk of >8% therefore would trigger an alert in our new model system. These medication combinations would not have triggered an alert using either of the previous system algorithms as both algorithms are based on the presence of three or more concomitant nephrotoxic medications.

From the list of suspected nephrotoxins, we identified 11 unique medications in 10 different combinations with a predicted AKI risk of >8% that were prescribed frequently (at least twice a month on average; Table 3). Notably, six out of 10 medication combinations involved vancomycin. Piperacillin-tazobactam was also represented in several combinations. These findings support the concern that others have reported regarding these two medications particularly when prescribed together.^22,23

Acknowledgments

Dr. Karyn E. Yonekawa conceptualized and designed the study, directed the data analysis, interpreted the data, drafted, revised and gave final approval of the manuscript. Dr. Chuan Zhou contributed to the study design, acquired data, conducted the data analysis, critically reviewed, and gave final approval of the manuscript. Ms. Wren L. Haaland contributed to the study design, acquired data, conducted the data analysis, critically reviewed, and gave final approval of the manuscript. Dr. Davene R. Wright contributed to the study design, data analysis, critically reviewed, revised, and gave final approval of the manuscript.

The authors would like to thank Holly Clifton and Suzanne Spencer for their assistance with data acquisition and Drs. Derya Caglar, Corrie McDaniel, and Thida Ong for their writing support.

All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

Disclosures

The authors have no conflicts of interest to report.

Patients frequently present to dermatologists with nail disorders as their chief concern. Alternatively, nail conditions may be encountered by the examining physician as an incidental finding that may be a clue to underlying systemic disease. Competence in the diagnosis and treatment of nail diseases can drastically improve patient quality of life and can be lifesaving,¹ but many dermatologists find management of nail diseases challenging.² Bridging this educational gap begins with dermatology resident and medical student education. In a collaboration with dermatology educators, the American Academy of Dermatology (AAD) prepared a free online core curriculum for medical students that covers the essential concepts of dermatology. We sought to determine the integration of nail education in the AAD Basic Dermatology Curriculum.

Methods

A cross-sectional study of the AAD Basic Dermatology Curriculum was conducted to determine nail disease content. The curriculum modules were downloaded in June 2018,³ and mentions of nails were recorded and evaluated for overall quantities and relevant content. References to nail procedures and diagnostic techniques including nail biopsies, fungal cultures, microscopy on nail scrapings, nail clippings, and nail-related cancers also were assessed in the analysis.

Results

Of 342 patients discussed in cases and quizzes, nails were mentioned for 19 patients (89 times total)(Table 1). Additionally, there were 2 mentions each of nail clippings and nail tumors, 0 mentions of nail biopsies, and 1 mention each of fungal cultures and microscopy on nail scrapings (Table 1). Of the 40 modules, nails were mentioned in 12 modules (Table 2) and 6 introductions to the modules (Table 1). There were no mentions of the terms nails, subungual, or onychomycosis in the learning objectives.³

Comment

Our study demonstrates a paucity of content relevant to nails in the AAD Basic Dermatology Curriculum. Medical students are missing an important opportunity to learn about diagnosis and management of nail conditions and may incorrectly conclude that nail expertise is not essential to becoming a competent board-certified dermatologist.

Particularly concerning is the exclusion of nail examinations in the skin exam module addressing full-body skin examinations (0 mentions in 31 slides). This curriculum may negatively influence medical students and may then follow at the resident level, with a study reporting that 50.3% (69/137) of residents examine nails only when the patient brings it to their attention.⁴

Most concerning was the inadequate coverage of nail unit melanoma in the melanoma module (1 mention in 53 slides). Furthermore, the ABCDE—asymmetry, border, color, diameter, and evolving—mnemonic for cutaneous melanoma was covered in 6 slides in this module, and the ABCDEF—family history added—mnemonic for nail unit melanoma was completely excluded. Not surprisingly, resident knowledge of melanonychia diagnosis is deficient, with a prior study demonstrating that 62% (88/142) of residents were not confident diagnosing and managing patients with melanonychia, and only 88% (125/142) of residents were aware of the nail melanoma mnemonic.⁴

Similarly, nail biopsy for melanonychia diagnosis was excluded from the curriculum, whereas skin biopsy was thoroughly discussed in the context of a cutaneous melanoma diagnosis. This deficient teaching may track to the dermatology resident curriculum, as a survey of third-year dermatology residents (N=240) showed that 58% performed 10 or fewer nail procedures, and one-third of residents felt incompetent in nail surgery.⁵

We acknowledge that the AAD Basic Dermatology Curriculum is simply an introduction to dermatology. However, given that dermatologists are among the major specialists who care for nail patients, we advocate for more content on nail diseases in this curriculum. Nails can easily be incorporated into existing modules, and a new module specifically dedicated to nail disease should be added. Moreover, we envision that our findings will positively reflect on competence in treating nail disease for dermatology residents.

Patients frequently present to dermatologists with nail disorders as their chief concern. Alternatively, nail conditions may be encountered by the examining physician as an incidental finding that may be a clue to underlying systemic disease. Competence in the diagnosis and treatment of nail diseases can drastically improve patient quality of life and can be lifesaving,¹ but many dermatologists find management of nail diseases challenging.² Bridging this educational gap begins with dermatology resident and medical student education. In a collaboration with dermatology educators, the American Academy of Dermatology (AAD) prepared a free online core curriculum for medical students that covers the essential concepts of dermatology. We sought to determine the integration of nail education in the AAD Basic Dermatology Curriculum.

Methods

A cross-sectional study of the AAD Basic Dermatology Curriculum was conducted to determine nail disease content. The curriculum modules were downloaded in June 2018,³ and mentions of nails were recorded and evaluated for overall quantities and relevant content. References to nail procedures and diagnostic techniques including nail biopsies, fungal cultures, microscopy on nail scrapings, nail clippings, and nail-related cancers also were assessed in the analysis.

Results

Of 342 patients discussed in cases and quizzes, nails were mentioned for 19 patients (89 times total)(Table 1). Additionally, there were 2 mentions each of nail clippings and nail tumors, 0 mentions of nail biopsies, and 1 mention each of fungal cultures and microscopy on nail scrapings (Table 1). Of the 40 modules, nails were mentioned in 12 modules (Table 2) and 6 introductions to the modules (Table 1). There were no mentions of the terms nails, subungual, or onychomycosis in the learning objectives.³

Comment

Our study demonstrates a paucity of content relevant to nails in the AAD Basic Dermatology Curriculum. Medical students are missing an important opportunity to learn about diagnosis and management of nail conditions and may incorrectly conclude that nail expertise is not essential to becoming a competent board-certified dermatologist.

Particularly concerning is the exclusion of nail examinations in the skin exam module addressing full-body skin examinations (0 mentions in 31 slides). This curriculum may negatively influence medical students and may then follow at the resident level, with a study reporting that 50.3% (69/137) of residents examine nails only when the patient brings it to their attention.⁴

Most concerning was the inadequate coverage of nail unit melanoma in the melanoma module (1 mention in 53 slides). Furthermore, the ABCDE—asymmetry, border, color, diameter, and evolving—mnemonic for cutaneous melanoma was covered in 6 slides in this module, and the ABCDEF—family history added—mnemonic for nail unit melanoma was completely excluded. Not surprisingly, resident knowledge of melanonychia diagnosis is deficient, with a prior study demonstrating that 62% (88/142) of residents were not confident diagnosing and managing patients with melanonychia, and only 88% (125/142) of residents were aware of the nail melanoma mnemonic.⁴

Similarly, nail biopsy for melanonychia diagnosis was excluded from the curriculum, whereas skin biopsy was thoroughly discussed in the context of a cutaneous melanoma diagnosis. This deficient teaching may track to the dermatology resident curriculum, as a survey of third-year dermatology residents (N=240) showed that 58% performed 10 or fewer nail procedures, and one-third of residents felt incompetent in nail surgery.⁵

We acknowledge that the AAD Basic Dermatology Curriculum is simply an introduction to dermatology. However, given that dermatologists are among the major specialists who care for nail patients, we advocate for more content on nail diseases in this curriculum. Nails can easily be incorporated into existing modules, and a new module specifically dedicated to nail disease should be added. Moreover, we envision that our findings will positively reflect on competence in treating nail disease for dermatology residents.

Patients frequently present to dermatologists with nail disorders as their chief concern. Alternatively, nail conditions may be encountered by the examining physician as an incidental finding that may be a clue to underlying systemic disease. Competence in the diagnosis and treatment of nail diseases can drastically improve patient quality of life and can be lifesaving,¹ but many dermatologists find management of nail diseases challenging.² Bridging this educational gap begins with dermatology resident and medical student education. In a collaboration with dermatology educators, the American Academy of Dermatology (AAD) prepared a free online core curriculum for medical students that covers the essential concepts of dermatology. We sought to determine the integration of nail education in the AAD Basic Dermatology Curriculum.

Methods

A cross-sectional study of the AAD Basic Dermatology Curriculum was conducted to determine nail disease content. The curriculum modules were downloaded in June 2018,³ and mentions of nails were recorded and evaluated for overall quantities and relevant content. References to nail procedures and diagnostic techniques including nail biopsies, fungal cultures, microscopy on nail scrapings, nail clippings, and nail-related cancers also were assessed in the analysis.

Results

Of 342 patients discussed in cases and quizzes, nails were mentioned for 19 patients (89 times total)(Table 1). Additionally, there were 2 mentions each of nail clippings and nail tumors, 0 mentions of nail biopsies, and 1 mention each of fungal cultures and microscopy on nail scrapings (Table 1). Of the 40 modules, nails were mentioned in 12 modules (Table 2) and 6 introductions to the modules (Table 1). There were no mentions of the terms nails, subungual, or onychomycosis in the learning objectives.³

Comment

Our study demonstrates a paucity of content relevant to nails in the AAD Basic Dermatology Curriculum. Medical students are missing an important opportunity to learn about diagnosis and management of nail conditions and may incorrectly conclude that nail expertise is not essential to becoming a competent board-certified dermatologist.

Particularly concerning is the exclusion of nail examinations in the skin exam module addressing full-body skin examinations (0 mentions in 31 slides). This curriculum may negatively influence medical students and may then follow at the resident level, with a study reporting that 50.3% (69/137) of residents examine nails only when the patient brings it to their attention.⁴

Most concerning was the inadequate coverage of nail unit melanoma in the melanoma module (1 mention in 53 slides). Furthermore, the ABCDE—asymmetry, border, color, diameter, and evolving—mnemonic for cutaneous melanoma was covered in 6 slides in this module, and the ABCDEF—family history added—mnemonic for nail unit melanoma was completely excluded. Not surprisingly, resident knowledge of melanonychia diagnosis is deficient, with a prior study demonstrating that 62% (88/142) of residents were not confident diagnosing and managing patients with melanonychia, and only 88% (125/142) of residents were aware of the nail melanoma mnemonic.⁴

Similarly, nail biopsy for melanonychia diagnosis was excluded from the curriculum, whereas skin biopsy was thoroughly discussed in the context of a cutaneous melanoma diagnosis. This deficient teaching may track to the dermatology resident curriculum, as a survey of third-year dermatology residents (N=240) showed that 58% performed 10 or fewer nail procedures, and one-third of residents felt incompetent in nail surgery.⁵

We acknowledge that the AAD Basic Dermatology Curriculum is simply an introduction to dermatology. However, given that dermatologists are among the major specialists who care for nail patients, we advocate for more content on nail diseases in this curriculum. Nails can easily be incorporated into existing modules, and a new module specifically dedicated to nail disease should be added. Moreover, we envision that our findings will positively reflect on competence in treating nail disease for dermatology residents.

Delirium is an acute change in mental status, affecting more than seven million hospitalized patients in the United States annually.¹ Several factors increase the risk of developing delirium, including advanced age,² cognitive dysfunction,³ hearing and vision impairment,^4-6 and severe illness or major surgery.⁷ Delirium may be precipitated during hospitalization by common inpatient interventions, such as the use of physical restraints, polypharmacy, or bladder catheters.^4,8 In-hospital delirium impacts an estimated 10%-15% of the general medical admissions and as many as 81% of patients in the intensive care unit (ICU).^9-11 Despite the relative frequency with which delirium is encountered in the hospital, subsequent emergency department (ED) presentations or hospital readmissions for these patients are poorly characterized.

The development of delirium is associated with several negative outcomes during the hospital stay. Delirium is an independent predictor of prolonged hospital stay,^7,9,12,13 prolonged mechanical ventilation,¹⁴ and mortality during admission.^14,15 Inpatient delirium is associated with functional decline at discharge, leading to a new nursing home placement.^16-19 Preexisting dementia is exacerbated by inpatient delirium, and a new diagnosis of cognitive impairment²⁰ or dementia becomes more common after an episode of delirium.²¹

These data suggest that people diagnosed with delirium may be particularly vulnerable in the posthospitalization period. Hospitals with high rates of unplanned readmissions face penalties from the Centers for Medicare and Medicaid Services.^22,23 However, few investigations have focused on postdischarge healthcare utilization, such as readmission rates and ED visits. Studies that address this topic are limited to postoperative patient populations.²⁴

Using a cohort of hospitalized patients, we examined whether those diagnosed with delirium experienced worse outcomes compared with patients with no such condition. We hypothesized that the patients diagnosed with delirium during hospitalization would experience more readmissions and ED visits within 30 days of discharge compared with those without delirium.

Study Design

This single-center retrospective cohort study took place at the Kaiser Permanente San Rafael Medical Center (KP-SRF), a 116-bed general community medical and surgical hospital located in Northern California, from September 6, 2010 to March 31, 2015. The Kaiser Permanente Northern California institutional review board, in accordance with the provisions of the Declaration of the Helsinki and International Conference on Harmonization Guidelines for Good Clinical Practice (CN-15-2491-H), approved this study.

Participants and Eligibility Criteria

This study included Kaiser Permanente members at least 65 years old who were hospitalized at KP-SRF from September 2010 to March 2015. Patient data were obtained from the electronic medical records. Patients with delirium were identified from a delirium registry; all other patients served as controls.

Starting on September 6, 2010, a hospital-wide program was initiated to screen hospitalized medical and surgical patients using the Confusion Assessment Method (CAM).²⁵ As part of this program, nurses completed a four-hour training on delirium; the program included delirium identification and CAM administration. Patients deemed at risk for delirium by their nurse or displaying symptoms of delirium (fluctuation in attention or awareness, disorientation, restlessness, agitation, and psychomotor slowing) were screened by nurses one to two times within a 24-hour period. Physicians were notified by the nurse if their patient screened positive. Nurses were prohibited from performing CAMs in languages that they were not fluent in, thus resulting in screening of primarily English-speaking patients. Psychiatry was consulted at the discretion of the primary team physician to assist with diagnosis and management of delirium. As psychiatry consultation was left up to the discretion of the primary team physician, not all CAM-positive patients were evaluated. The psychiatrists conducted no routine evaluation on the CAM-negative patients unless requested by the primary team physician. The psychiatrist confirmed the delirium diagnosis with a clinical interview and assessment. The patients confirmed with delirium at any point during their hospitalization were prospectively added to a delirium registry. The patients assessed by the psychiatrist as not delirious were excluded from the registry. Only those patients added to the delirium registry during the study period were classified as delirious for this study. All other patients were included as controls. The presence of the nursing screening program using the CAM enriched the cohort, but a positive CAM was unnecessary nor was it sufficient for inclusion in the delirium group (Table 1).

Patient Characteristics

Patient demographics and clinical data were obtained from the electronic medical records. We used several scores to characterize illness severity, including the Charlson comorbidity index,²⁷ Laboratory-Based Acute Physiology, version 2 (LAPS2) score²⁸—an externally validated score for acute severity of illness—and disease categories as defined by the Healthcare Cost and Utilization Project (HCUP).²⁹

Outcomes

The primary outcome was the rate of readmission to the hospital within 30 days of discharge from the hospitalization in which delirium was first diagnosed. Readmissions and ED visits to any Kaiser Permanente hospital and to hospitals outside of the Kaiser Permanente network with Kaiser Permanente insurance were captured. To avoid incorrectly coding patients transferred from the index hospital to another hospital as readmissions, we excluded readmissions that occurred on the day of discharge or the following calendar day. This action was expected to lower the absolute number of readmissions but restrict the analysis to true readmissions. The models of postdischarge outcomes are based on the subset of patients discharged alive. The secondary outcome measures included discharge from the index hospitalization to a skilled nursing facility or hospice rather than to home and emergency room visits within 30 days of discharge. We also quantified rates of mortality during hospitalization and at 30 days postdischarge.

Statistical Analysis

Comparisons between patients with delirium and those without were performed using Pearson’s X² test for categorical variables and student t-test for continuous variables. The estimated odds of our outcome measures for delirious and nondelirious subjects were calculated from multivariable logistic regression models, which controlled for predictors of delirium and additional information obtained during the hospitalization. For inpatient outcomes (in-hospital mortality and discharge to skilled nursing facility or hospice), we adjusted only for admission characteristics: age, race/ethnicity, admission to ICU, Charlson comorbidity index, HCUP category, and admission category. To limit the number of variables in our model, we consolidated the initial 30 HCUP categories (Appendix Table 1) by illness type into 13 categories (Appendix Table 2). For postdischarge outcomes, we adjusted for all the variables, including disposition (Table 2). The average estimated odds were calculated based on the observed marginal distribution of the control variables. The P value indicates how likely the odds on each outcome for delirious subjects differed significantly from those for other subjects. All statistical analyses were performed using SAS 9.3 (SAS Institute Inc., Cary, North Carolina).

Demographics and Clinical Characteristics

A total of 718 patients with delirium and 7,927 patients without delirium were included in this study. The related demographic information is outlined in Table 2. On average, the patients with delirium were older (83 ± 8 years versus 77 ± 8 years, P < .0001) but no difference in gender distribution was observed between groups. A similar racial breakdown was noted between groups, with white patients accounting for 87% of both patients with delirium and those without. The majority of admissions were unplanned medical admissions. The delirium cohort included more emergent surgical admissions compared with patients who did not develop delirium. Patients who developed delirium exhibited higher levels of illness severity on admission, as measured by the Charlson and LAPS2 scores, and were more often admitted to the ICU. Significant differences were also observed between admission illness categories between patients with delirium and those without.

Primary Outcome

Delirium during admission was significantly associated with hospital readmission within 30 days of discharge (adjusted odds ratio [aOR] = 2.60, 95% CI: 1.96–3.44; P < .0001; Table 3).

Secondary Outcomes

Delirium during admission was significantly (P < .0001; Table 3) associated with an ED visit within 30 days of discharge (OR: 2.18; 95% CI: 1.77–2.69) and discharge to a skilled nursing facility or hospice rather than home (OR: 2.52; 95% CI: 2.09–3.01). Delirium was not associated (P > .1) with death during hospitalization nor death 30 days following discharge.

As the delirious patients were much more likely to be discharged to a skilled nursing facility than nondelirious patients, we tested whether discharge disposition influenced readmission rates and ED visits between delirious and nondelirious patients in an unadjusted univariate analysis. The association between delirium and readmission and ED utilization was present regardless of disposition. Among patients discharged to skilled nursing, readmission rates were 4.76% and 13.38% (P < .001), and ED visit rates were 12.29% and 23.24% (P < .001) for nondelirious and delirious patients, respectively. Among patients discharged home, readmission rates were 4.96% and 14.37% (P < .001), and ED visit rates were 11.93% and 29.04% (P < .001) for nondelirious and delirious patients, respectively.

In this study of patients in a community hospital in Northern California, we observed a significant association between inpatient delirium and risk of hospital readmission within 30 days of discharge. We also demonstrated increased skilled nursing facility placement and ED utilization after discharge among hospitalized patients with delirium compared with those without. Patients with delirium in this study were diagnosed by a psychiatrist—a gold standard³⁰—and the study was conducted in a health system database with near comprehensive ascertainment of readmissions. These results suggest that patients with delirium are particularly vulnerable in the posthospitalization period and are a key group to focusing on reducing readmission rates and postdischarge healthcare utilization.

Identifying the risk factors for hospital readmission is important for the benefit of both the patient and the hospital. In an analysis of Medicare claims data from 2003 to 2004, 19.6% of beneficiaries were readmitted within 30 days of discharge.³¹ There is a national effort to reduce unplanned hospital readmissions for both patient safety as hospitals with high readmission rates face penalties from the Centers for Medicare and Medicaid Services.^22,23 Why delirium is associated with readmission remains unclear. Delirium may precipitate aspiration events, reduce oral intake which complicates medication administration and nutrition, or reduced mobility, leading to pulmonary emboli and skin breakdown, any of which could lead to readmission.³² Delirium may also accelerate the progression of cognitive decline and overall loss of functional independence.²⁰ Delirious patients can be difficult to care for at home, and persistent delirium may lead to returns to the ED and readmission. Strategies to reduce readmissions associated with delirium may need to focus on both prevention of hospital-acquired delirium and targeted caregiver and patient support after discharge.

Hospital readmission and ED visits are not mutually exclusive experiences. In the United States, the majority of patients admitted to the hospital are admitted through the ED.³³ Thus, most of the readmissions in this cohort were also likely counted as 30-day ED visits. However, as ED utilization occurs regardless of whether a patient is discharged or admitted from the ED, we reported all ED visits in this analysis, similar to other studies.³⁴ More delirium patients returned to the ED 30 days postdischarge than were ultimately readmitted to the hospital, and delirious patients were more likely to visit the ED or be readmitted than nondelirious patients. These observations point toward the first 30 days after discharge as a crucial period for these patients.

Our study features several strengths. To our knowledge, this study is one of the largest investigations of inpatients with delirium. One distinguishing feature was that all cases of delirium in this study were diagnosed by a psychiatrist, which is considered a gold standard. Many studies rely solely on brief nursing-administered surveys for delirium diagnosis. Using Kaiser Permanente data allowed for more complete follow-up of patients, including vital status. Kaiser Permanente is both a medical system and an insurer, resulting in acquisition of detailed health information from all hospitalizations where Kaiser Permanente insurance was used for each patient. Therefore, patients were only lost to follow-up following discharge in the event of a membership lapse; these patients were excluded from analysis. The obtained data are also more generalizable than those of other studies examining readmission rates in delirious patients as the hospital where these data were collected is a 116-bed general community medical and surgical hospital. Thus, the patients enrolled in this study covered multiple hospital services with a variety of admission diagnoses. This condition contrasts with much of the existing literature on inpatient delirium; these studies mostly center on specific medical conditions or surgeries and are often conducted at academic medical centers. At the same time, Kaiser Permanente is a unique health maintenance organization focused on preventive care, and readmission rates are possibly lower than elsewhere given the universal access to primary care for Kaiser Permanente members. Our results may not generalize to patients hospitalized in other health systems.

The diagnosis of delirium is a clinical diagnosis without biomarkers or radiographic markers and is also underdiagnosed and poorly coded.³² For these reasons, delirium can be challenging to study in large administrative databases or data derived from electronic medical records. We addressed this limitation by classifying the delirium patients only when they had been diagnosed by a staff psychiatrist. However, not all patients who screened positive with the CAM were evaluated by the staff psychiatrist during the study period. Thus, several CAM-positive patients who were not evaluated by psychiatry were included in the control population. This situation may cause bias toward identification of more severe cases of delirium. Although the physicians were encouraged to consult the psychiatry department for any patients who screened positive for delirium with the CAM, the psychiatrist may not have been involved if patients were managed without consultation. These patients may have exhibited less severe delirium or hypoactive delirium. In addition, the CAM fails to detect all delirious patients; interrater variability may occur with CAM administration, and non-English speaking patients are more likely to be excluded.³⁵ These situations are another possible way for our control population to include some delirious patients and those patients with less severe or hypoactive subtypes. While this might bias toward the null hypothesis, it is also possible our results only indicate an association between more clinically apparent delirium and readmission. A major limitation of this study is that we were unable to quantify the number of cohort patients screened with the CAM or the results of screening, thus limiting our ability to quantify the impact of potential biases introduced by the screening program.

This study may have underestimated readmission rates. We defined readmissions as all hospitalizations at any Kaiser Permanente facility, or to an alternate facility where Kaiser Permanente insurance was used, within 30 days of discharge. We excluded the day of discharge or the following calendar day to avoid mischaracterizing transfers from the index hospital to another Kaiser Permanente facility as readmissions. This step was conducted to avoid biasing our comparison, as delirious patients are less frequently discharged home than nondelirious patients. Therefore, while the relative odds of readmission between delirious and nondelirious patients reported in this study should be generalizable to other community hospitals, the absolute readmission rates reported here may not be comparable to those reported in other studies.

Delirium may represent a marker of more severe illness or medical complications accrued during the hospitalization, which could lead to the associations observed in this study due to confounding.³² Patients with delirium are more likely to be admitted emergently, admitted to the ICU, and feature higher acuity conditions than patients without delirium. We attempted to mitigate this possibility by using a multivariable model to control for variables related to illness severity, including the Charlson comorbidity index, HCUP diagnostic categories, and ICU admission. Despite including HCUP diagnostic categories in our model, we were unable to capture the contribution of certain diseases with finer granularity, such as preexistent dementia, which may also affect clinical outcomes.³⁶ Similarly, although we incorporated markers of illness severity into our model, we were unable to adjust for baseline functional status or frailty, which were not reliably recorded in the electronic medical record but are potential confounders when investigating clinical outcomes including hospital readmission.

We also lacked information regarding the duration of delirium in our cohort. Therefore, we were unable to test whether longer episodes of delirium were more predictive of readmission than shorter episodes.

In-hospital delirium is associated with several negative patient outcomes. Our study demonstrates that delirium predicts 30-day readmission and emergency department utilization after hospital discharge. Bearing in mind that a third of hospital-acquired delirium cases may be preventable,³² hospitals should prioritize interventions to reduce postdischarge healthcare utilization and complications in this particularly vulnerable group.

Acknowledgments

The authors would like to acknowledge Dr. Andrew L. Avins for his guidance with the initial development of this project and Julie Fourie for contributing data to the overall study.

Disclosures

Dr. Liu receives funding from NIH K23GM112018 and NIGMS R35128672. Dr. Josephson receives compensation as the JAMA Neurology Editor in Chief and Continuum Audio Associate Editor. The remaining authors have no conflicts of interest.

Funding

This study was funded by Kaiser Permanente Graduate Medical Education, who approved the design, conduct, and reporting of this study.

Delirium is an acute change in mental status, affecting more than seven million hospitalized patients in the United States annually.¹ Several factors increase the risk of developing delirium, including advanced age,² cognitive dysfunction,³ hearing and vision impairment,^4-6 and severe illness or major surgery.⁷ Delirium may be precipitated during hospitalization by common inpatient interventions, such as the use of physical restraints, polypharmacy, or bladder catheters.^4,8 In-hospital delirium impacts an estimated 10%-15% of the general medical admissions and as many as 81% of patients in the intensive care unit (ICU).^9-11 Despite the relative frequency with which delirium is encountered in the hospital, subsequent emergency department (ED) presentations or hospital readmissions for these patients are poorly characterized.

The development of delirium is associated with several negative outcomes during the hospital stay. Delirium is an independent predictor of prolonged hospital stay,^7,9,12,13 prolonged mechanical ventilation,¹⁴ and mortality during admission.^14,15 Inpatient delirium is associated with functional decline at discharge, leading to a new nursing home placement.^16-19 Preexisting dementia is exacerbated by inpatient delirium, and a new diagnosis of cognitive impairment²⁰ or dementia becomes more common after an episode of delirium.²¹

These data suggest that people diagnosed with delirium may be particularly vulnerable in the posthospitalization period. Hospitals with high rates of unplanned readmissions face penalties from the Centers for Medicare and Medicaid Services.^22,23 However, few investigations have focused on postdischarge healthcare utilization, such as readmission rates and ED visits. Studies that address this topic are limited to postoperative patient populations.²⁴

Using a cohort of hospitalized patients, we examined whether those diagnosed with delirium experienced worse outcomes compared with patients with no such condition. We hypothesized that the patients diagnosed with delirium during hospitalization would experience more readmissions and ED visits within 30 days of discharge compared with those without delirium.

Study Design

This single-center retrospective cohort study took place at the Kaiser Permanente San Rafael Medical Center (KP-SRF), a 116-bed general community medical and surgical hospital located in Northern California, from September 6, 2010 to March 31, 2015. The Kaiser Permanente Northern California institutional review board, in accordance with the provisions of the Declaration of the Helsinki and International Conference on Harmonization Guidelines for Good Clinical Practice (CN-15-2491-H), approved this study.

Participants and Eligibility Criteria

This study included Kaiser Permanente members at least 65 years old who were hospitalized at KP-SRF from September 2010 to March 2015. Patient data were obtained from the electronic medical records. Patients with delirium were identified from a delirium registry; all other patients served as controls.

Starting on September 6, 2010, a hospital-wide program was initiated to screen hospitalized medical and surgical patients using the Confusion Assessment Method (CAM).²⁵ As part of this program, nurses completed a four-hour training on delirium; the program included delirium identification and CAM administration. Patients deemed at risk for delirium by their nurse or displaying symptoms of delirium (fluctuation in attention or awareness, disorientation, restlessness, agitation, and psychomotor slowing) were screened by nurses one to two times within a 24-hour period. Physicians were notified by the nurse if their patient screened positive. Nurses were prohibited from performing CAMs in languages that they were not fluent in, thus resulting in screening of primarily English-speaking patients. Psychiatry was consulted at the discretion of the primary team physician to assist with diagnosis and management of delirium. As psychiatry consultation was left up to the discretion of the primary team physician, not all CAM-positive patients were evaluated. The psychiatrists conducted no routine evaluation on the CAM-negative patients unless requested by the primary team physician. The psychiatrist confirmed the delirium diagnosis with a clinical interview and assessment. The patients confirmed with delirium at any point during their hospitalization were prospectively added to a delirium registry. The patients assessed by the psychiatrist as not delirious were excluded from the registry. Only those patients added to the delirium registry during the study period were classified as delirious for this study. All other patients were included as controls. The presence of the nursing screening program using the CAM enriched the cohort, but a positive CAM was unnecessary nor was it sufficient for inclusion in the delirium group (Table 1).

Patient Characteristics

Patient demographics and clinical data were obtained from the electronic medical records. We used several scores to characterize illness severity, including the Charlson comorbidity index,²⁷ Laboratory-Based Acute Physiology, version 2 (LAPS2) score²⁸—an externally validated score for acute severity of illness—and disease categories as defined by the Healthcare Cost and Utilization Project (HCUP).²⁹

Outcomes

The primary outcome was the rate of readmission to the hospital within 30 days of discharge from the hospitalization in which delirium was first diagnosed. Readmissions and ED visits to any Kaiser Permanente hospital and to hospitals outside of the Kaiser Permanente network with Kaiser Permanente insurance were captured. To avoid incorrectly coding patients transferred from the index hospital to another hospital as readmissions, we excluded readmissions that occurred on the day of discharge or the following calendar day. This action was expected to lower the absolute number of readmissions but restrict the analysis to true readmissions. The models of postdischarge outcomes are based on the subset of patients discharged alive. The secondary outcome measures included discharge from the index hospitalization to a skilled nursing facility or hospice rather than to home and emergency room visits within 30 days of discharge. We also quantified rates of mortality during hospitalization and at 30 days postdischarge.

Statistical Analysis

Comparisons between patients with delirium and those without were performed using Pearson’s X² test for categorical variables and student t-test for continuous variables. The estimated odds of our outcome measures for delirious and nondelirious subjects were calculated from multivariable logistic regression models, which controlled for predictors of delirium and additional information obtained during the hospitalization. For inpatient outcomes (in-hospital mortality and discharge to skilled nursing facility or hospice), we adjusted only for admission characteristics: age, race/ethnicity, admission to ICU, Charlson comorbidity index, HCUP category, and admission category. To limit the number of variables in our model, we consolidated the initial 30 HCUP categories (Appendix Table 1) by illness type into 13 categories (Appendix Table 2). For postdischarge outcomes, we adjusted for all the variables, including disposition (Table 2). The average estimated odds were calculated based on the observed marginal distribution of the control variables. The P value indicates how likely the odds on each outcome for delirious subjects differed significantly from those for other subjects. All statistical analyses were performed using SAS 9.3 (SAS Institute Inc., Cary, North Carolina).

Demographics and Clinical Characteristics

A total of 718 patients with delirium and 7,927 patients without delirium were included in this study. The related demographic information is outlined in Table 2. On average, the patients with delirium were older (83 ± 8 years versus 77 ± 8 years, P < .0001) but no difference in gender distribution was observed between groups. A similar racial breakdown was noted between groups, with white patients accounting for 87% of both patients with delirium and those without. The majority of admissions were unplanned medical admissions. The delirium cohort included more emergent surgical admissions compared with patients who did not develop delirium. Patients who developed delirium exhibited higher levels of illness severity on admission, as measured by the Charlson and LAPS2 scores, and were more often admitted to the ICU. Significant differences were also observed between admission illness categories between patients with delirium and those without.

Primary Outcome

Delirium during admission was significantly associated with hospital readmission within 30 days of discharge (adjusted odds ratio [aOR] = 2.60, 95% CI: 1.96–3.44; P < .0001; Table 3).

Secondary Outcomes

Delirium during admission was significantly (P < .0001; Table 3) associated with an ED visit within 30 days of discharge (OR: 2.18; 95% CI: 1.77–2.69) and discharge to a skilled nursing facility or hospice rather than home (OR: 2.52; 95% CI: 2.09–3.01). Delirium was not associated (P > .1) with death during hospitalization nor death 30 days following discharge.

As the delirious patients were much more likely to be discharged to a skilled nursing facility than nondelirious patients, we tested whether discharge disposition influenced readmission rates and ED visits between delirious and nondelirious patients in an unadjusted univariate analysis. The association between delirium and readmission and ED utilization was present regardless of disposition. Among patients discharged to skilled nursing, readmission rates were 4.76% and 13.38% (P < .001), and ED visit rates were 12.29% and 23.24% (P < .001) for nondelirious and delirious patients, respectively. Among patients discharged home, readmission rates were 4.96% and 14.37% (P < .001), and ED visit rates were 11.93% and 29.04% (P < .001) for nondelirious and delirious patients, respectively.

In this study of patients in a community hospital in Northern California, we observed a significant association between inpatient delirium and risk of hospital readmission within 30 days of discharge. We also demonstrated increased skilled nursing facility placement and ED utilization after discharge among hospitalized patients with delirium compared with those without. Patients with delirium in this study were diagnosed by a psychiatrist—a gold standard³⁰—and the study was conducted in a health system database with near comprehensive ascertainment of readmissions. These results suggest that patients with delirium are particularly vulnerable in the posthospitalization period and are a key group to focusing on reducing readmission rates and postdischarge healthcare utilization.

Identifying the risk factors for hospital readmission is important for the benefit of both the patient and the hospital. In an analysis of Medicare claims data from 2003 to 2004, 19.6% of beneficiaries were readmitted within 30 days of discharge.³¹ There is a national effort to reduce unplanned hospital readmissions for both patient safety as hospitals with high readmission rates face penalties from the Centers for Medicare and Medicaid Services.^22,23 Why delirium is associated with readmission remains unclear. Delirium may precipitate aspiration events, reduce oral intake which complicates medication administration and nutrition, or reduced mobility, leading to pulmonary emboli and skin breakdown, any of which could lead to readmission.³² Delirium may also accelerate the progression of cognitive decline and overall loss of functional independence.²⁰ Delirious patients can be difficult to care for at home, and persistent delirium may lead to returns to the ED and readmission. Strategies to reduce readmissions associated with delirium may need to focus on both prevention of hospital-acquired delirium and targeted caregiver and patient support after discharge.

Hospital readmission and ED visits are not mutually exclusive experiences. In the United States, the majority of patients admitted to the hospital are admitted through the ED.³³ Thus, most of the readmissions in this cohort were also likely counted as 30-day ED visits. However, as ED utilization occurs regardless of whether a patient is discharged or admitted from the ED, we reported all ED visits in this analysis, similar to other studies.³⁴ More delirium patients returned to the ED 30 days postdischarge than were ultimately readmitted to the hospital, and delirious patients were more likely to visit the ED or be readmitted than nondelirious patients. These observations point toward the first 30 days after discharge as a crucial period for these patients.

Our study features several strengths. To our knowledge, this study is one of the largest investigations of inpatients with delirium. One distinguishing feature was that all cases of delirium in this study were diagnosed by a psychiatrist, which is considered a gold standard. Many studies rely solely on brief nursing-administered surveys for delirium diagnosis. Using Kaiser Permanente data allowed for more complete follow-up of patients, including vital status. Kaiser Permanente is both a medical system and an insurer, resulting in acquisition of detailed health information from all hospitalizations where Kaiser Permanente insurance was used for each patient. Therefore, patients were only lost to follow-up following discharge in the event of a membership lapse; these patients were excluded from analysis. The obtained data are also more generalizable than those of other studies examining readmission rates in delirious patients as the hospital where these data were collected is a 116-bed general community medical and surgical hospital. Thus, the patients enrolled in this study covered multiple hospital services with a variety of admission diagnoses. This condition contrasts with much of the existing literature on inpatient delirium; these studies mostly center on specific medical conditions or surgeries and are often conducted at academic medical centers. At the same time, Kaiser Permanente is a unique health maintenance organization focused on preventive care, and readmission rates are possibly lower than elsewhere given the universal access to primary care for Kaiser Permanente members. Our results may not generalize to patients hospitalized in other health systems.

The diagnosis of delirium is a clinical diagnosis without biomarkers or radiographic markers and is also underdiagnosed and poorly coded.³² For these reasons, delirium can be challenging to study in large administrative databases or data derived from electronic medical records. We addressed this limitation by classifying the delirium patients only when they had been diagnosed by a staff psychiatrist. However, not all patients who screened positive with the CAM were evaluated by the staff psychiatrist during the study period. Thus, several CAM-positive patients who were not evaluated by psychiatry were included in the control population. This situation may cause bias toward identification of more severe cases of delirium. Although the physicians were encouraged to consult the psychiatry department for any patients who screened positive for delirium with the CAM, the psychiatrist may not have been involved if patients were managed without consultation. These patients may have exhibited less severe delirium or hypoactive delirium. In addition, the CAM fails to detect all delirious patients; interrater variability may occur with CAM administration, and non-English speaking patients are more likely to be excluded.³⁵ These situations are another possible way for our control population to include some delirious patients and those patients with less severe or hypoactive subtypes. While this might bias toward the null hypothesis, it is also possible our results only indicate an association between more clinically apparent delirium and readmission. A major limitation of this study is that we were unable to quantify the number of cohort patients screened with the CAM or the results of screening, thus limiting our ability to quantify the impact of potential biases introduced by the screening program.

This study may have underestimated readmission rates. We defined readmissions as all hospitalizations at any Kaiser Permanente facility, or to an alternate facility where Kaiser Permanente insurance was used, within 30 days of discharge. We excluded the day of discharge or the following calendar day to avoid mischaracterizing transfers from the index hospital to another Kaiser Permanente facility as readmissions. This step was conducted to avoid biasing our comparison, as delirious patients are less frequently discharged home than nondelirious patients. Therefore, while the relative odds of readmission between delirious and nondelirious patients reported in this study should be generalizable to other community hospitals, the absolute readmission rates reported here may not be comparable to those reported in other studies.

Delirium may represent a marker of more severe illness or medical complications accrued during the hospitalization, which could lead to the associations observed in this study due to confounding.³² Patients with delirium are more likely to be admitted emergently, admitted to the ICU, and feature higher acuity conditions than patients without delirium. We attempted to mitigate this possibility by using a multivariable model to control for variables related to illness severity, including the Charlson comorbidity index, HCUP diagnostic categories, and ICU admission. Despite including HCUP diagnostic categories in our model, we were unable to capture the contribution of certain diseases with finer granularity, such as preexistent dementia, which may also affect clinical outcomes.³⁶ Similarly, although we incorporated markers of illness severity into our model, we were unable to adjust for baseline functional status or frailty, which were not reliably recorded in the electronic medical record but are potential confounders when investigating clinical outcomes including hospital readmission.

We also lacked information regarding the duration of delirium in our cohort. Therefore, we were unable to test whether longer episodes of delirium were more predictive of readmission than shorter episodes.

In-hospital delirium is associated with several negative patient outcomes. Our study demonstrates that delirium predicts 30-day readmission and emergency department utilization after hospital discharge. Bearing in mind that a third of hospital-acquired delirium cases may be preventable,³² hospitals should prioritize interventions to reduce postdischarge healthcare utilization and complications in this particularly vulnerable group.

Acknowledgments

The authors would like to acknowledge Dr. Andrew L. Avins for his guidance with the initial development of this project and Julie Fourie for contributing data to the overall study.

Disclosures

Dr. Liu receives funding from NIH K23GM112018 and NIGMS R35128672. Dr. Josephson receives compensation as the JAMA Neurology Editor in Chief and Continuum Audio Associate Editor. The remaining authors have no conflicts of interest.

Funding

This study was funded by Kaiser Permanente Graduate Medical Education, who approved the design, conduct, and reporting of this study.

Delirium is an acute change in mental status, affecting more than seven million hospitalized patients in the United States annually.¹ Several factors increase the risk of developing delirium, including advanced age,² cognitive dysfunction,³ hearing and vision impairment,^4-6 and severe illness or major surgery.⁷ Delirium may be precipitated during hospitalization by common inpatient interventions, such as the use of physical restraints, polypharmacy, or bladder catheters.^4,8 In-hospital delirium impacts an estimated 10%-15% of the general medical admissions and as many as 81% of patients in the intensive care unit (ICU).^9-11 Despite the relative frequency with which delirium is encountered in the hospital, subsequent emergency department (ED) presentations or hospital readmissions for these patients are poorly characterized.

The development of delirium is associated with several negative outcomes during the hospital stay. Delirium is an independent predictor of prolonged hospital stay,^7,9,12,13 prolonged mechanical ventilation,¹⁴ and mortality during admission.^14,15 Inpatient delirium is associated with functional decline at discharge, leading to a new nursing home placement.^16-19 Preexisting dementia is exacerbated by inpatient delirium, and a new diagnosis of cognitive impairment²⁰ or dementia becomes more common after an episode of delirium.²¹

These data suggest that people diagnosed with delirium may be particularly vulnerable in the posthospitalization period. Hospitals with high rates of unplanned readmissions face penalties from the Centers for Medicare and Medicaid Services.^22,23 However, few investigations have focused on postdischarge healthcare utilization, such as readmission rates and ED visits. Studies that address this topic are limited to postoperative patient populations.²⁴

Using a cohort of hospitalized patients, we examined whether those diagnosed with delirium experienced worse outcomes compared with patients with no such condition. We hypothesized that the patients diagnosed with delirium during hospitalization would experience more readmissions and ED visits within 30 days of discharge compared with those without delirium.

Study Design

This single-center retrospective cohort study took place at the Kaiser Permanente San Rafael Medical Center (KP-SRF), a 116-bed general community medical and surgical hospital located in Northern California, from September 6, 2010 to March 31, 2015. The Kaiser Permanente Northern California institutional review board, in accordance with the provisions of the Declaration of the Helsinki and International Conference on Harmonization Guidelines for Good Clinical Practice (CN-15-2491-H), approved this study.

Participants and Eligibility Criteria

This study included Kaiser Permanente members at least 65 years old who were hospitalized at KP-SRF from September 2010 to March 2015. Patient data were obtained from the electronic medical records. Patients with delirium were identified from a delirium registry; all other patients served as controls.

Starting on September 6, 2010, a hospital-wide program was initiated to screen hospitalized medical and surgical patients using the Confusion Assessment Method (CAM).²⁵ As part of this program, nurses completed a four-hour training on delirium; the program included delirium identification and CAM administration. Patients deemed at risk for delirium by their nurse or displaying symptoms of delirium (fluctuation in attention or awareness, disorientation, restlessness, agitation, and psychomotor slowing) were screened by nurses one to two times within a 24-hour period. Physicians were notified by the nurse if their patient screened positive. Nurses were prohibited from performing CAMs in languages that they were not fluent in, thus resulting in screening of primarily English-speaking patients. Psychiatry was consulted at the discretion of the primary team physician to assist with diagnosis and management of delirium. As psychiatry consultation was left up to the discretion of the primary team physician, not all CAM-positive patients were evaluated. The psychiatrists conducted no routine evaluation on the CAM-negative patients unless requested by the primary team physician. The psychiatrist confirmed the delirium diagnosis with a clinical interview and assessment. The patients confirmed with delirium at any point during their hospitalization were prospectively added to a delirium registry. The patients assessed by the psychiatrist as not delirious were excluded from the registry. Only those patients added to the delirium registry during the study period were classified as delirious for this study. All other patients were included as controls. The presence of the nursing screening program using the CAM enriched the cohort, but a positive CAM was unnecessary nor was it sufficient for inclusion in the delirium group (Table 1).

Patient Characteristics

Patient demographics and clinical data were obtained from the electronic medical records. We used several scores to characterize illness severity, including the Charlson comorbidity index,²⁷ Laboratory-Based Acute Physiology, version 2 (LAPS2) score²⁸—an externally validated score for acute severity of illness—and disease categories as defined by the Healthcare Cost and Utilization Project (HCUP).²⁹

Outcomes

The primary outcome was the rate of readmission to the hospital within 30 days of discharge from the hospitalization in which delirium was first diagnosed. Readmissions and ED visits to any Kaiser Permanente hospital and to hospitals outside of the Kaiser Permanente network with Kaiser Permanente insurance were captured. To avoid incorrectly coding patients transferred from the index hospital to another hospital as readmissions, we excluded readmissions that occurred on the day of discharge or the following calendar day. This action was expected to lower the absolute number of readmissions but restrict the analysis to true readmissions. The models of postdischarge outcomes are based on the subset of patients discharged alive. The secondary outcome measures included discharge from the index hospitalization to a skilled nursing facility or hospice rather than to home and emergency room visits within 30 days of discharge. We also quantified rates of mortality during hospitalization and at 30 days postdischarge.

Statistical Analysis

Comparisons between patients with delirium and those without were performed using Pearson’s X² test for categorical variables and student t-test for continuous variables. The estimated odds of our outcome measures for delirious and nondelirious subjects were calculated from multivariable logistic regression models, which controlled for predictors of delirium and additional information obtained during the hospitalization. For inpatient outcomes (in-hospital mortality and discharge to skilled nursing facility or hospice), we adjusted only for admission characteristics: age, race/ethnicity, admission to ICU, Charlson comorbidity index, HCUP category, and admission category. To limit the number of variables in our model, we consolidated the initial 30 HCUP categories (Appendix Table 1) by illness type into 13 categories (Appendix Table 2). For postdischarge outcomes, we adjusted for all the variables, including disposition (Table 2). The average estimated odds were calculated based on the observed marginal distribution of the control variables. The P value indicates how likely the odds on each outcome for delirious subjects differed significantly from those for other subjects. All statistical analyses were performed using SAS 9.3 (SAS Institute Inc., Cary, North Carolina).

Demographics and Clinical Characteristics

A total of 718 patients with delirium and 7,927 patients without delirium were included in this study. The related demographic information is outlined in Table 2. On average, the patients with delirium were older (83 ± 8 years versus 77 ± 8 years, P < .0001) but no difference in gender distribution was observed between groups. A similar racial breakdown was noted between groups, with white patients accounting for 87% of both patients with delirium and those without. The majority of admissions were unplanned medical admissions. The delirium cohort included more emergent surgical admissions compared with patients who did not develop delirium. Patients who developed delirium exhibited higher levels of illness severity on admission, as measured by the Charlson and LAPS2 scores, and were more often admitted to the ICU. Significant differences were also observed between admission illness categories between patients with delirium and those without.

Primary Outcome

Delirium during admission was significantly associated with hospital readmission within 30 days of discharge (adjusted odds ratio [aOR] = 2.60, 95% CI: 1.96–3.44; P < .0001; Table 3).

Secondary Outcomes

Delirium during admission was significantly (P < .0001; Table 3) associated with an ED visit within 30 days of discharge (OR: 2.18; 95% CI: 1.77–2.69) and discharge to a skilled nursing facility or hospice rather than home (OR: 2.52; 95% CI: 2.09–3.01). Delirium was not associated (P > .1) with death during hospitalization nor death 30 days following discharge.

As the delirious patients were much more likely to be discharged to a skilled nursing facility than nondelirious patients, we tested whether discharge disposition influenced readmission rates and ED visits between delirious and nondelirious patients in an unadjusted univariate analysis. The association between delirium and readmission and ED utilization was present regardless of disposition. Among patients discharged to skilled nursing, readmission rates were 4.76% and 13.38% (P < .001), and ED visit rates were 12.29% and 23.24% (P < .001) for nondelirious and delirious patients, respectively. Among patients discharged home, readmission rates were 4.96% and 14.37% (P < .001), and ED visit rates were 11.93% and 29.04% (P < .001) for nondelirious and delirious patients, respectively.

In this study of patients in a community hospital in Northern California, we observed a significant association between inpatient delirium and risk of hospital readmission within 30 days of discharge. We also demonstrated increased skilled nursing facility placement and ED utilization after discharge among hospitalized patients with delirium compared with those without. Patients with delirium in this study were diagnosed by a psychiatrist—a gold standard³⁰—and the study was conducted in a health system database with near comprehensive ascertainment of readmissions. These results suggest that patients with delirium are particularly vulnerable in the posthospitalization period and are a key group to focusing on reducing readmission rates and postdischarge healthcare utilization.

Identifying the risk factors for hospital readmission is important for the benefit of both the patient and the hospital. In an analysis of Medicare claims data from 2003 to 2004, 19.6% of beneficiaries were readmitted within 30 days of discharge.³¹ There is a national effort to reduce unplanned hospital readmissions for both patient safety as hospitals with high readmission rates face penalties from the Centers for Medicare and Medicaid Services.^22,23 Why delirium is associated with readmission remains unclear. Delirium may precipitate aspiration events, reduce oral intake which complicates medication administration and nutrition, or reduced mobility, leading to pulmonary emboli and skin breakdown, any of which could lead to readmission.³² Delirium may also accelerate the progression of cognitive decline and overall loss of functional independence.²⁰ Delirious patients can be difficult to care for at home, and persistent delirium may lead to returns to the ED and readmission. Strategies to reduce readmissions associated with delirium may need to focus on both prevention of hospital-acquired delirium and targeted caregiver and patient support after discharge.

Hospital readmission and ED visits are not mutually exclusive experiences. In the United States, the majority of patients admitted to the hospital are admitted through the ED.³³ Thus, most of the readmissions in this cohort were also likely counted as 30-day ED visits. However, as ED utilization occurs regardless of whether a patient is discharged or admitted from the ED, we reported all ED visits in this analysis, similar to other studies.³⁴ More delirium patients returned to the ED 30 days postdischarge than were ultimately readmitted to the hospital, and delirious patients were more likely to visit the ED or be readmitted than nondelirious patients. These observations point toward the first 30 days after discharge as a crucial period for these patients.

Our study features several strengths. To our knowledge, this study is one of the largest investigations of inpatients with delirium. One distinguishing feature was that all cases of delirium in this study were diagnosed by a psychiatrist, which is considered a gold standard. Many studies rely solely on brief nursing-administered surveys for delirium diagnosis. Using Kaiser Permanente data allowed for more complete follow-up of patients, including vital status. Kaiser Permanente is both a medical system and an insurer, resulting in acquisition of detailed health information from all hospitalizations where Kaiser Permanente insurance was used for each patient. Therefore, patients were only lost to follow-up following discharge in the event of a membership lapse; these patients were excluded from analysis. The obtained data are also more generalizable than those of other studies examining readmission rates in delirious patients as the hospital where these data were collected is a 116-bed general community medical and surgical hospital. Thus, the patients enrolled in this study covered multiple hospital services with a variety of admission diagnoses. This condition contrasts with much of the existing literature on inpatient delirium; these studies mostly center on specific medical conditions or surgeries and are often conducted at academic medical centers. At the same time, Kaiser Permanente is a unique health maintenance organization focused on preventive care, and readmission rates are possibly lower than elsewhere given the universal access to primary care for Kaiser Permanente members. Our results may not generalize to patients hospitalized in other health systems.

The diagnosis of delirium is a clinical diagnosis without biomarkers or radiographic markers and is also underdiagnosed and poorly coded.³² For these reasons, delirium can be challenging to study in large administrative databases or data derived from electronic medical records. We addressed this limitation by classifying the delirium patients only when they had been diagnosed by a staff psychiatrist. However, not all patients who screened positive with the CAM were evaluated by the staff psychiatrist during the study period. Thus, several CAM-positive patients who were not evaluated by psychiatry were included in the control population. This situation may cause bias toward identification of more severe cases of delirium. Although the physicians were encouraged to consult the psychiatry department for any patients who screened positive for delirium with the CAM, the psychiatrist may not have been involved if patients were managed without consultation. These patients may have exhibited less severe delirium or hypoactive delirium. In addition, the CAM fails to detect all delirious patients; interrater variability may occur with CAM administration, and non-English speaking patients are more likely to be excluded.³⁵ These situations are another possible way for our control population to include some delirious patients and those patients with less severe or hypoactive subtypes. While this might bias toward the null hypothesis, it is also possible our results only indicate an association between more clinically apparent delirium and readmission. A major limitation of this study is that we were unable to quantify the number of cohort patients screened with the CAM or the results of screening, thus limiting our ability to quantify the impact of potential biases introduced by the screening program.

This study may have underestimated readmission rates. We defined readmissions as all hospitalizations at any Kaiser Permanente facility, or to an alternate facility where Kaiser Permanente insurance was used, within 30 days of discharge. We excluded the day of discharge or the following calendar day to avoid mischaracterizing transfers from the index hospital to another Kaiser Permanente facility as readmissions. This step was conducted to avoid biasing our comparison, as delirious patients are less frequently discharged home than nondelirious patients. Therefore, while the relative odds of readmission between delirious and nondelirious patients reported in this study should be generalizable to other community hospitals, the absolute readmission rates reported here may not be comparable to those reported in other studies.

Delirium may represent a marker of more severe illness or medical complications accrued during the hospitalization, which could lead to the associations observed in this study due to confounding.³² Patients with delirium are more likely to be admitted emergently, admitted to the ICU, and feature higher acuity conditions than patients without delirium. We attempted to mitigate this possibility by using a multivariable model to control for variables related to illness severity, including the Charlson comorbidity index, HCUP diagnostic categories, and ICU admission. Despite including HCUP diagnostic categories in our model, we were unable to capture the contribution of certain diseases with finer granularity, such as preexistent dementia, which may also affect clinical outcomes.³⁶ Similarly, although we incorporated markers of illness severity into our model, we were unable to adjust for baseline functional status or frailty, which were not reliably recorded in the electronic medical record but are potential confounders when investigating clinical outcomes including hospital readmission.

We also lacked information regarding the duration of delirium in our cohort. Therefore, we were unable to test whether longer episodes of delirium were more predictive of readmission than shorter episodes.

In-hospital delirium is associated with several negative patient outcomes. Our study demonstrates that delirium predicts 30-day readmission and emergency department utilization after hospital discharge. Bearing in mind that a third of hospital-acquired delirium cases may be preventable,³² hospitals should prioritize interventions to reduce postdischarge healthcare utilization and complications in this particularly vulnerable group.

Acknowledgments

The authors would like to acknowledge Dr. Andrew L. Avins for his guidance with the initial development of this project and Julie Fourie for contributing data to the overall study.

Disclosures

Dr. Liu receives funding from NIH K23GM112018 and NIGMS R35128672. Dr. Josephson receives compensation as the JAMA Neurology Editor in Chief and Continuum Audio Associate Editor. The remaining authors have no conflicts of interest.

Funding

This study was funded by Kaiser Permanente Graduate Medical Education, who approved the design, conduct, and reporting of this study.

Almost all specialties in internal medicine have a sound scientific research base through which clinical practice is informed.¹ For the field of Hospital Medicine (HM), this evidence has largely comprised research generated from fields outside of the specialty. The need to develop, invest, and grow investigators in hospital-based medicine remains unmet as HM and its footprint in hospital systems continue to grow.^2,3

Despite this fact, little is known about the current state of research in HM. A 2014 survey of the members of the Society of Hospital Medicine (SHM) found that research output across the field of HM, as measured on the basis of peer-reviewed publications, was growing.⁴ Since then, however, the numbers of individuals engaged in research activities, their background and training, publication output, or funding sources have not been quantified. Similarly, little is known about which institutions support the development of junior investigators (ie, HM research fellowships), how these programs are funded, and whether or not matriculants enter the field as investigators. These gaps must be measured, evaluated, and ideally addressed through strategic policy and funding initiatives to advance the state of science within HM.

Members of the SHM Research Committee developed, designed, and deployed a survey to improve the understanding of the state of research in HM. In this study, we aimed to establish the baseline of research in HM to enable the measurement of progress through periodic waves of data collection. Specifically, we sought to quantify and describe the characteristics of existing research programs, the sources and types of funding, the number and background of faculty, and the availability of resources for training researchers in HM.

Study Setting and Participants

Given that no defined list, database, or external resource that identifies research programs and contacts in HM exists, we began by creating a strategy to identify and sample adult HM programs and their leaders engaged in research activity. We iteratively developed a two-step approach to maximize inclusivity. First, we partnered with SHM to identify programs and leaders actively engaging in research activities. SHM is the largest professional organization within HM and maintains an extensive membership database that includes the titles, e-mail addresses, and affiliations of hospitalists in the United States, including academic and nonacademic sites. This list was manually scanned, and the leaders of academic and research programs in adult HM were identified by examining their titles (eg, Division Chief, Research Lead, etc.) and academic affiliations. During this step, members of the committee noticed that certain key individuals were either missing, no longer occupying their role/title, or had been replaced by others. Therefore, we performed a second step and asked the members of the SHM Research Committee to identify academic and research leaders by using current personal contacts, publication history, and social networks. We asked members to identify individuals and programs that had received grant funding, were actively presenting research at SHM (or other major national venues), and/or were producing peer-reviewed publications related to HM. These programs were purposefully chosen (ie, over HM programs known for clinical activities) to create an enriched sample of those engaged in research in HM. The research committee performed the “second pass” to ensure that established investigators who may not be accurately captured within the SHM database were included to maximize yield for the survey. Finally, these two sources were merged to ensure the absence of duplicate contacts and the identification of a primary respondent for each affiliate. As a result, a convenience sample of 100 programs and corresponding individuals was compiled for the purposes of this survey.

Survey Development

A workgroup within the SHM Research Committee was tasked to create a survey that would achieve four distinct goals: (1) identify institutions currently engaging in hospital-based research; (2) define the characteristics, including sources of research funding, training opportunities, criteria for promotion, and grant support, of research programs within institutions; (3) understand the prevalence of research fellowship programs, including size, training curricula, and funding sources; and (4) evaluate the productivity and funding sources of HM investigators at each site.

Survey questions that target each of these domains were drafted by the workgroup. Questions were pretested with colleagues outside the workgroup focused on this project (ie, from the main research committee). The instrument was refined and edited to improve the readability and clarity of questions on the basis of the feedback obtained through the iterative process. The revised instrument was then programmed into an online survey administration tool (SurveyMonkey^®) to facilitate electronic dissemination. Finally, the members of the workgroup tested the online survey to ensure functionality. No identifiable information was collected from respondents, and no monetary incentive was offered for the completion of the survey. An invitation to participate in the survey was sent via e-mail to each of the program contacts identified.

Statistical Analysis

Descriptive statistics, including proportions, means, and percentages, were used to tabulate results. All analyses were conducted using Stata 13 MP/SE (StataCorp, College Station, Texas).

Ethical and Regulatory Considerations

The study was reviewed and deemed exempt from regulation by the University of Michigan Institutional Review Board (HUM000138628).

General Characteristics of Research Programs and Faculty

Out of 100 program contacts, 28 (representing 1,586 faculty members) responded and were included in the survey (program response rate = 28%). When comparing programs that did respond with those that did not, a greater proportion of programs in university settings were noted among respondents (79% vs 21%). Respondents represented programs from all regions of the United States, with most representing university-based (79%), university-affiliated (14%) or Veterans Health Administration (VHA; 11%) programs. Most respondents were in leadership roles, including division chiefs (32%), research directors/leads (21%), section chiefs (18%), and related titles, such as program director. Respondents indicated that the total number of faculty members in their programs (including nonclinicians and advance practice providers) varied from eight to 152 (mean [SD] = 57 [36]) members, with physicians representing the majority of faculty members (Table 1).

Among the 1,586 faculty members within the 28 programs, respondents identified 192 faculty members (12%) as currently receiving extra- or intramural support for research activities. Of these faculty, over half (58%) received <25% of effort from intra or extramural sources, and 28 (15%) and 52 (27%) faculty members received 25%-50% or >50% of support for their effort, respectively. The number of investigators who received funding across programs ranged from 0 to 28 faculty members. Compared with the 192 funded investigators, respondents indicated that a larger number of faculty in their programs (n = 656 or 41%) were involved in local quality improvement (QI) efforts. Of the 656 faculty members involved in QI efforts, 241 individuals (37%) were internally funded and received protected time/effort for their work.

Key Attributes of Research Programs

In the evaluation of the amount of total grant funding, respondents from 17 programs indicated that they received $500,000 in annual extra and intramural funding, and those from three programs stated that they received $500,000 to $999,999 in funding. Five respondents indicated that their programs currently received $1 million to $5 million in grant funding, and three reported >$5 million in research support. The sources of research funding included several divisions within the National Institute of Health (NIH, 12 programs), Agency for Healthcare Research and Quality (AHRQ, four programs), foundations (four programs), and internal grants (six programs). Additionally, six programs indicated “other” sources of funding that included the VHA, Patient-Centered Outcomes Research Institute (PCORI), Centers for Medicare and Medicaid Services, Centers for Disease Control (CDC), and industry sources.

A range of grants, including career development awards (11 programs); small grants, such as R21 and R03s (eight programs); R-level grants, including VA merit awards (five programs); program series grants, such as P and U grants (five programs), and foundation grants (eight programs), were reported as types of awards. Respondents from 16 programs indicated that they provided internal pilot grants. Amounts for such grants ranged from <$50,000 (14 programs) to $50,000-$100,000 (two programs).

Research Fellowship Programs/Training Programs

Only five of the 28 surveyed programs indicated that they currently had a research training or fellowship program for developing hospitalist investigators. The age of these programs varied from <1 year to 10 years. Three of the five programs stated that they had two fellows per year, and two stated they had spots for one trainee annually. All respondents indicated that fellows received training on study design, research methods, quantitative (eg, large database and secondary analyses) and qualitative data analysis. In addition, two programs included training in systematic review and meta-analyses, and three included focused courses on healthcare policy. Four of the five programs included training in QI tools, such as LEAN and Six Sigma. Funding for four of the five fellowship programs came from internal sources (eg, department and CTSA). However, two programs added they received some support from extramural funding and philanthropy. Following training, respondents from programs indicated that the majority of their graduates (60%) went on to hybrid research/QI roles (50/50 research/clinical effort), whereas 40% obtained dedicated research investigator (80/20) positions (Table 2).

The 23 institutions without research training programs cited that the most important barrier for establishing such programs was lack of funding (12 programs) and the lack of a pipeline of hospitalists seeking such training (six programs). However, 15 programs indicated that opportunities for hospitalists to gain research training in the form of courses were available internally (eg, courses in the department or medical school) or externally (eg, School of Public Health). Seven programs indicated that they were planning to start a HM research fellowship within the next five years.

Research Faculty

Among the 28 respondents, 15 stated that they have faculty members who conduct research as their main professional activity (ie, >50% effort). The number of faculty members in each program in such roles varied from one to 10. Respondents indicated that faculty members in this category were most often midcareer assistant or associate professors with few full professors. All programs indicated that scholarship in the form of peer-reviewed publications was required for the promotion of faculty. Faculty members who performed research as their main activity had all received formal fellowship training and consequently had dual degrees (MD with MPH or MD, with MSc being the two most common combinations). With respect to clinical activities, most respondents indicated that research faculty spent 10% to 49% of their effort on clinical work. However, five respondents indicated that research faculty had <10% effort on clinical duties (Table 3).

Eleven respondents (39%) identified the main focus of faculty as health service research, where four (14%) identified their main focus as clinical trials. Regardless of funding status, all respondents stated that their faculty were interested in studying quality and process improvement efforts (eg, transitions or readmissions, n = 19), patient safety initiatives (eg, hospital-acquired complications, n = 17), and disease-specific areas (eg, thrombosis, n = 15).

In terms of research output, 12 respondents stated that their research/QI faculty collectively published 11-50 peer-reviewed papers during the academic year, and 10 programs indicated that their faculty published 0-10 papers per year. Only three programs reported that their faculty collectively published 50-99 peer-reviewed papers per year. With respect to abstract presentations at national conferences, 13 programs indicated that they presented 0-10 abstracts, and 12 indicated that they presented 11-50.

In this first survey quantifying research activities in HM, respondents from 28 programs shared important insights into research activities at their institutions. Although our sample size was small, substantial variation in the size, composition, and structure of research programs in HM among respondents was observed. For example, few respondents indicated the availability of training programs for research in HM at their institutions. Similarly, among faculty who focused mainly on research, variation in funding streams and effort protection was observed. A preponderance of midcareer faculty with a range of funding sources, including NIH, AHRQ, VHA, CMS, and CDC was reported. Collectively, these data not only provide a unique glimpse into the state of research in HM but also help establish a baseline of the status of the field at large.

Some findings of our study are intuitive given our sampling strategy and the types of programs that responded. For example, the fact that most respondents for research programs represented university-based or affiliated institutions is expected given the tripartite academic mission. However, even within our sample of highly motivated programs, some findings are surprising and merit further exploration. For example, the observation that some respondents identified HM investigators within their program with <25% in intra- or extramural funding was unexpected. On the other extreme, we were surprised to find that three programs reported >$5 million in research funding. Understanding whether specific factors, such as the availability of experienced mentors within and outside departments or assistance from support staff (eg, statisticians and project managers), are associated with success and funding within these programs are important questions to answer. By focusing on these issues, we will be well poised as a field to understand what works, what does not work, and why.

Likewise, the finding that few programs within our sample offer formal training in the form of fellowships to research investigators represents an improvement opportunity. A pipeline for growing investigators is critical for the specialty that is HM. Notably, this call is not new; rather, previous investigators have highlighted the importance of developing academically oriented hospitalists for the future of the field.⁵ The implementation of faculty scholarship development programs has improved the scholarly output, mentoring activities, and succession planning of academics within HM.^6,7 Conversely, lack of adequate mentorship and support for academic activities remains a challenge and as a factor associated with the failure to produce academic work.⁸ Without a cadre of investigators asking critical questions related to care delivery, the legitimacy of our field may be threatened.

While extrapolating to the field is difficult given the small number of our respondents, highlighting the progress that has been made is important. For example, while misalignment between funding and clinical and research mission persists, our survey found that several programs have been successful in securing extramural funding for their investigators. Additionally, internal funding for QI work appears to be increasing, with hospitalists receiving dedicated effort for much of this work. Innovation in how best to support and develop these types of efforts have also emerged. For example, the University of Michigan Specialist Hospitalist Allied Research Program offers dedicated effort and funding for hospitalists tackling projects germane to HM (eg, ordering of blood cultures for febrile inpatients) that overlap with subspecialists (eg, infectious diseases).⁹ Thus, hospitalists are linked with other specialties in the development of research agendas and academic products. Similarly, the launch of the HOMERUN network, a coalition of investigators who bridge health systems to study problems central to HM, has helped usher in a new era of research opportunities in the specialty.¹⁰ Fundamentally, the culture of HM has begun to place an emphasis on academic and scholarly productivity in addition to clinical prowess.^11-13 Increased support and funding for training programs geared toward innovation and research in HM is needed to continue this mission. The Society for General Internal Medicine, American College of Physicians, and SHM have important roles to play as the largest professional organizations for generalists in this respect. Support for research, QI, and investigators in HM remains an urgent and largely unmet need.

Our study has limitations. First, our response rate was low at 28% but is consistent with the response rates of other surveys of physician groups.¹⁴ Caution in making inferences to the field at large is necessary given the potential for selection and nonresponse bias. However, we expect that respondents are likely biased toward programs actively conducting research and engaged in QI, thus better reflecting the state of these activities in HM. Second, given that we did not ask for any identifying information, we have no way of establishing the accuracy of the data provided by respondents. However, we have no reason to believe that responses would be altered in a systematic fashion. Future studies that link our findings to publicly available data (eg, databases of active grants and funding) might be useful. Third, while our survey instrument was created and internally validated by hospitalist researchers, its lack of external validation could limit findings. Finally, our results vary on the basis of how respondents answered questions related to effort and time allocation given that these measures differ across programs.

In summary, the findings from this study highlight substantial variations in the number, training, and funding of research faculty across HM programs. Understanding the factors behind the success of some programs and the failures of others appears important in informing and growing the research in the field. Future studies that aim to expand survey participation, raise the awareness of the state of research in HM, and identify barriers and facilitators to academic success in HM are needed.

Disclosures

Dr. Chopra discloses grant funding from the Agency for Healthcare Research and Quality (AHRQ), VA Health Services and Research Department, and Centers for Disease Control. Dr. Jones discloses grant funding from AHRQ. All other authors disclose no conflicts of interest.

Almost all specialties in internal medicine have a sound scientific research base through which clinical practice is informed.¹ For the field of Hospital Medicine (HM), this evidence has largely comprised research generated from fields outside of the specialty. The need to develop, invest, and grow investigators in hospital-based medicine remains unmet as HM and its footprint in hospital systems continue to grow.^2,3

Despite this fact, little is known about the current state of research in HM. A 2014 survey of the members of the Society of Hospital Medicine (SHM) found that research output across the field of HM, as measured on the basis of peer-reviewed publications, was growing.⁴ Since then, however, the numbers of individuals engaged in research activities, their background and training, publication output, or funding sources have not been quantified. Similarly, little is known about which institutions support the development of junior investigators (ie, HM research fellowships), how these programs are funded, and whether or not matriculants enter the field as investigators. These gaps must be measured, evaluated, and ideally addressed through strategic policy and funding initiatives to advance the state of science within HM.

Members of the SHM Research Committee developed, designed, and deployed a survey to improve the understanding of the state of research in HM. In this study, we aimed to establish the baseline of research in HM to enable the measurement of progress through periodic waves of data collection. Specifically, we sought to quantify and describe the characteristics of existing research programs, the sources and types of funding, the number and background of faculty, and the availability of resources for training researchers in HM.

Study Setting and Participants

Given that no defined list, database, or external resource that identifies research programs and contacts in HM exists, we began by creating a strategy to identify and sample adult HM programs and their leaders engaged in research activity. We iteratively developed a two-step approach to maximize inclusivity. First, we partnered with SHM to identify programs and leaders actively engaging in research activities. SHM is the largest professional organization within HM and maintains an extensive membership database that includes the titles, e-mail addresses, and affiliations of hospitalists in the United States, including academic and nonacademic sites. This list was manually scanned, and the leaders of academic and research programs in adult HM were identified by examining their titles (eg, Division Chief, Research Lead, etc.) and academic affiliations. During this step, members of the committee noticed that certain key individuals were either missing, no longer occupying their role/title, or had been replaced by others. Therefore, we performed a second step and asked the members of the SHM Research Committee to identify academic and research leaders by using current personal contacts, publication history, and social networks. We asked members to identify individuals and programs that had received grant funding, were actively presenting research at SHM (or other major national venues), and/or were producing peer-reviewed publications related to HM. These programs were purposefully chosen (ie, over HM programs known for clinical activities) to create an enriched sample of those engaged in research in HM. The research committee performed the “second pass” to ensure that established investigators who may not be accurately captured within the SHM database were included to maximize yield for the survey. Finally, these two sources were merged to ensure the absence of duplicate contacts and the identification of a primary respondent for each affiliate. As a result, a convenience sample of 100 programs and corresponding individuals was compiled for the purposes of this survey.

Survey Development

A workgroup within the SHM Research Committee was tasked to create a survey that would achieve four distinct goals: (1) identify institutions currently engaging in hospital-based research; (2) define the characteristics, including sources of research funding, training opportunities, criteria for promotion, and grant support, of research programs within institutions; (3) understand the prevalence of research fellowship programs, including size, training curricula, and funding sources; and (4) evaluate the productivity and funding sources of HM investigators at each site.

Survey questions that target each of these domains were drafted by the workgroup. Questions were pretested with colleagues outside the workgroup focused on this project (ie, from the main research committee). The instrument was refined and edited to improve the readability and clarity of questions on the basis of the feedback obtained through the iterative process. The revised instrument was then programmed into an online survey administration tool (SurveyMonkey^®) to facilitate electronic dissemination. Finally, the members of the workgroup tested the online survey to ensure functionality. No identifiable information was collected from respondents, and no monetary incentive was offered for the completion of the survey. An invitation to participate in the survey was sent via e-mail to each of the program contacts identified.

Statistical Analysis

Descriptive statistics, including proportions, means, and percentages, were used to tabulate results. All analyses were conducted using Stata 13 MP/SE (StataCorp, College Station, Texas).

Ethical and Regulatory Considerations

The study was reviewed and deemed exempt from regulation by the University of Michigan Institutional Review Board (HUM000138628).

General Characteristics of Research Programs and Faculty

Out of 100 program contacts, 28 (representing 1,586 faculty members) responded and were included in the survey (program response rate = 28%). When comparing programs that did respond with those that did not, a greater proportion of programs in university settings were noted among respondents (79% vs 21%). Respondents represented programs from all regions of the United States, with most representing university-based (79%), university-affiliated (14%) or Veterans Health Administration (VHA; 11%) programs. Most respondents were in leadership roles, including division chiefs (32%), research directors/leads (21%), section chiefs (18%), and related titles, such as program director. Respondents indicated that the total number of faculty members in their programs (including nonclinicians and advance practice providers) varied from eight to 152 (mean [SD] = 57 [36]) members, with physicians representing the majority of faculty members (Table 1).

Among the 1,586 faculty members within the 28 programs, respondents identified 192 faculty members (12%) as currently receiving extra- or intramural support for research activities. Of these faculty, over half (58%) received <25% of effort from intra or extramural sources, and 28 (15%) and 52 (27%) faculty members received 25%-50% or >50% of support for their effort, respectively. The number of investigators who received funding across programs ranged from 0 to 28 faculty members. Compared with the 192 funded investigators, respondents indicated that a larger number of faculty in their programs (n = 656 or 41%) were involved in local quality improvement (QI) efforts. Of the 656 faculty members involved in QI efforts, 241 individuals (37%) were internally funded and received protected time/effort for their work.

Key Attributes of Research Programs

In the evaluation of the amount of total grant funding, respondents from 17 programs indicated that they received $500,000 in annual extra and intramural funding, and those from three programs stated that they received $500,000 to $999,999 in funding. Five respondents indicated that their programs currently received $1 million to $5 million in grant funding, and three reported >$5 million in research support. The sources of research funding included several divisions within the National Institute of Health (NIH, 12 programs), Agency for Healthcare Research and Quality (AHRQ, four programs), foundations (four programs), and internal grants (six programs). Additionally, six programs indicated “other” sources of funding that included the VHA, Patient-Centered Outcomes Research Institute (PCORI), Centers for Medicare and Medicaid Services, Centers for Disease Control (CDC), and industry sources.

A range of grants, including career development awards (11 programs); small grants, such as R21 and R03s (eight programs); R-level grants, including VA merit awards (five programs); program series grants, such as P and U grants (five programs), and foundation grants (eight programs), were reported as types of awards. Respondents from 16 programs indicated that they provided internal pilot grants. Amounts for such grants ranged from <$50,000 (14 programs) to $50,000-$100,000 (two programs).

Research Fellowship Programs/Training Programs

Only five of the 28 surveyed programs indicated that they currently had a research training or fellowship program for developing hospitalist investigators. The age of these programs varied from <1 year to 10 years. Three of the five programs stated that they had two fellows per year, and two stated they had spots for one trainee annually. All respondents indicated that fellows received training on study design, research methods, quantitative (eg, large database and secondary analyses) and qualitative data analysis. In addition, two programs included training in systematic review and meta-analyses, and three included focused courses on healthcare policy. Four of the five programs included training in QI tools, such as LEAN and Six Sigma. Funding for four of the five fellowship programs came from internal sources (eg, department and CTSA). However, two programs added they received some support from extramural funding and philanthropy. Following training, respondents from programs indicated that the majority of their graduates (60%) went on to hybrid research/QI roles (50/50 research/clinical effort), whereas 40% obtained dedicated research investigator (80/20) positions (Table 2).

The 23 institutions without research training programs cited that the most important barrier for establishing such programs was lack of funding (12 programs) and the lack of a pipeline of hospitalists seeking such training (six programs). However, 15 programs indicated that opportunities for hospitalists to gain research training in the form of courses were available internally (eg, courses in the department or medical school) or externally (eg, School of Public Health). Seven programs indicated that they were planning to start a HM research fellowship within the next five years.

Research Faculty

Among the 28 respondents, 15 stated that they have faculty members who conduct research as their main professional activity (ie, >50% effort). The number of faculty members in each program in such roles varied from one to 10. Respondents indicated that faculty members in this category were most often midcareer assistant or associate professors with few full professors. All programs indicated that scholarship in the form of peer-reviewed publications was required for the promotion of faculty. Faculty members who performed research as their main activity had all received formal fellowship training and consequently had dual degrees (MD with MPH or MD, with MSc being the two most common combinations). With respect to clinical activities, most respondents indicated that research faculty spent 10% to 49% of their effort on clinical work. However, five respondents indicated that research faculty had <10% effort on clinical duties (Table 3).

Eleven respondents (39%) identified the main focus of faculty as health service research, where four (14%) identified their main focus as clinical trials. Regardless of funding status, all respondents stated that their faculty were interested in studying quality and process improvement efforts (eg, transitions or readmissions, n = 19), patient safety initiatives (eg, hospital-acquired complications, n = 17), and disease-specific areas (eg, thrombosis, n = 15).

In terms of research output, 12 respondents stated that their research/QI faculty collectively published 11-50 peer-reviewed papers during the academic year, and 10 programs indicated that their faculty published 0-10 papers per year. Only three programs reported that their faculty collectively published 50-99 peer-reviewed papers per year. With respect to abstract presentations at national conferences, 13 programs indicated that they presented 0-10 abstracts, and 12 indicated that they presented 11-50.

In this first survey quantifying research activities in HM, respondents from 28 programs shared important insights into research activities at their institutions. Although our sample size was small, substantial variation in the size, composition, and structure of research programs in HM among respondents was observed. For example, few respondents indicated the availability of training programs for research in HM at their institutions. Similarly, among faculty who focused mainly on research, variation in funding streams and effort protection was observed. A preponderance of midcareer faculty with a range of funding sources, including NIH, AHRQ, VHA, CMS, and CDC was reported. Collectively, these data not only provide a unique glimpse into the state of research in HM but also help establish a baseline of the status of the field at large.

Some findings of our study are intuitive given our sampling strategy and the types of programs that responded. For example, the fact that most respondents for research programs represented university-based or affiliated institutions is expected given the tripartite academic mission. However, even within our sample of highly motivated programs, some findings are surprising and merit further exploration. For example, the observation that some respondents identified HM investigators within their program with <25% in intra- or extramural funding was unexpected. On the other extreme, we were surprised to find that three programs reported >$5 million in research funding. Understanding whether specific factors, such as the availability of experienced mentors within and outside departments or assistance from support staff (eg, statisticians and project managers), are associated with success and funding within these programs are important questions to answer. By focusing on these issues, we will be well poised as a field to understand what works, what does not work, and why.

Likewise, the finding that few programs within our sample offer formal training in the form of fellowships to research investigators represents an improvement opportunity. A pipeline for growing investigators is critical for the specialty that is HM. Notably, this call is not new; rather, previous investigators have highlighted the importance of developing academically oriented hospitalists for the future of the field.⁵ The implementation of faculty scholarship development programs has improved the scholarly output, mentoring activities, and succession planning of academics within HM.^6,7 Conversely, lack of adequate mentorship and support for academic activities remains a challenge and as a factor associated with the failure to produce academic work.⁸ Without a cadre of investigators asking critical questions related to care delivery, the legitimacy of our field may be threatened.

While extrapolating to the field is difficult given the small number of our respondents, highlighting the progress that has been made is important. For example, while misalignment between funding and clinical and research mission persists, our survey found that several programs have been successful in securing extramural funding for their investigators. Additionally, internal funding for QI work appears to be increasing, with hospitalists receiving dedicated effort for much of this work. Innovation in how best to support and develop these types of efforts have also emerged. For example, the University of Michigan Specialist Hospitalist Allied Research Program offers dedicated effort and funding for hospitalists tackling projects germane to HM (eg, ordering of blood cultures for febrile inpatients) that overlap with subspecialists (eg, infectious diseases).⁹ Thus, hospitalists are linked with other specialties in the development of research agendas and academic products. Similarly, the launch of the HOMERUN network, a coalition of investigators who bridge health systems to study problems central to HM, has helped usher in a new era of research opportunities in the specialty.¹⁰ Fundamentally, the culture of HM has begun to place an emphasis on academic and scholarly productivity in addition to clinical prowess.^11-13 Increased support and funding for training programs geared toward innovation and research in HM is needed to continue this mission. The Society for General Internal Medicine, American College of Physicians, and SHM have important roles to play as the largest professional organizations for generalists in this respect. Support for research, QI, and investigators in HM remains an urgent and largely unmet need.

Our study has limitations. First, our response rate was low at 28% but is consistent with the response rates of other surveys of physician groups.¹⁴ Caution in making inferences to the field at large is necessary given the potential for selection and nonresponse bias. However, we expect that respondents are likely biased toward programs actively conducting research and engaged in QI, thus better reflecting the state of these activities in HM. Second, given that we did not ask for any identifying information, we have no way of establishing the accuracy of the data provided by respondents. However, we have no reason to believe that responses would be altered in a systematic fashion. Future studies that link our findings to publicly available data (eg, databases of active grants and funding) might be useful. Third, while our survey instrument was created and internally validated by hospitalist researchers, its lack of external validation could limit findings. Finally, our results vary on the basis of how respondents answered questions related to effort and time allocation given that these measures differ across programs.

In summary, the findings from this study highlight substantial variations in the number, training, and funding of research faculty across HM programs. Understanding the factors behind the success of some programs and the failures of others appears important in informing and growing the research in the field. Future studies that aim to expand survey participation, raise the awareness of the state of research in HM, and identify barriers and facilitators to academic success in HM are needed.

Disclosures

Dr. Chopra discloses grant funding from the Agency for Healthcare Research and Quality (AHRQ), VA Health Services and Research Department, and Centers for Disease Control. Dr. Jones discloses grant funding from AHRQ. All other authors disclose no conflicts of interest.

Almost all specialties in internal medicine have a sound scientific research base through which clinical practice is informed.¹ For the field of Hospital Medicine (HM), this evidence has largely comprised research generated from fields outside of the specialty. The need to develop, invest, and grow investigators in hospital-based medicine remains unmet as HM and its footprint in hospital systems continue to grow.^2,3

Despite this fact, little is known about the current state of research in HM. A 2014 survey of the members of the Society of Hospital Medicine (SHM) found that research output across the field of HM, as measured on the basis of peer-reviewed publications, was growing.⁴ Since then, however, the numbers of individuals engaged in research activities, their background and training, publication output, or funding sources have not been quantified. Similarly, little is known about which institutions support the development of junior investigators (ie, HM research fellowships), how these programs are funded, and whether or not matriculants enter the field as investigators. These gaps must be measured, evaluated, and ideally addressed through strategic policy and funding initiatives to advance the state of science within HM.

Members of the SHM Research Committee developed, designed, and deployed a survey to improve the understanding of the state of research in HM. In this study, we aimed to establish the baseline of research in HM to enable the measurement of progress through periodic waves of data collection. Specifically, we sought to quantify and describe the characteristics of existing research programs, the sources and types of funding, the number and background of faculty, and the availability of resources for training researchers in HM.

Study Setting and Participants

Given that no defined list, database, or external resource that identifies research programs and contacts in HM exists, we began by creating a strategy to identify and sample adult HM programs and their leaders engaged in research activity. We iteratively developed a two-step approach to maximize inclusivity. First, we partnered with SHM to identify programs and leaders actively engaging in research activities. SHM is the largest professional organization within HM and maintains an extensive membership database that includes the titles, e-mail addresses, and affiliations of hospitalists in the United States, including academic and nonacademic sites. This list was manually scanned, and the leaders of academic and research programs in adult HM were identified by examining their titles (eg, Division Chief, Research Lead, etc.) and academic affiliations. During this step, members of the committee noticed that certain key individuals were either missing, no longer occupying their role/title, or had been replaced by others. Therefore, we performed a second step and asked the members of the SHM Research Committee to identify academic and research leaders by using current personal contacts, publication history, and social networks. We asked members to identify individuals and programs that had received grant funding, were actively presenting research at SHM (or other major national venues), and/or were producing peer-reviewed publications related to HM. These programs were purposefully chosen (ie, over HM programs known for clinical activities) to create an enriched sample of those engaged in research in HM. The research committee performed the “second pass” to ensure that established investigators who may not be accurately captured within the SHM database were included to maximize yield for the survey. Finally, these two sources were merged to ensure the absence of duplicate contacts and the identification of a primary respondent for each affiliate. As a result, a convenience sample of 100 programs and corresponding individuals was compiled for the purposes of this survey.

Survey Development

A workgroup within the SHM Research Committee was tasked to create a survey that would achieve four distinct goals: (1) identify institutions currently engaging in hospital-based research; (2) define the characteristics, including sources of research funding, training opportunities, criteria for promotion, and grant support, of research programs within institutions; (3) understand the prevalence of research fellowship programs, including size, training curricula, and funding sources; and (4) evaluate the productivity and funding sources of HM investigators at each site.

Survey questions that target each of these domains were drafted by the workgroup. Questions were pretested with colleagues outside the workgroup focused on this project (ie, from the main research committee). The instrument was refined and edited to improve the readability and clarity of questions on the basis of the feedback obtained through the iterative process. The revised instrument was then programmed into an online survey administration tool (SurveyMonkey^®) to facilitate electronic dissemination. Finally, the members of the workgroup tested the online survey to ensure functionality. No identifiable information was collected from respondents, and no monetary incentive was offered for the completion of the survey. An invitation to participate in the survey was sent via e-mail to each of the program contacts identified.

Statistical Analysis

Descriptive statistics, including proportions, means, and percentages, were used to tabulate results. All analyses were conducted using Stata 13 MP/SE (StataCorp, College Station, Texas).

Ethical and Regulatory Considerations

The study was reviewed and deemed exempt from regulation by the University of Michigan Institutional Review Board (HUM000138628).

General Characteristics of Research Programs and Faculty

Out of 100 program contacts, 28 (representing 1,586 faculty members) responded and were included in the survey (program response rate = 28%). When comparing programs that did respond with those that did not, a greater proportion of programs in university settings were noted among respondents (79% vs 21%). Respondents represented programs from all regions of the United States, with most representing university-based (79%), university-affiliated (14%) or Veterans Health Administration (VHA; 11%) programs. Most respondents were in leadership roles, including division chiefs (32%), research directors/leads (21%), section chiefs (18%), and related titles, such as program director. Respondents indicated that the total number of faculty members in their programs (including nonclinicians and advance practice providers) varied from eight to 152 (mean [SD] = 57 [36]) members, with physicians representing the majority of faculty members (Table 1).

Among the 1,586 faculty members within the 28 programs, respondents identified 192 faculty members (12%) as currently receiving extra- or intramural support for research activities. Of these faculty, over half (58%) received <25% of effort from intra or extramural sources, and 28 (15%) and 52 (27%) faculty members received 25%-50% or >50% of support for their effort, respectively. The number of investigators who received funding across programs ranged from 0 to 28 faculty members. Compared with the 192 funded investigators, respondents indicated that a larger number of faculty in their programs (n = 656 or 41%) were involved in local quality improvement (QI) efforts. Of the 656 faculty members involved in QI efforts, 241 individuals (37%) were internally funded and received protected time/effort for their work.

Key Attributes of Research Programs

In the evaluation of the amount of total grant funding, respondents from 17 programs indicated that they received $500,000 in annual extra and intramural funding, and those from three programs stated that they received $500,000 to $999,999 in funding. Five respondents indicated that their programs currently received $1 million to $5 million in grant funding, and three reported >$5 million in research support. The sources of research funding included several divisions within the National Institute of Health (NIH, 12 programs), Agency for Healthcare Research and Quality (AHRQ, four programs), foundations (four programs), and internal grants (six programs). Additionally, six programs indicated “other” sources of funding that included the VHA, Patient-Centered Outcomes Research Institute (PCORI), Centers for Medicare and Medicaid Services, Centers for Disease Control (CDC), and industry sources.

A range of grants, including career development awards (11 programs); small grants, such as R21 and R03s (eight programs); R-level grants, including VA merit awards (five programs); program series grants, such as P and U grants (five programs), and foundation grants (eight programs), were reported as types of awards. Respondents from 16 programs indicated that they provided internal pilot grants. Amounts for such grants ranged from <$50,000 (14 programs) to $50,000-$100,000 (two programs).

Research Fellowship Programs/Training Programs

Only five of the 28 surveyed programs indicated that they currently had a research training or fellowship program for developing hospitalist investigators. The age of these programs varied from <1 year to 10 years. Three of the five programs stated that they had two fellows per year, and two stated they had spots for one trainee annually. All respondents indicated that fellows received training on study design, research methods, quantitative (eg, large database and secondary analyses) and qualitative data analysis. In addition, two programs included training in systematic review and meta-analyses, and three included focused courses on healthcare policy. Four of the five programs included training in QI tools, such as LEAN and Six Sigma. Funding for four of the five fellowship programs came from internal sources (eg, department and CTSA). However, two programs added they received some support from extramural funding and philanthropy. Following training, respondents from programs indicated that the majority of their graduates (60%) went on to hybrid research/QI roles (50/50 research/clinical effort), whereas 40% obtained dedicated research investigator (80/20) positions (Table 2).

The 23 institutions without research training programs cited that the most important barrier for establishing such programs was lack of funding (12 programs) and the lack of a pipeline of hospitalists seeking such training (six programs). However, 15 programs indicated that opportunities for hospitalists to gain research training in the form of courses were available internally (eg, courses in the department or medical school) or externally (eg, School of Public Health). Seven programs indicated that they were planning to start a HM research fellowship within the next five years.

Research Faculty

Among the 28 respondents, 15 stated that they have faculty members who conduct research as their main professional activity (ie, >50% effort). The number of faculty members in each program in such roles varied from one to 10. Respondents indicated that faculty members in this category were most often midcareer assistant or associate professors with few full professors. All programs indicated that scholarship in the form of peer-reviewed publications was required for the promotion of faculty. Faculty members who performed research as their main activity had all received formal fellowship training and consequently had dual degrees (MD with MPH or MD, with MSc being the two most common combinations). With respect to clinical activities, most respondents indicated that research faculty spent 10% to 49% of their effort on clinical work. However, five respondents indicated that research faculty had <10% effort on clinical duties (Table 3).

Eleven respondents (39%) identified the main focus of faculty as health service research, where four (14%) identified their main focus as clinical trials. Regardless of funding status, all respondents stated that their faculty were interested in studying quality and process improvement efforts (eg, transitions or readmissions, n = 19), patient safety initiatives (eg, hospital-acquired complications, n = 17), and disease-specific areas (eg, thrombosis, n = 15).

In terms of research output, 12 respondents stated that their research/QI faculty collectively published 11-50 peer-reviewed papers during the academic year, and 10 programs indicated that their faculty published 0-10 papers per year. Only three programs reported that their faculty collectively published 50-99 peer-reviewed papers per year. With respect to abstract presentations at national conferences, 13 programs indicated that they presented 0-10 abstracts, and 12 indicated that they presented 11-50.

In this first survey quantifying research activities in HM, respondents from 28 programs shared important insights into research activities at their institutions. Although our sample size was small, substantial variation in the size, composition, and structure of research programs in HM among respondents was observed. For example, few respondents indicated the availability of training programs for research in HM at their institutions. Similarly, among faculty who focused mainly on research, variation in funding streams and effort protection was observed. A preponderance of midcareer faculty with a range of funding sources, including NIH, AHRQ, VHA, CMS, and CDC was reported. Collectively, these data not only provide a unique glimpse into the state of research in HM but also help establish a baseline of the status of the field at large.

Some findings of our study are intuitive given our sampling strategy and the types of programs that responded. For example, the fact that most respondents for research programs represented university-based or affiliated institutions is expected given the tripartite academic mission. However, even within our sample of highly motivated programs, some findings are surprising and merit further exploration. For example, the observation that some respondents identified HM investigators within their program with <25% in intra- or extramural funding was unexpected. On the other extreme, we were surprised to find that three programs reported >$5 million in research funding. Understanding whether specific factors, such as the availability of experienced mentors within and outside departments or assistance from support staff (eg, statisticians and project managers), are associated with success and funding within these programs are important questions to answer. By focusing on these issues, we will be well poised as a field to understand what works, what does not work, and why.

Likewise, the finding that few programs within our sample offer formal training in the form of fellowships to research investigators represents an improvement opportunity. A pipeline for growing investigators is critical for the specialty that is HM. Notably, this call is not new; rather, previous investigators have highlighted the importance of developing academically oriented hospitalists for the future of the field.⁵ The implementation of faculty scholarship development programs has improved the scholarly output, mentoring activities, and succession planning of academics within HM.^6,7 Conversely, lack of adequate mentorship and support for academic activities remains a challenge and as a factor associated with the failure to produce academic work.⁸ Without a cadre of investigators asking critical questions related to care delivery, the legitimacy of our field may be threatened.

While extrapolating to the field is difficult given the small number of our respondents, highlighting the progress that has been made is important. For example, while misalignment between funding and clinical and research mission persists, our survey found that several programs have been successful in securing extramural funding for their investigators. Additionally, internal funding for QI work appears to be increasing, with hospitalists receiving dedicated effort for much of this work. Innovation in how best to support and develop these types of efforts have also emerged. For example, the University of Michigan Specialist Hospitalist Allied Research Program offers dedicated effort and funding for hospitalists tackling projects germane to HM (eg, ordering of blood cultures for febrile inpatients) that overlap with subspecialists (eg, infectious diseases).⁹ Thus, hospitalists are linked with other specialties in the development of research agendas and academic products. Similarly, the launch of the HOMERUN network, a coalition of investigators who bridge health systems to study problems central to HM, has helped usher in a new era of research opportunities in the specialty.¹⁰ Fundamentally, the culture of HM has begun to place an emphasis on academic and scholarly productivity in addition to clinical prowess.^11-13 Increased support and funding for training programs geared toward innovation and research in HM is needed to continue this mission. The Society for General Internal Medicine, American College of Physicians, and SHM have important roles to play as the largest professional organizations for generalists in this respect. Support for research, QI, and investigators in HM remains an urgent and largely unmet need.

Our study has limitations. First, our response rate was low at 28% but is consistent with the response rates of other surveys of physician groups.¹⁴ Caution in making inferences to the field at large is necessary given the potential for selection and nonresponse bias. However, we expect that respondents are likely biased toward programs actively conducting research and engaged in QI, thus better reflecting the state of these activities in HM. Second, given that we did not ask for any identifying information, we have no way of establishing the accuracy of the data provided by respondents. However, we have no reason to believe that responses would be altered in a systematic fashion. Future studies that link our findings to publicly available data (eg, databases of active grants and funding) might be useful. Third, while our survey instrument was created and internally validated by hospitalist researchers, its lack of external validation could limit findings. Finally, our results vary on the basis of how respondents answered questions related to effort and time allocation given that these measures differ across programs.

In summary, the findings from this study highlight substantial variations in the number, training, and funding of research faculty across HM programs. Understanding the factors behind the success of some programs and the failures of others appears important in informing and growing the research in the field. Future studies that aim to expand survey participation, raise the awareness of the state of research in HM, and identify barriers and facilitators to academic success in HM are needed.

Disclosures

Dr. Chopra discloses grant funding from the Agency for Healthcare Research and Quality (AHRQ), VA Health Services and Research Department, and Centers for Disease Control. Dr. Jones discloses grant funding from AHRQ. All other authors disclose no conflicts of interest.

Acute respiratory illnesses (ARIs), including acute exacerbations of asthma, croup, pneumonia, and bronchiolitis, are among the most common illnesses in childhood.¹ Although most ARIs can be managed in the outpatient setting, hospitalization is common with respiratory illnesses accounting for >425,000 hospitalizations annually.¹ Pneumonia, asthma, and bronchiolitis each rank among the top five reasons for pediatric hospitalization in the United States.¹ Successful efforts to prevent or mitigate the severity of ARIs could have a major impact on child health.

Exposure to secondhand smoke (SHS) is a preventable risk factor for ARI in children, particularly when there is regular exposure in the home.² Chronic exposure to SHS impacts systemic inflammation by suppressing serum interferon-gamma,³ which can lead to increased susceptibility to viral and bacterial infections,⁴ and increasing Th2 (atopic) cytokine expression, which is associated with asthma.⁵ SHS exposure in children has also been linked to diminished lung function.⁶ As a result, SHS exposure is associated with increased ARI susceptibility and severity in children.^7-10

Much research has focused on the clinical impact of SHS exposure on respiratory health in children, but little is known about the impact on patient-reported outcomes, such as health-related quality of life (HRQOL). Patient-reported outcomes help provide a comprehensive evaluation of the effectiveness of healthcare delivery systems. These outcomes are increasingly used by health service researchers to better understand patient and caregiver perspectives.¹¹ Given the known associations between SHS exposure and ARI morbidity, we postulated that regular SHS exposure would also impact HRQOL in children. In this study, we assessed the relationship between SHS exposure and HRQOL within a large, multicenter, prospective cohort of children presenting to the emergency department (ED) and/or hospital with ARI.

Study Population

This study was nested within the Pediatric Respiratory Illness Measurement System (PRIMES) study, a prospective cohort study of children with ARI in the ED and inpatient settings at five tertiary care children’s hospitals within the Pediatric Research in Inpatient Settings Network in Colorado, Pennsylvania, Tennessee, Texas, and Washington. Eligible children were two weeks to 16 years of age hospitalized after presenting to the ED with a primary diagnosis of asthma, croup, bronchiolitis, or pneumonia between July 1, 2014 and June 30, 2016. Because of an anticipated low frequency of croup hospitalizations, we also included children presenting to the ED and then discharged to home with this diagnosis. Children were assigned to a PRIMES diagnosis group based on their final discharge diagnosis. If there was a discrepancy between admission and discharge diagnoses, the discharge diagnosis was used. If a child had more than one discharge diagnosis for a PRIMES condition (eg, acute asthma and pneumonia), we chose the PRIMES condition with the lowest total enrollments overall. If the final discharge diagnosis was not a PRIMES condition, the case was excluded from further analysis. Patients with immunodeficiency, cystic fibrosis, a history of prematurity <32 weeks, chronic neuromuscular disease, cardiovascular disease, pulmonary diseases (other than asthma), and moderate to severe developmental delay were also excluded. Children admitted to intensive care were eligible only if they were transferred to an acute care ward <72 hours following admission. A survey was administered at the time of enrollment that collected information on SHS exposure, HRQOL, healthcare utilization, and demographics. All study procedures were reviewed and approved by the institutional review boards at each of the participating hospitals.

To ascertain SHS exposure, we asked caregivers, “How many persons living in the child’s home smoke?” Responses were dichotomized into non-SHS exposed (0 smokers) and SHS exposed (≥1 smokers). Children with missing data on home SHS exposure were excluded.

Health-Related Quality of Life Outcomes

We estimated HRQOL using the Pediatric Quality of Life (PedsQL^TM) 4.0 Generic Core and Infant Scales. The PedsQL instruments are validated, population HRQOL measures that evaluate the physical, mental, emotional, and social functioning of children two to 18 years old based on self- or caregiver-proxy report.^12-15 These instruments have also shown responsiveness as well as construct and predictive validity in hospitalized children.¹¹ For this study, we focused on the PedsQL physical functioning subscale, which assesses for problems with physical activities (eg, sports activity or exercise, low energy, and hurts or aches) on a five-point Likert scale (never to almost always a problem). Scores range from 0 to 100 with higher scores indicating a better HRQOL. The reported minimal clinically important difference (MCID), defined as the smallest difference in which individuals would perceive a benefit or would necessitate a change in management, for this scale is 4.5 points.^16,17

Children >8 years old were invited to complete the self-report version of the PedsQL. For children <8 years old, and for older children who were unable to complete them, surveys were completed by a parent or legal guardian. Respondents were asked to assess perceptions of their (or their child’s) HRQOL during periods of baseline health (the child’s usual state of health in the month preceding the current illness) and during the acute illness (the child’s state of health at the time of admission) as SHS exposure may influence perceptions of general health and/or contribute to worse outcomes during periods of acute illness.

Covariates collected at the time of enrollment included sociodemographics (child age, gender, race/ethnicity, and caregiver education), and healthcare utilization (caregiver-reported patient visits to a healthcare provider in the preceding six months). Insurance status and level of medical complexity (using the Pediatric Medical Complexity Algorithm)¹⁸ were obtained using the Pediatric Hospital Information System database, an administrative database containing clinical and resource utilization data from >45 children’s hospitals in the United States including all of the PRIMES study hospitals.¹³

Analysis

Descriptive statistics included frequency (%) and mean (standard deviation). Bivariate comparisons according to SHS exposure status were analyzed using chi-squared tests for categorical variables and analysis of variance for continuous variables. Multivariable linear mixed regression models were used to examine associations between home SHS exposure and HRQOL for baseline health and during admission, overall and stratified by diagnosis. Covariates in each model included age, sex, race/ethnicity, caregiver education, and healthcare visits in the preceding six months. We also included a hospital random effect to account for clustering of patients within hospitals and used robust standard errors for inference.

In a secondary analysis to explore potential dose-response effects of SHS exposure, we examined associations between an ordinal exposure variable (0 smokers, 1 smoker, ≥2 smokers) and HRQOL for baseline health and during admission for the acute illness. Because of sample size limitations, diagnosis-specific analyses examining dose-response effects were not conducted.

Study Population

Of the 2,334 children enrolled in the PRIMES study, 25 (1%) respondents did not report on home SHS exposure and were excluded, yielding a final study population of 2,309 children, of whom 728 (32%) had reported home SHS exposure. The study population included 664 children with asthma (mean age seven years [3.5]; 38% with home SHS exposure), 740 with bronchiolitis (mean age 0.7 years [0.5]; 32% with home SHS exposure), 342 with croup (mean age 1.7 [1.1]; 25% with home SHS exposure), and 563 with pneumonia (mean age 4.4 [3.8]; 27% with home SHS exposure; Table 1). Compared with non-SHS-exposed children, those with home SHS exposure tend to be slightly older (3.9 vs 3.4 years, P = .01), more likely to be non-Hispanic Black (29% vs 19%, P < .001), to have a chronic condition (52% vs 41%, P < .001), to come from a household where caregiver(s) did not graduate from college (45% vs 29%, P < .001), and to have public insurance (73% vs 49%, P < .001).

Home SHS Exposure and Health-related Quality of Life

The overall mean HRQOL score for baseline health was 83 (15), with a range across diagnoses of 82 to 87. Compared with non-SHS-exposed children, children with home SHS exposure had a lower mean HRQOL score for baseline health (adjusted mean difference –3.04 [95% CI -4.34, –1.74]). In analyses stratified by diagnosis, baseline health scores were lower for SHS-exposed children for all four conditions, but differences were statistically significant only for bronchiolitis (adjusted mean difference –2.94 [–5.0, –0.89]) and pneumonia (adjusted mean value –4.13 [–6.82, –1.44]; Table 2); none of these differences met the MCID threshold.

The overall mean HRQOL score at the time of admission was 56 (23), with a range across diagnoses of 49 to 61, with lower scores noted among SHS-exposed children compared with non-SHS-exposed children (adjusted mean difference –2.16 [–4.22, –0.10]). Similar to scores representing baseline health, admission scores were lower across all four conditions for SHS-exposed children. Only children with croup, however, had significantly lower admission scores that also met the MCID threshold (adjusted mean difference –5.71 [–10.67, –0.75]; Table 2).

To assess for potential dose-response effects of SHS exposure on HRQOL, we stratified SHS-exposed children into those with one smoker in the home (n = 513) and those with ≥2 smokers in the home (n = 215). Compared with non-SHS-exposed children, both HRQOL scores (baseline health and admission) were lower for SHS-exposed children. Consistent with a dose-response association, scores were lowest for children with ≥2 smokers in the home, both at baseline health (adjusted mean difference –3.92 [–6.03, –1.81]) and on admission (adjusted mean difference –3.67 [–6.98, –0.36]; Table 3).

Within a multicenter cohort of 2,309 children hospitalized with ARI, we noted significantly lower HRQOL scores among children exposed to SHS in the home as compared with nonexposed children. Differences were greatest for children living with ≥2 smokers in the home. In analyses stratified by diagnosis, differences in baseline health HRQOL scores were greatest for children with bronchiolitis and pneumonia. Differences in acute illness scores were greatest for children with croup.¹⁶

Our study provides evidence for acute and chronic impacts of SHS on HRQOL in children hospitalized with ARI. Although several studies have linked SHS exposure to reduced HRQOL in adults,^19,20 few similar studies have been conducted in children. Nonetheless, a wealth of studies have documented the negative impact of SHS exposure on clinical outcomes among children with ARI.^8,10,21-23 Our findings that home SHS exposure was associated with reduced HRQOL among our cohort of children with ARI are therefore consistent with related findings in adults and children. The observation that the effects of SHS exposure on HRQOL were greatest among children living with ≥2 smokers provides further evidence of a potential causal link between regular SHS exposure and HRQOL.

Although the magnitude and significance of associations between SHS exposure and HRQOL varied for each of the four diagnoses for baseline health and the acute illness, it is important to note that the point estimates for the adjusted mean differences were uniformly lower for the SHS-exposed children in each subgroup. Even so, only acute illness scores for croup exceeded the MCID threshold.¹⁶ Croup is the only included condition of the upper airway and is characterized by laryngotracheal inflammation leading to the typical cough and, in moderate to severe cases, stridor. Given that chronic SHS exposure induces a proinflammatory state,³ it is possible that SHS-exposed children with croup had more severe illness compared with nonexposed children with croup resulting in lower HRQOL scores on admission. Further, perceived differences in illness severity and HRQOL may be more readily apparent in children with croup (eg, stridor at rest vs intermittent or no stridor) as compared with children with lower respiratory tract diseases.

Of the four included diagnoses, the link between SHS exposure and asthma outcomes has been most studied. Prior work has demonstrated more frequent and severe acute exacerbations, as well as worse long-term lung function among SHS-exposed children as compared with nonexposed children.^22-24 It was, therefore, surprising that our study failed to demonstrate associations between SHS exposure and HRQOL among children with asthma. Reasons for this finding are unclear. One hypothesis is that caregivers of SHS-exposed children with asthma may be more aware of the impacts of SHS exposure on respiratory health (through prior education) and, thus, more likely to modify their smoking behaviors, or for their children to be on daily asthma controller therapy. Alternatively, caregivers of children with asthma may be more likely to underreport home SHS exposure. Thirty-eight percent of children with asthma, however, were classified as SHS-exposed. This percentage was greater than the other three conditions studied (25%-32%), suggesting that differential bias in underreporting was minimal. Given that children with asthma were older, on average, than children with the other three conditions, it may also be that these children spent more time in smoke-free environments (eg, school).

Nearly one-third of children in our study were exposed to SHS in the home. This is similar to the prevalence of exposure in other studies conducted among hospitalized children^8,10,21,25 but higher than the national prevalence of home SHS exposure among children in the United States.²⁶ Thus, hospitalized children represent a particularly vulnerable population and an important target for interventions aiming to reduce exposure to SHS. Although longitudinal interventions are likely necessary to affect long-term success, hospitalization for ARI may serve as a powerful teachable moment to begin cessation efforts. Hospitalization also offers time beyond a typical primary care outpatient encounter to focus on cessation counseling and may be the only opportunity to engage in counseling activities for some families with limited time or access. Further, prior studies have demonstrated both the feasibility and the effectiveness of smoking cessation interventions in hospitalized children.^27-30 Unfortunately, however, SHS exposure is often not documented at the time of hospitalization, and many opportunities to intervene are missed.^25,31 Thus, there is a need for improved strategies to reliably identify and intervene on SHS-exposed children in the hospital setting.

These findings should be considered in the context of several limitations. The observational nature of our study raises the potential for confounding, specifically with regard to socioeconomic status, as this is associated with both SHS exposure and lower HRQOL. Our modeling approach attempted to control for several factors associated with socioeconomic status, including caregiver education and insurance coverage, but there is potential for residual confounding. No single question is sufficient to fully assess SHS exposure as the intensity of home SHS exposure likely varies widely, and some children may be exposed to SHS outside of the home environment.³² The home, however, is often the most likely source of regular SHS exposure,^33,34 especially among young children (our cohort’s mean age was 3.6 years). Misclassification of SHS exposure is also possible due to underreporting of smoking.^35,36 As a result, some children regularly exposed to SHS may have been misclassified as nonexposed, and the observed associations between SHS exposure and HRQOL may be underestimated. Confirming our study’s findings using objective assessments of SHS exposure, such as cotinine, are warranted. Given the young age of our cohort, the PedsQL surveys were completed by the parent or legal guardian only in >90% of the enrolled subjects, and caregiver perceptions may not accurately reflect the child’s perceptions. Prior work, however, has demonstrated the validity of parent-proxy reporting of the PedsQL, including correlation with child self-report.³⁷ In our study, correlation between child and caregiver reporting (when available) was also very good (r = 0.72, 95% CI 0.64, 0.77). It is also possible that the timing of the HRQOL assessments (on admission) may have biased perceptions of baseline HRQOL, although we anticipate any bias would likely be nondifferential between SHS-exposed and nonexposed children and across diagnoses.

Nearly one-third of children in our study were exposed to SHS exposure in the home, and SHS exposure was associated with lower HRQOL for baseline health and during acute illness, providing further evidence of the dangers of SHS. Much work is needed in order to eliminate the impact of SHS on child health and families of children hospitalized for respiratory illness should be considered a priority population for smoking cessation efforts.

Acknowledgment

The authors wish to acknowledge the efforts of PRIS-PRIMES study team. The authors also wish to thank the children and families who consented to be a part of the PRIMES study.

Disclosures

The authors have no conflicts of interest relevant to this article to disclose.

Funding

This study was supported by NIH-NHLBI 1R01HL121067 to RMS.

Acute respiratory illnesses (ARIs), including acute exacerbations of asthma, croup, pneumonia, and bronchiolitis, are among the most common illnesses in childhood.¹ Although most ARIs can be managed in the outpatient setting, hospitalization is common with respiratory illnesses accounting for >425,000 hospitalizations annually.¹ Pneumonia, asthma, and bronchiolitis each rank among the top five reasons for pediatric hospitalization in the United States.¹ Successful efforts to prevent or mitigate the severity of ARIs could have a major impact on child health.

Exposure to secondhand smoke (SHS) is a preventable risk factor for ARI in children, particularly when there is regular exposure in the home.² Chronic exposure to SHS impacts systemic inflammation by suppressing serum interferon-gamma,³ which can lead to increased susceptibility to viral and bacterial infections,⁴ and increasing Th2 (atopic) cytokine expression, which is associated with asthma.⁵ SHS exposure in children has also been linked to diminished lung function.⁶ As a result, SHS exposure is associated with increased ARI susceptibility and severity in children.^7-10

Much research has focused on the clinical impact of SHS exposure on respiratory health in children, but little is known about the impact on patient-reported outcomes, such as health-related quality of life (HRQOL). Patient-reported outcomes help provide a comprehensive evaluation of the effectiveness of healthcare delivery systems. These outcomes are increasingly used by health service researchers to better understand patient and caregiver perspectives.¹¹ Given the known associations between SHS exposure and ARI morbidity, we postulated that regular SHS exposure would also impact HRQOL in children. In this study, we assessed the relationship between SHS exposure and HRQOL within a large, multicenter, prospective cohort of children presenting to the emergency department (ED) and/or hospital with ARI.

Study Population

This study was nested within the Pediatric Respiratory Illness Measurement System (PRIMES) study, a prospective cohort study of children with ARI in the ED and inpatient settings at five tertiary care children’s hospitals within the Pediatric Research in Inpatient Settings Network in Colorado, Pennsylvania, Tennessee, Texas, and Washington. Eligible children were two weeks to 16 years of age hospitalized after presenting to the ED with a primary diagnosis of asthma, croup, bronchiolitis, or pneumonia between July 1, 2014 and June 30, 2016. Because of an anticipated low frequency of croup hospitalizations, we also included children presenting to the ED and then discharged to home with this diagnosis. Children were assigned to a PRIMES diagnosis group based on their final discharge diagnosis. If there was a discrepancy between admission and discharge diagnoses, the discharge diagnosis was used. If a child had more than one discharge diagnosis for a PRIMES condition (eg, acute asthma and pneumonia), we chose the PRIMES condition with the lowest total enrollments overall. If the final discharge diagnosis was not a PRIMES condition, the case was excluded from further analysis. Patients with immunodeficiency, cystic fibrosis, a history of prematurity <32 weeks, chronic neuromuscular disease, cardiovascular disease, pulmonary diseases (other than asthma), and moderate to severe developmental delay were also excluded. Children admitted to intensive care were eligible only if they were transferred to an acute care ward <72 hours following admission. A survey was administered at the time of enrollment that collected information on SHS exposure, HRQOL, healthcare utilization, and demographics. All study procedures were reviewed and approved by the institutional review boards at each of the participating hospitals.

To ascertain SHS exposure, we asked caregivers, “How many persons living in the child’s home smoke?” Responses were dichotomized into non-SHS exposed (0 smokers) and SHS exposed (≥1 smokers). Children with missing data on home SHS exposure were excluded.

Health-Related Quality of Life Outcomes

We estimated HRQOL using the Pediatric Quality of Life (PedsQL^TM) 4.0 Generic Core and Infant Scales. The PedsQL instruments are validated, population HRQOL measures that evaluate the physical, mental, emotional, and social functioning of children two to 18 years old based on self- or caregiver-proxy report.^12-15 These instruments have also shown responsiveness as well as construct and predictive validity in hospitalized children.¹¹ For this study, we focused on the PedsQL physical functioning subscale, which assesses for problems with physical activities (eg, sports activity or exercise, low energy, and hurts or aches) on a five-point Likert scale (never to almost always a problem). Scores range from 0 to 100 with higher scores indicating a better HRQOL. The reported minimal clinically important difference (MCID), defined as the smallest difference in which individuals would perceive a benefit or would necessitate a change in management, for this scale is 4.5 points.^16,17

Children >8 years old were invited to complete the self-report version of the PedsQL. For children <8 years old, and for older children who were unable to complete them, surveys were completed by a parent or legal guardian. Respondents were asked to assess perceptions of their (or their child’s) HRQOL during periods of baseline health (the child’s usual state of health in the month preceding the current illness) and during the acute illness (the child’s state of health at the time of admission) as SHS exposure may influence perceptions of general health and/or contribute to worse outcomes during periods of acute illness.

Covariates collected at the time of enrollment included sociodemographics (child age, gender, race/ethnicity, and caregiver education), and healthcare utilization (caregiver-reported patient visits to a healthcare provider in the preceding six months). Insurance status and level of medical complexity (using the Pediatric Medical Complexity Algorithm)¹⁸ were obtained using the Pediatric Hospital Information System database, an administrative database containing clinical and resource utilization data from >45 children’s hospitals in the United States including all of the PRIMES study hospitals.¹³

Analysis

Descriptive statistics included frequency (%) and mean (standard deviation). Bivariate comparisons according to SHS exposure status were analyzed using chi-squared tests for categorical variables and analysis of variance for continuous variables. Multivariable linear mixed regression models were used to examine associations between home SHS exposure and HRQOL for baseline health and during admission, overall and stratified by diagnosis. Covariates in each model included age, sex, race/ethnicity, caregiver education, and healthcare visits in the preceding six months. We also included a hospital random effect to account for clustering of patients within hospitals and used robust standard errors for inference.

In a secondary analysis to explore potential dose-response effects of SHS exposure, we examined associations between an ordinal exposure variable (0 smokers, 1 smoker, ≥2 smokers) and HRQOL for baseline health and during admission for the acute illness. Because of sample size limitations, diagnosis-specific analyses examining dose-response effects were not conducted.

Study Population

Of the 2,334 children enrolled in the PRIMES study, 25 (1%) respondents did not report on home SHS exposure and were excluded, yielding a final study population of 2,309 children, of whom 728 (32%) had reported home SHS exposure. The study population included 664 children with asthma (mean age seven years [3.5]; 38% with home SHS exposure), 740 with bronchiolitis (mean age 0.7 years [0.5]; 32% with home SHS exposure), 342 with croup (mean age 1.7 [1.1]; 25% with home SHS exposure), and 563 with pneumonia (mean age 4.4 [3.8]; 27% with home SHS exposure; Table 1). Compared with non-SHS-exposed children, those with home SHS exposure tend to be slightly older (3.9 vs 3.4 years, P = .01), more likely to be non-Hispanic Black (29% vs 19%, P < .001), to have a chronic condition (52% vs 41%, P < .001), to come from a household where caregiver(s) did not graduate from college (45% vs 29%, P < .001), and to have public insurance (73% vs 49%, P < .001).

Home SHS Exposure and Health-related Quality of Life

The overall mean HRQOL score for baseline health was 83 (15), with a range across diagnoses of 82 to 87. Compared with non-SHS-exposed children, children with home SHS exposure had a lower mean HRQOL score for baseline health (adjusted mean difference –3.04 [95% CI -4.34, –1.74]). In analyses stratified by diagnosis, baseline health scores were lower for SHS-exposed children for all four conditions, but differences were statistically significant only for bronchiolitis (adjusted mean difference –2.94 [–5.0, –0.89]) and pneumonia (adjusted mean value –4.13 [–6.82, –1.44]; Table 2); none of these differences met the MCID threshold.

The overall mean HRQOL score at the time of admission was 56 (23), with a range across diagnoses of 49 to 61, with lower scores noted among SHS-exposed children compared with non-SHS-exposed children (adjusted mean difference –2.16 [–4.22, –0.10]). Similar to scores representing baseline health, admission scores were lower across all four conditions for SHS-exposed children. Only children with croup, however, had significantly lower admission scores that also met the MCID threshold (adjusted mean difference –5.71 [–10.67, –0.75]; Table 2).

To assess for potential dose-response effects of SHS exposure on HRQOL, we stratified SHS-exposed children into those with one smoker in the home (n = 513) and those with ≥2 smokers in the home (n = 215). Compared with non-SHS-exposed children, both HRQOL scores (baseline health and admission) were lower for SHS-exposed children. Consistent with a dose-response association, scores were lowest for children with ≥2 smokers in the home, both at baseline health (adjusted mean difference –3.92 [–6.03, –1.81]) and on admission (adjusted mean difference –3.67 [–6.98, –0.36]; Table 3).

Within a multicenter cohort of 2,309 children hospitalized with ARI, we noted significantly lower HRQOL scores among children exposed to SHS in the home as compared with nonexposed children. Differences were greatest for children living with ≥2 smokers in the home. In analyses stratified by diagnosis, differences in baseline health HRQOL scores were greatest for children with bronchiolitis and pneumonia. Differences in acute illness scores were greatest for children with croup.¹⁶

Our study provides evidence for acute and chronic impacts of SHS on HRQOL in children hospitalized with ARI. Although several studies have linked SHS exposure to reduced HRQOL in adults,^19,20 few similar studies have been conducted in children. Nonetheless, a wealth of studies have documented the negative impact of SHS exposure on clinical outcomes among children with ARI.^8,10,21-23 Our findings that home SHS exposure was associated with reduced HRQOL among our cohort of children with ARI are therefore consistent with related findings in adults and children. The observation that the effects of SHS exposure on HRQOL were greatest among children living with ≥2 smokers provides further evidence of a potential causal link between regular SHS exposure and HRQOL.

Although the magnitude and significance of associations between SHS exposure and HRQOL varied for each of the four diagnoses for baseline health and the acute illness, it is important to note that the point estimates for the adjusted mean differences were uniformly lower for the SHS-exposed children in each subgroup. Even so, only acute illness scores for croup exceeded the MCID threshold.¹⁶ Croup is the only included condition of the upper airway and is characterized by laryngotracheal inflammation leading to the typical cough and, in moderate to severe cases, stridor. Given that chronic SHS exposure induces a proinflammatory state,³ it is possible that SHS-exposed children with croup had more severe illness compared with nonexposed children with croup resulting in lower HRQOL scores on admission. Further, perceived differences in illness severity and HRQOL may be more readily apparent in children with croup (eg, stridor at rest vs intermittent or no stridor) as compared with children with lower respiratory tract diseases.

Of the four included diagnoses, the link between SHS exposure and asthma outcomes has been most studied. Prior work has demonstrated more frequent and severe acute exacerbations, as well as worse long-term lung function among SHS-exposed children as compared with nonexposed children.^22-24 It was, therefore, surprising that our study failed to demonstrate associations between SHS exposure and HRQOL among children with asthma. Reasons for this finding are unclear. One hypothesis is that caregivers of SHS-exposed children with asthma may be more aware of the impacts of SHS exposure on respiratory health (through prior education) and, thus, more likely to modify their smoking behaviors, or for their children to be on daily asthma controller therapy. Alternatively, caregivers of children with asthma may be more likely to underreport home SHS exposure. Thirty-eight percent of children with asthma, however, were classified as SHS-exposed. This percentage was greater than the other three conditions studied (25%-32%), suggesting that differential bias in underreporting was minimal. Given that children with asthma were older, on average, than children with the other three conditions, it may also be that these children spent more time in smoke-free environments (eg, school).

Nearly one-third of children in our study were exposed to SHS in the home. This is similar to the prevalence of exposure in other studies conducted among hospitalized children^8,10,21,25 but higher than the national prevalence of home SHS exposure among children in the United States.²⁶ Thus, hospitalized children represent a particularly vulnerable population and an important target for interventions aiming to reduce exposure to SHS. Although longitudinal interventions are likely necessary to affect long-term success, hospitalization for ARI may serve as a powerful teachable moment to begin cessation efforts. Hospitalization also offers time beyond a typical primary care outpatient encounter to focus on cessation counseling and may be the only opportunity to engage in counseling activities for some families with limited time or access. Further, prior studies have demonstrated both the feasibility and the effectiveness of smoking cessation interventions in hospitalized children.^27-30 Unfortunately, however, SHS exposure is often not documented at the time of hospitalization, and many opportunities to intervene are missed.^25,31 Thus, there is a need for improved strategies to reliably identify and intervene on SHS-exposed children in the hospital setting.

These findings should be considered in the context of several limitations. The observational nature of our study raises the potential for confounding, specifically with regard to socioeconomic status, as this is associated with both SHS exposure and lower HRQOL. Our modeling approach attempted to control for several factors associated with socioeconomic status, including caregiver education and insurance coverage, but there is potential for residual confounding. No single question is sufficient to fully assess SHS exposure as the intensity of home SHS exposure likely varies widely, and some children may be exposed to SHS outside of the home environment.³² The home, however, is often the most likely source of regular SHS exposure,^33,34 especially among young children (our cohort’s mean age was 3.6 years). Misclassification of SHS exposure is also possible due to underreporting of smoking.^35,36 As a result, some children regularly exposed to SHS may have been misclassified as nonexposed, and the observed associations between SHS exposure and HRQOL may be underestimated. Confirming our study’s findings using objective assessments of SHS exposure, such as cotinine, are warranted. Given the young age of our cohort, the PedsQL surveys were completed by the parent or legal guardian only in >90% of the enrolled subjects, and caregiver perceptions may not accurately reflect the child’s perceptions. Prior work, however, has demonstrated the validity of parent-proxy reporting of the PedsQL, including correlation with child self-report.³⁷ In our study, correlation between child and caregiver reporting (when available) was also very good (r = 0.72, 95% CI 0.64, 0.77). It is also possible that the timing of the HRQOL assessments (on admission) may have biased perceptions of baseline HRQOL, although we anticipate any bias would likely be nondifferential between SHS-exposed and nonexposed children and across diagnoses.

Nearly one-third of children in our study were exposed to SHS exposure in the home, and SHS exposure was associated with lower HRQOL for baseline health and during acute illness, providing further evidence of the dangers of SHS. Much work is needed in order to eliminate the impact of SHS on child health and families of children hospitalized for respiratory illness should be considered a priority population for smoking cessation efforts.

Acknowledgment

The authors wish to acknowledge the efforts of PRIS-PRIMES study team. The authors also wish to thank the children and families who consented to be a part of the PRIMES study.

Disclosures

The authors have no conflicts of interest relevant to this article to disclose.

Funding

This study was supported by NIH-NHLBI 1R01HL121067 to RMS.

Acute respiratory illnesses (ARIs), including acute exacerbations of asthma, croup, pneumonia, and bronchiolitis, are among the most common illnesses in childhood.¹ Although most ARIs can be managed in the outpatient setting, hospitalization is common with respiratory illnesses accounting for >425,000 hospitalizations annually.¹ Pneumonia, asthma, and bronchiolitis each rank among the top five reasons for pediatric hospitalization in the United States.¹ Successful efforts to prevent or mitigate the severity of ARIs could have a major impact on child health.

Exposure to secondhand smoke (SHS) is a preventable risk factor for ARI in children, particularly when there is regular exposure in the home.² Chronic exposure to SHS impacts systemic inflammation by suppressing serum interferon-gamma,³ which can lead to increased susceptibility to viral and bacterial infections,⁴ and increasing Th2 (atopic) cytokine expression, which is associated with asthma.⁵ SHS exposure in children has also been linked to diminished lung function.⁶ As a result, SHS exposure is associated with increased ARI susceptibility and severity in children.^7-10

Much research has focused on the clinical impact of SHS exposure on respiratory health in children, but little is known about the impact on patient-reported outcomes, such as health-related quality of life (HRQOL). Patient-reported outcomes help provide a comprehensive evaluation of the effectiveness of healthcare delivery systems. These outcomes are increasingly used by health service researchers to better understand patient and caregiver perspectives.¹¹ Given the known associations between SHS exposure and ARI morbidity, we postulated that regular SHS exposure would also impact HRQOL in children. In this study, we assessed the relationship between SHS exposure and HRQOL within a large, multicenter, prospective cohort of children presenting to the emergency department (ED) and/or hospital with ARI.

Study Population

This study was nested within the Pediatric Respiratory Illness Measurement System (PRIMES) study, a prospective cohort study of children with ARI in the ED and inpatient settings at five tertiary care children’s hospitals within the Pediatric Research in Inpatient Settings Network in Colorado, Pennsylvania, Tennessee, Texas, and Washington. Eligible children were two weeks to 16 years of age hospitalized after presenting to the ED with a primary diagnosis of asthma, croup, bronchiolitis, or pneumonia between July 1, 2014 and June 30, 2016. Because of an anticipated low frequency of croup hospitalizations, we also included children presenting to the ED and then discharged to home with this diagnosis. Children were assigned to a PRIMES diagnosis group based on their final discharge diagnosis. If there was a discrepancy between admission and discharge diagnoses, the discharge diagnosis was used. If a child had more than one discharge diagnosis for a PRIMES condition (eg, acute asthma and pneumonia), we chose the PRIMES condition with the lowest total enrollments overall. If the final discharge diagnosis was not a PRIMES condition, the case was excluded from further analysis. Patients with immunodeficiency, cystic fibrosis, a history of prematurity <32 weeks, chronic neuromuscular disease, cardiovascular disease, pulmonary diseases (other than asthma), and moderate to severe developmental delay were also excluded. Children admitted to intensive care were eligible only if they were transferred to an acute care ward <72 hours following admission. A survey was administered at the time of enrollment that collected information on SHS exposure, HRQOL, healthcare utilization, and demographics. All study procedures were reviewed and approved by the institutional review boards at each of the participating hospitals.

To ascertain SHS exposure, we asked caregivers, “How many persons living in the child’s home smoke?” Responses were dichotomized into non-SHS exposed (0 smokers) and SHS exposed (≥1 smokers). Children with missing data on home SHS exposure were excluded.

Health-Related Quality of Life Outcomes

We estimated HRQOL using the Pediatric Quality of Life (PedsQL^TM) 4.0 Generic Core and Infant Scales. The PedsQL instruments are validated, population HRQOL measures that evaluate the physical, mental, emotional, and social functioning of children two to 18 years old based on self- or caregiver-proxy report.^12-15 These instruments have also shown responsiveness as well as construct and predictive validity in hospitalized children.¹¹ For this study, we focused on the PedsQL physical functioning subscale, which assesses for problems with physical activities (eg, sports activity or exercise, low energy, and hurts or aches) on a five-point Likert scale (never to almost always a problem). Scores range from 0 to 100 with higher scores indicating a better HRQOL. The reported minimal clinically important difference (MCID), defined as the smallest difference in which individuals would perceive a benefit or would necessitate a change in management, for this scale is 4.5 points.^16,17

Children >8 years old were invited to complete the self-report version of the PedsQL. For children <8 years old, and for older children who were unable to complete them, surveys were completed by a parent or legal guardian. Respondents were asked to assess perceptions of their (or their child’s) HRQOL during periods of baseline health (the child’s usual state of health in the month preceding the current illness) and during the acute illness (the child’s state of health at the time of admission) as SHS exposure may influence perceptions of general health and/or contribute to worse outcomes during periods of acute illness.

Covariates collected at the time of enrollment included sociodemographics (child age, gender, race/ethnicity, and caregiver education), and healthcare utilization (caregiver-reported patient visits to a healthcare provider in the preceding six months). Insurance status and level of medical complexity (using the Pediatric Medical Complexity Algorithm)¹⁸ were obtained using the Pediatric Hospital Information System database, an administrative database containing clinical and resource utilization data from >45 children’s hospitals in the United States including all of the PRIMES study hospitals.¹³

Analysis

Descriptive statistics included frequency (%) and mean (standard deviation). Bivariate comparisons according to SHS exposure status were analyzed using chi-squared tests for categorical variables and analysis of variance for continuous variables. Multivariable linear mixed regression models were used to examine associations between home SHS exposure and HRQOL for baseline health and during admission, overall and stratified by diagnosis. Covariates in each model included age, sex, race/ethnicity, caregiver education, and healthcare visits in the preceding six months. We also included a hospital random effect to account for clustering of patients within hospitals and used robust standard errors for inference.

In a secondary analysis to explore potential dose-response effects of SHS exposure, we examined associations between an ordinal exposure variable (0 smokers, 1 smoker, ≥2 smokers) and HRQOL for baseline health and during admission for the acute illness. Because of sample size limitations, diagnosis-specific analyses examining dose-response effects were not conducted.

Study Population

Of the 2,334 children enrolled in the PRIMES study, 25 (1%) respondents did not report on home SHS exposure and were excluded, yielding a final study population of 2,309 children, of whom 728 (32%) had reported home SHS exposure. The study population included 664 children with asthma (mean age seven years [3.5]; 38% with home SHS exposure), 740 with bronchiolitis (mean age 0.7 years [0.5]; 32% with home SHS exposure), 342 with croup (mean age 1.7 [1.1]; 25% with home SHS exposure), and 563 with pneumonia (mean age 4.4 [3.8]; 27% with home SHS exposure; Table 1). Compared with non-SHS-exposed children, those with home SHS exposure tend to be slightly older (3.9 vs 3.4 years, P = .01), more likely to be non-Hispanic Black (29% vs 19%, P < .001), to have a chronic condition (52% vs 41%, P < .001), to come from a household where caregiver(s) did not graduate from college (45% vs 29%, P < .001), and to have public insurance (73% vs 49%, P < .001).

Home SHS Exposure and Health-related Quality of Life

The overall mean HRQOL score for baseline health was 83 (15), with a range across diagnoses of 82 to 87. Compared with non-SHS-exposed children, children with home SHS exposure had a lower mean HRQOL score for baseline health (adjusted mean difference –3.04 [95% CI -4.34, –1.74]). In analyses stratified by diagnosis, baseline health scores were lower for SHS-exposed children for all four conditions, but differences were statistically significant only for bronchiolitis (adjusted mean difference –2.94 [–5.0, –0.89]) and pneumonia (adjusted mean value –4.13 [–6.82, –1.44]; Table 2); none of these differences met the MCID threshold.

The overall mean HRQOL score at the time of admission was 56 (23), with a range across diagnoses of 49 to 61, with lower scores noted among SHS-exposed children compared with non-SHS-exposed children (adjusted mean difference –2.16 [–4.22, –0.10]). Similar to scores representing baseline health, admission scores were lower across all four conditions for SHS-exposed children. Only children with croup, however, had significantly lower admission scores that also met the MCID threshold (adjusted mean difference –5.71 [–10.67, –0.75]; Table 2).

To assess for potential dose-response effects of SHS exposure on HRQOL, we stratified SHS-exposed children into those with one smoker in the home (n = 513) and those with ≥2 smokers in the home (n = 215). Compared with non-SHS-exposed children, both HRQOL scores (baseline health and admission) were lower for SHS-exposed children. Consistent with a dose-response association, scores were lowest for children with ≥2 smokers in the home, both at baseline health (adjusted mean difference –3.92 [–6.03, –1.81]) and on admission (adjusted mean difference –3.67 [–6.98, –0.36]; Table 3).

Within a multicenter cohort of 2,309 children hospitalized with ARI, we noted significantly lower HRQOL scores among children exposed to SHS in the home as compared with nonexposed children. Differences were greatest for children living with ≥2 smokers in the home. In analyses stratified by diagnosis, differences in baseline health HRQOL scores were greatest for children with bronchiolitis and pneumonia. Differences in acute illness scores were greatest for children with croup.¹⁶

Our study provides evidence for acute and chronic impacts of SHS on HRQOL in children hospitalized with ARI. Although several studies have linked SHS exposure to reduced HRQOL in adults,^19,20 few similar studies have been conducted in children. Nonetheless, a wealth of studies have documented the negative impact of SHS exposure on clinical outcomes among children with ARI.^8,10,21-23 Our findings that home SHS exposure was associated with reduced HRQOL among our cohort of children with ARI are therefore consistent with related findings in adults and children. The observation that the effects of SHS exposure on HRQOL were greatest among children living with ≥2 smokers provides further evidence of a potential causal link between regular SHS exposure and HRQOL.

Although the magnitude and significance of associations between SHS exposure and HRQOL varied for each of the four diagnoses for baseline health and the acute illness, it is important to note that the point estimates for the adjusted mean differences were uniformly lower for the SHS-exposed children in each subgroup. Even so, only acute illness scores for croup exceeded the MCID threshold.¹⁶ Croup is the only included condition of the upper airway and is characterized by laryngotracheal inflammation leading to the typical cough and, in moderate to severe cases, stridor. Given that chronic SHS exposure induces a proinflammatory state,³ it is possible that SHS-exposed children with croup had more severe illness compared with nonexposed children with croup resulting in lower HRQOL scores on admission. Further, perceived differences in illness severity and HRQOL may be more readily apparent in children with croup (eg, stridor at rest vs intermittent or no stridor) as compared with children with lower respiratory tract diseases.

Of the four included diagnoses, the link between SHS exposure and asthma outcomes has been most studied. Prior work has demonstrated more frequent and severe acute exacerbations, as well as worse long-term lung function among SHS-exposed children as compared with nonexposed children.^22-24 It was, therefore, surprising that our study failed to demonstrate associations between SHS exposure and HRQOL among children with asthma. Reasons for this finding are unclear. One hypothesis is that caregivers of SHS-exposed children with asthma may be more aware of the impacts of SHS exposure on respiratory health (through prior education) and, thus, more likely to modify their smoking behaviors, or for their children to be on daily asthma controller therapy. Alternatively, caregivers of children with asthma may be more likely to underreport home SHS exposure. Thirty-eight percent of children with asthma, however, were classified as SHS-exposed. This percentage was greater than the other three conditions studied (25%-32%), suggesting that differential bias in underreporting was minimal. Given that children with asthma were older, on average, than children with the other three conditions, it may also be that these children spent more time in smoke-free environments (eg, school).

Nearly one-third of children in our study were exposed to SHS in the home. This is similar to the prevalence of exposure in other studies conducted among hospitalized children^8,10,21,25 but higher than the national prevalence of home SHS exposure among children in the United States.²⁶ Thus, hospitalized children represent a particularly vulnerable population and an important target for interventions aiming to reduce exposure to SHS. Although longitudinal interventions are likely necessary to affect long-term success, hospitalization for ARI may serve as a powerful teachable moment to begin cessation efforts. Hospitalization also offers time beyond a typical primary care outpatient encounter to focus on cessation counseling and may be the only opportunity to engage in counseling activities for some families with limited time or access. Further, prior studies have demonstrated both the feasibility and the effectiveness of smoking cessation interventions in hospitalized children.^27-30 Unfortunately, however, SHS exposure is often not documented at the time of hospitalization, and many opportunities to intervene are missed.^25,31 Thus, there is a need for improved strategies to reliably identify and intervene on SHS-exposed children in the hospital setting.

These findings should be considered in the context of several limitations. The observational nature of our study raises the potential for confounding, specifically with regard to socioeconomic status, as this is associated with both SHS exposure and lower HRQOL. Our modeling approach attempted to control for several factors associated with socioeconomic status, including caregiver education and insurance coverage, but there is potential for residual confounding. No single question is sufficient to fully assess SHS exposure as the intensity of home SHS exposure likely varies widely, and some children may be exposed to SHS outside of the home environment.³² The home, however, is often the most likely source of regular SHS exposure,^33,34 especially among young children (our cohort’s mean age was 3.6 years). Misclassification of SHS exposure is also possible due to underreporting of smoking.^35,36 As a result, some children regularly exposed to SHS may have been misclassified as nonexposed, and the observed associations between SHS exposure and HRQOL may be underestimated. Confirming our study’s findings using objective assessments of SHS exposure, such as cotinine, are warranted. Given the young age of our cohort, the PedsQL surveys were completed by the parent or legal guardian only in >90% of the enrolled subjects, and caregiver perceptions may not accurately reflect the child’s perceptions. Prior work, however, has demonstrated the validity of parent-proxy reporting of the PedsQL, including correlation with child self-report.³⁷ In our study, correlation between child and caregiver reporting (when available) was also very good (r = 0.72, 95% CI 0.64, 0.77). It is also possible that the timing of the HRQOL assessments (on admission) may have biased perceptions of baseline HRQOL, although we anticipate any bias would likely be nondifferential between SHS-exposed and nonexposed children and across diagnoses.

Nearly one-third of children in our study were exposed to SHS exposure in the home, and SHS exposure was associated with lower HRQOL for baseline health and during acute illness, providing further evidence of the dangers of SHS. Much work is needed in order to eliminate the impact of SHS on child health and families of children hospitalized for respiratory illness should be considered a priority population for smoking cessation efforts.

Acknowledgment

The authors wish to acknowledge the efforts of PRIS-PRIMES study team. The authors also wish to thank the children and families who consented to be a part of the PRIMES study.

Disclosures

The authors have no conflicts of interest relevant to this article to disclose.

Funding

This study was supported by NIH-NHLBI 1R01HL121067 to RMS.