Original Research

Acute kidney injury (AKI) is increasingly common in the hospitalized patient^1,2 with recent adult and pediatric multinational studies reporting AKI rates of 57% and 27%, respectively.^3,4 The development of AKI is associated with significant adverse outcomes including an increased risk of mortality.^5-7 For those that survive, the history of AKI may contribute to a lifetime of impaired health with chronic kidney disease.^8,9 This is particularly concerning for pediatric patients as AKI may impact morbidity for many decades, influence available therapies for these morbidities, and ultimately contribute to a shortened lifespan.¹⁰

AKI in the hospitalized patient is no longer accepted as an unfortunate and unavoidable consequence of illness or the indicated therapy. Currently, there is strong interest in this hospital-acquired condition with global initiatives aimed at increased prevention and early detection and treatment of AKI.^11,12 To this objective, risk stratification tools or prediction models could assist clinicians in decision making. Numerous studies have tested AKI prediction models either in particular high-risk populations or based on associated comorbidities, biomarkers, and critical illness scores. These studies are predominantly in adult populations, and few have been externally validated.¹³ While associations between certain medications and AKI are well known, an AKI prediction model that is applicable to pediatric or adult populations and is based on medication exposure is difficult. However, there is a growing recognition of the potential to develop such a model using the electronic health record (EHR).¹⁴

In 2013, Seattle Children’s Hospital (SCH) implemented a nephrotoxin and AKI detection system to assist in clinical decision making within the EHR. This system instituted the automatic ordering of serum creatinines to screen for AKI when the provider ordered three or more medications that were suspected to be nephrotoxic. Other clinical factors such as the diagnoses or preexisting conditions were not considered in the decision-tool algorithm. This original algorithm (Algorithm 1) was later modified and the list of suspected nephrotoxins was expanded (Table 1) in order to align with a national pediatric AKI collaborative (Algorithm 2). However, it was unclear whether the algorithm modification would improve AKI detection.

The present study had two objectives. The first was to evaluate the impact of the modifications on the sensitivity and specificity of our system. The second objective, if either the sensitivity or specificity was determined to be suboptimal, was to develop an improved model for nephrotoxin-related AKI detection. Having either the sensitivity or the specificity under 50% would be equivalent to or worse than a random guess, which we would consider unacceptable.

METHODS

Context

SCH is a tertiary care academic teaching hospital affiliated with the University of Washington School of Medicine, Harborview Medical Center, and the Seattle Cancer Care Alliance. The hospital has 371 licensed beds and approximately 18 medical subspecialty services.

 

Study Population

This was a retrospective cohort study examining all patients ages 0-21 years admitted to SCH between December 1, 2013 and November 30, 2015. The detection system was modified to align with the national pediatric AKI collaborative, Nephrotoxic Injury Negated by Just-in-Time Action (NINJA) in November 2014. Both acute care and intensive care patients were included (data not separated by location). Patients who had end-stage kidney disease and were receiving dialysis and patients who were evaluated in the emergency department without being admitted or admitted as observation status were excluded from analysis. Patients were also excluded if they did not have a baseline serum creatinine as defined below.

Study Measures

AKI is defined at SCH using the Kidney Disease: Improving Global Outcomes Stage 1 criteria as a guideline. The diagnosis of AKI is based on an increase in the baseline serum creatinine by 0.3 mg/dL or an increase in the serum creatinine by >1.5 times the baseline assuming the incoming creatinine is 0.5 mg/dL or higher. For our definition, the increase in serum creatinine needs to have occurred within a one-week timeframe and urine output is not a diagnostic criterion.¹⁵ Baseline serum creatinine is defined as the lowest serum creatinine in the previous six months. Forty medications were classified as nephrotoxins based on previous analysis¹⁶ and adapted for our institutional formulary.

Statistical Analysis

To evaluate the efficacy of our systems in detecting nephrotoxin-related AKI, the sensitivity and the specificity using both our original algorithm (Algorithm 1) and the modified algorithm (Algorithm 2) were generated on our complete data set. To test sensitivity, the proportion of AKI patients who would trigger alert using Algorithm 1 and then with Algorithm 2 was identified. Similarly, to test specificity, the proportion of non-AKI patients who did not trigger an alert by the surveillance systems was identified. The differences in sensitivity and specificity between the two algorithms were evaluated using two-sample tests of proportion.

The statistical method of Combinatorial Inference has been utilized in studies of cancer biology¹⁷ and in genomics.¹⁸ A variation of this approach was used in this study to identify the specific medication combinations most associated with AKI. First, all of the nephrotoxic medications and medication combinations that were prescribed during our study period were identified from a data set (ie, a training set) containing 75% of all encounters selected at random without replacement. Using this training set, the prevalence of each medication combination and the rate of AKI associated with each combination were identified. The predicted overall AKI risk of an individual medication is the average of all the AKI rates associated with each combination containing that specific medication. Also incorporated into the determination of the predicted AKI risk was the prevalence of that medication combination.

To test our model’s predictive capability, the algorithm was applied to the remaining 25% of the total patient data (ie, the test set). The predicted AKI risk was compared with the actual AKI rate in the test data set. Our model’s predictive capability was represented in a receiver operator characteristic (ROC) analysis. The goal was to achieve an area under the ROC curve (AUC) approaching one as this would reflect 100% sensitivity and 100% specificity, whereas an AUC of 0.5 would represent a random guess (50% chance of being correct).

Lastly, our final step was to use our model’s ROC curve to determine an optimal threshold of AKI risk for which to trigger an alert. This predicted risk threshold was based on our goal to increase our surveillance system’s sensitivity balanced with maintaining an acceptable specificity.

An a priori threshold of P = .05 was used to determine statistical significance of all results. Analyses were conducted in Stata 12.1 (StataCorp LP, College Station, Texas) and R 3.3.2 (R Foundation for Statistical Computing, Vienna, Austria). A sample data set containing replication code for our model can be found in an online repository (https://dataverse.harvard.edu/dataverse/chuan). This study was approved by the Seattle Children’s Institutional Review Board.

 

RESULTS

Sensitivity and Specificity

Of the patient encounters, 14,779 were eligible during the study period. The sensitivity of the system’s ability to identify nephrotoxin-related AKI decreased from 46.9% using Algorithm 1 to 43.3% using Algorithm 2, a change of 3.6% (P = .22). The specificity increased from 73.6% to 89.3%, a change of 15.7% (P < .001; Table 2).

Improvement of Our Nephrotoxin-Related AKI Detection System Using a Novel AKI Prediction Strategy

A total of 838 medication combinations were identified in our training set and the predicted AKI risk for every medication combination was determined. By comparing the predicted risk of AKI to the actual AKI occurrence, an ROC curve with an AUC of 0.756 (Figure) was generated. An increase in system sensitivity was prioritized when determining the optimal AKI risk at which the model would trigger an alert. Setting an alert threshold at a predicted AKI risk of >8%, our model performed with a sensitivity of 74% while decreasing the specificity to 70%.

Identification of High-Risk Nephrotoxic Medications and Medication Combinations

Approximately 200 medication combinations were associated with >8% AKI risk, our new AKI prediction model’s alert threshold. Medication combinations consisting of up to 11 concomitantly prescribed medications were present in our data set. However, many of these combinations were infrequently prescribed. Further analysis, conducted in order to increase the clinical relevance of our findings, identified 10 medications or medication combinations that were both associated with a predicted AKI risk of >8% and that were prescribed on average greater than twice a month (Table 3).

DISCUSSION

The nephrotoxin-related AKI detection system at SCH automatically places orders for serum creatinines on patients who have met criteria for concomitant nephrotoxin exposure. This has given us a robust database from which to develop our clinical decision-making tool. Both our original and updated systems were based on the absolute number of concomitant nephrotoxic medications prescribed.¹⁶ This is a reasonable approach given the complexity of building a surveillance system¹⁹ and resource limitations. However, a system based on observed rather than theoretical or in vitro data, adaptable to the institution and designed for ongoing refinement, would be more valuable.

The interest in AKI prediction tools continues to be high. Bedford et al. employed numerous variables and diagnostic codes to predict the development of AKI in adults during hospitalization. They were able to produce a prediction model with a reasonable fit (AUC 0.72) to identify patients at higher risk for AKI but were less successful in their attempts to predict progression to severe AKI.²⁰ Hodgson et al. recently developed an adult AKI prediction score (AUC 0.65-0.72) also based on numerous clinical factors that was able to positively impact inpatient mortality.²¹ To our knowledge, our model is unique in that it focuses on nephrotoxins using a predicted AKI risk algorithm based on observed AKI rates of previously ordered medications/medication combinations (two to 11 medications). Having a decision tool targeting medications gives the clinician guidance that can be used to make a specific intervention rather than identifying a patient at risk due to a diagnosis code or other difficult to modify factors.

There are abundant case studies and reports using logistic regression models identifying specific medications associated with AKI. Our choice of methodology was based on our assessment that logistic regression models would be inadequate for the development of a real-time clinical decision-making tool for several reasons. Using logistic regression to explore every medication combination based on our medication list would be challenging as there are approximately 5.5 × 10¹⁰ potential medication combinations. Additionally, logistic regression ignores any potential interactions between the medications. This is an important point as medication interactions can be synergistic, neutral, or antagonist. Consequently, the outcome generated from a set of combined variables may be different from one generated from the sum of each variable taken independently. Logistic regression also does not account for the potential prescribing trends among providers as it assumes that all medications or medication combinations are equally available at the same time. However, in practice, depending on numerous factors, such as hospital culture (eg, the presence of clinical standard work pathways), local bacterial resistance patterns, or medication shortages; certain medication combinations may occur more frequently while others not at all. Finally, logistic regression cannot account for the possibility of a medication combination occurring; therefore, logistic regression may identify a combination strongly associated with AKI that is rarely prescribed.

We theorized that AKI detection would improve with the Algorithm 2 modifications, including the expanded nephrotoxin list, which accompanied alignment with the national pediatric AKI collaborative, NINJA. The finding that our surveillance sensitivity did not improve with this system update supported our subsequent objective to develop a novel nephrotoxin-related AKI decision tool or detection system using our EHR data to identify which specific medications and/or medication combinations were associated with a higher rate of AKI. However, it should be noted that two factors related to measurement bias introduce limitations to our sensitivity and specificity analyses. First, regarding the presence of the alert system, our system will order serum creatinines on patients when they have been exposed to nephrotoxins. Consequently, the proportion of patients with creatinines measured will increase in the nephrotoxin-exposed patients. Unexposed patients may have AKI that is not detected because creatinines may not be ordered. Therefore, there is the potential for a relative increase in AKI detection among nephrotoxin-exposed patients as compared with unexposed patients, which would then affect the measured sensitivity and specificity of the alert. Second, the automated alerts require a baseline creatinine in order to trigger therefore are unable to identify AKI among patients who do not have a baseline serum creatinine measurement.

Our new nephrotoxin-related AKI detection model performed best when an alert was triggered for those medications or medication combinations with a predicted AKI risk of >8%. Forty-six medication combinations consisting of exactly two medications were determined to have a predicted AKI risk of >8% therefore would trigger an alert in our new model system. These medication combinations would not have triggered an alert using either of the previous system algorithms as both algorithms are based on the presence of three or more concomitant nephrotoxic medications.

From the list of suspected nephrotoxins, we identified 11 unique medications in 10 different combinations with a predicted AKI risk of >8% that were prescribed frequently (at least twice a month on average; Table 3). Notably, six out of 10 medication combinations involved vancomycin. Piperacillin-tazobactam was also represented in several combinations. These findings support the concern that others have reported regarding these two medications particularly when prescribed together.^22,23

Interestingly, enalapril was identified as a higher-risk medication both alone and in combination with another medication. We do not suspect that enalapril carries a higher risk than other angiotensin-converting enzyme (ACE) inhibitors to increase a patient’s serum creatinine. Rather, we suspect that in our hospitalized patients, this relatively short-acting ACE inhibitor is commonly used in several of our vulnerable populations such as in cardiac and bone marrow transplant patients.

The alert threshold of our model can be adjusted to increase either the sensitivity or the specificity of AKI detection. Our detection sensitivity increased by >1.5-fold with the alert trigger threshold set at a predicted AKI risk of >8%. As a screening tool, our alert limits could be set such that our sensitivity would be greater; however, balancing the potential for alert fatigue is important in determining the acceptance and, ultimately, the success of a working surveillance system.²⁴

A patient’s overall risk of AKI is influenced by many factors such as the presence of underlying chronic comorbidities and the nature or severity of the acute illness as this may affect the patient’s intravascular volume status, systemic blood pressures, or drug metabolism. Our study is limited as we are a children’s hospital and our patients may have fewer comorbidities than seen in the adult population. One could argue that this permits a perspective not clouded by the confounders of chronic disease and allows for the effect of the medications prescribed to be more apparent. However, our study includes critically ill patients and patients who may have been hemodynamically unstable. This may explain why the NINJA algorithm did not improve the sensitivity of our AKI detection as the NINJA collaborative excludes critically ill patients.

Dose and dosing frequency of the prescribed medications could not be taken into account, which could explain the finding that nonsteroidal anti-inflammatory drugs (NSAIDs) such as aspirin, ibuprofen, or ketorolac when used alone were associated with a low (<1%) rate of AKI despite being frequently prescribed. Additionally, as many providers are aware of the AKI risk of NSAIDs, these medications may have been used intermittently (as needed) or in select, perhaps healthier, patients or in patients that take these medications chronically who were admitted for reasons that did not alter their outpatient medication regimen.

Our study also reflects the prescribing habits of our institution and may not be directly applicable to nontertiary care hospitals or centers that do not have large cystic fibrosis, bone marrow, or solid organ transplant populations. Despite our study’s limitations, we feel that there are several findings that are relevant across centers and populations. Our data were derived from the systematic ordering of daily serum creatinines when a patient is at risk for nephrotoxin-related AKI. This is in step with the philosophy advocated by others that AKI identification can only occur if the providers are aware of this risk and are vigilant.²⁵ In this vigilance, we also recognize that not all risks are of the same magnitude and may not deserve the same attention when resources are limited. Our identification of those medication combinations most associated with AKI at our institution has helped us narrow our focus and identify specific areas of potential education and intervention. The specific combinations identified may also be relevant to similar institutions serving similarly complex patients. Those with dissimilar populations could use this methodology to identify those medication combinations most relevant for their patient population and their prescriber’s habits. More studies of this type would be beneficial to the medical community as a whole as certain medication combinations may be found to be high risk regardless of the institution and the age or demographics of the populations they serve.

 

Acknowledgments

Dr. Karyn E. Yonekawa conceptualized and designed the study, directed the data analysis, interpreted the data, drafted, revised and gave final approval of the manuscript. Dr. Chuan Zhou contributed to the study design, acquired data, conducted the data analysis, critically reviewed, and gave final approval of the manuscript. Ms. Wren L. Haaland contributed to the study design, acquired data, conducted the data analysis, critically reviewed, and gave final approval of the manuscript. Dr. Davene R. Wright contributed to the study design, data analysis, critically reviewed, revised, and gave final approval of the manuscript.

The authors would like to thank Holly Clifton and Suzanne Spencer for their assistance with data acquisition and Drs. Derya Caglar, Corrie McDaniel, and Thida Ong for their writing support.

All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

Disclosures

The authors have no conflicts of interest to report.

Acute kidney injury (AKI) is increasingly common in the hospitalized patient^1,2 with recent adult and pediatric multinational studies reporting AKI rates of 57% and 27%, respectively.^3,4 The development of AKI is associated with significant adverse outcomes including an increased risk of mortality.^5-7 For those that survive, the history of AKI may contribute to a lifetime of impaired health with chronic kidney disease.^8,9 This is particularly concerning for pediatric patients as AKI may impact morbidity for many decades, influence available therapies for these morbidities, and ultimately contribute to a shortened lifespan.¹⁰

AKI in the hospitalized patient is no longer accepted as an unfortunate and unavoidable consequence of illness or the indicated therapy. Currently, there is strong interest in this hospital-acquired condition with global initiatives aimed at increased prevention and early detection and treatment of AKI.^11,12 To this objective, risk stratification tools or prediction models could assist clinicians in decision making. Numerous studies have tested AKI prediction models either in particular high-risk populations or based on associated comorbidities, biomarkers, and critical illness scores. These studies are predominantly in adult populations, and few have been externally validated.¹³ While associations between certain medications and AKI are well known, an AKI prediction model that is applicable to pediatric or adult populations and is based on medication exposure is difficult. However, there is a growing recognition of the potential to develop such a model using the electronic health record (EHR).¹⁴

In 2013, Seattle Children’s Hospital (SCH) implemented a nephrotoxin and AKI detection system to assist in clinical decision making within the EHR. This system instituted the automatic ordering of serum creatinines to screen for AKI when the provider ordered three or more medications that were suspected to be nephrotoxic. Other clinical factors such as the diagnoses or preexisting conditions were not considered in the decision-tool algorithm. This original algorithm (Algorithm 1) was later modified and the list of suspected nephrotoxins was expanded (Table 1) in order to align with a national pediatric AKI collaborative (Algorithm 2). However, it was unclear whether the algorithm modification would improve AKI detection.

The present study had two objectives. The first was to evaluate the impact of the modifications on the sensitivity and specificity of our system. The second objective, if either the sensitivity or specificity was determined to be suboptimal, was to develop an improved model for nephrotoxin-related AKI detection. Having either the sensitivity or the specificity under 50% would be equivalent to or worse than a random guess, which we would consider unacceptable.

METHODS

Context

SCH is a tertiary care academic teaching hospital affiliated with the University of Washington School of Medicine, Harborview Medical Center, and the Seattle Cancer Care Alliance. The hospital has 371 licensed beds and approximately 18 medical subspecialty services.

 

Study Population

This was a retrospective cohort study examining all patients ages 0-21 years admitted to SCH between December 1, 2013 and November 30, 2015. The detection system was modified to align with the national pediatric AKI collaborative, Nephrotoxic Injury Negated by Just-in-Time Action (NINJA) in November 2014. Both acute care and intensive care patients were included (data not separated by location). Patients who had end-stage kidney disease and were receiving dialysis and patients who were evaluated in the emergency department without being admitted or admitted as observation status were excluded from analysis. Patients were also excluded if they did not have a baseline serum creatinine as defined below.

Study Measures

AKI is defined at SCH using the Kidney Disease: Improving Global Outcomes Stage 1 criteria as a guideline. The diagnosis of AKI is based on an increase in the baseline serum creatinine by 0.3 mg/dL or an increase in the serum creatinine by >1.5 times the baseline assuming the incoming creatinine is 0.5 mg/dL or higher. For our definition, the increase in serum creatinine needs to have occurred within a one-week timeframe and urine output is not a diagnostic criterion.¹⁵ Baseline serum creatinine is defined as the lowest serum creatinine in the previous six months. Forty medications were classified as nephrotoxins based on previous analysis¹⁶ and adapted for our institutional formulary.

Statistical Analysis

To evaluate the efficacy of our systems in detecting nephrotoxin-related AKI, the sensitivity and the specificity using both our original algorithm (Algorithm 1) and the modified algorithm (Algorithm 2) were generated on our complete data set. To test sensitivity, the proportion of AKI patients who would trigger alert using Algorithm 1 and then with Algorithm 2 was identified. Similarly, to test specificity, the proportion of non-AKI patients who did not trigger an alert by the surveillance systems was identified. The differences in sensitivity and specificity between the two algorithms were evaluated using two-sample tests of proportion.

The statistical method of Combinatorial Inference has been utilized in studies of cancer biology¹⁷ and in genomics.¹⁸ A variation of this approach was used in this study to identify the specific medication combinations most associated with AKI. First, all of the nephrotoxic medications and medication combinations that were prescribed during our study period were identified from a data set (ie, a training set) containing 75% of all encounters selected at random without replacement. Using this training set, the prevalence of each medication combination and the rate of AKI associated with each combination were identified. The predicted overall AKI risk of an individual medication is the average of all the AKI rates associated with each combination containing that specific medication. Also incorporated into the determination of the predicted AKI risk was the prevalence of that medication combination.

To test our model’s predictive capability, the algorithm was applied to the remaining 25% of the total patient data (ie, the test set). The predicted AKI risk was compared with the actual AKI rate in the test data set. Our model’s predictive capability was represented in a receiver operator characteristic (ROC) analysis. The goal was to achieve an area under the ROC curve (AUC) approaching one as this would reflect 100% sensitivity and 100% specificity, whereas an AUC of 0.5 would represent a random guess (50% chance of being correct).

Lastly, our final step was to use our model’s ROC curve to determine an optimal threshold of AKI risk for which to trigger an alert. This predicted risk threshold was based on our goal to increase our surveillance system’s sensitivity balanced with maintaining an acceptable specificity.

An a priori threshold of P = .05 was used to determine statistical significance of all results. Analyses were conducted in Stata 12.1 (StataCorp LP, College Station, Texas) and R 3.3.2 (R Foundation for Statistical Computing, Vienna, Austria). A sample data set containing replication code for our model can be found in an online repository (https://dataverse.harvard.edu/dataverse/chuan). This study was approved by the Seattle Children’s Institutional Review Board.

 

RESULTS

Sensitivity and Specificity

Of the patient encounters, 14,779 were eligible during the study period. The sensitivity of the system’s ability to identify nephrotoxin-related AKI decreased from 46.9% using Algorithm 1 to 43.3% using Algorithm 2, a change of 3.6% (P = .22). The specificity increased from 73.6% to 89.3%, a change of 15.7% (P < .001; Table 2).

Improvement of Our Nephrotoxin-Related AKI Detection System Using a Novel AKI Prediction Strategy

A total of 838 medication combinations were identified in our training set and the predicted AKI risk for every medication combination was determined. By comparing the predicted risk of AKI to the actual AKI occurrence, an ROC curve with an AUC of 0.756 (Figure) was generated. An increase in system sensitivity was prioritized when determining the optimal AKI risk at which the model would trigger an alert. Setting an alert threshold at a predicted AKI risk of >8%, our model performed with a sensitivity of 74% while decreasing the specificity to 70%.

Identification of High-Risk Nephrotoxic Medications and Medication Combinations

Approximately 200 medication combinations were associated with >8% AKI risk, our new AKI prediction model’s alert threshold. Medication combinations consisting of up to 11 concomitantly prescribed medications were present in our data set. However, many of these combinations were infrequently prescribed. Further analysis, conducted in order to increase the clinical relevance of our findings, identified 10 medications or medication combinations that were both associated with a predicted AKI risk of >8% and that were prescribed on average greater than twice a month (Table 3).

DISCUSSION

The nephrotoxin-related AKI detection system at SCH automatically places orders for serum creatinines on patients who have met criteria for concomitant nephrotoxin exposure. This has given us a robust database from which to develop our clinical decision-making tool. Both our original and updated systems were based on the absolute number of concomitant nephrotoxic medications prescribed.¹⁶ This is a reasonable approach given the complexity of building a surveillance system¹⁹ and resource limitations. However, a system based on observed rather than theoretical or in vitro data, adaptable to the institution and designed for ongoing refinement, would be more valuable.

The interest in AKI prediction tools continues to be high. Bedford et al. employed numerous variables and diagnostic codes to predict the development of AKI in adults during hospitalization. They were able to produce a prediction model with a reasonable fit (AUC 0.72) to identify patients at higher risk for AKI but were less successful in their attempts to predict progression to severe AKI.²⁰ Hodgson et al. recently developed an adult AKI prediction score (AUC 0.65-0.72) also based on numerous clinical factors that was able to positively impact inpatient mortality.²¹ To our knowledge, our model is unique in that it focuses on nephrotoxins using a predicted AKI risk algorithm based on observed AKI rates of previously ordered medications/medication combinations (two to 11 medications). Having a decision tool targeting medications gives the clinician guidance that can be used to make a specific intervention rather than identifying a patient at risk due to a diagnosis code or other difficult to modify factors.

There are abundant case studies and reports using logistic regression models identifying specific medications associated with AKI. Our choice of methodology was based on our assessment that logistic regression models would be inadequate for the development of a real-time clinical decision-making tool for several reasons. Using logistic regression to explore every medication combination based on our medication list would be challenging as there are approximately 5.5 × 10¹⁰ potential medication combinations. Additionally, logistic regression ignores any potential interactions between the medications. This is an important point as medication interactions can be synergistic, neutral, or antagonist. Consequently, the outcome generated from a set of combined variables may be different from one generated from the sum of each variable taken independently. Logistic regression also does not account for the potential prescribing trends among providers as it assumes that all medications or medication combinations are equally available at the same time. However, in practice, depending on numerous factors, such as hospital culture (eg, the presence of clinical standard work pathways), local bacterial resistance patterns, or medication shortages; certain medication combinations may occur more frequently while others not at all. Finally, logistic regression cannot account for the possibility of a medication combination occurring; therefore, logistic regression may identify a combination strongly associated with AKI that is rarely prescribed.

We theorized that AKI detection would improve with the Algorithm 2 modifications, including the expanded nephrotoxin list, which accompanied alignment with the national pediatric AKI collaborative, NINJA. The finding that our surveillance sensitivity did not improve with this system update supported our subsequent objective to develop a novel nephrotoxin-related AKI decision tool or detection system using our EHR data to identify which specific medications and/or medication combinations were associated with a higher rate of AKI. However, it should be noted that two factors related to measurement bias introduce limitations to our sensitivity and specificity analyses. First, regarding the presence of the alert system, our system will order serum creatinines on patients when they have been exposed to nephrotoxins. Consequently, the proportion of patients with creatinines measured will increase in the nephrotoxin-exposed patients. Unexposed patients may have AKI that is not detected because creatinines may not be ordered. Therefore, there is the potential for a relative increase in AKI detection among nephrotoxin-exposed patients as compared with unexposed patients, which would then affect the measured sensitivity and specificity of the alert. Second, the automated alerts require a baseline creatinine in order to trigger therefore are unable to identify AKI among patients who do not have a baseline serum creatinine measurement.

Our new nephrotoxin-related AKI detection model performed best when an alert was triggered for those medications or medication combinations with a predicted AKI risk of >8%. Forty-six medication combinations consisting of exactly two medications were determined to have a predicted AKI risk of >8% therefore would trigger an alert in our new model system. These medication combinations would not have triggered an alert using either of the previous system algorithms as both algorithms are based on the presence of three or more concomitant nephrotoxic medications.

From the list of suspected nephrotoxins, we identified 11 unique medications in 10 different combinations with a predicted AKI risk of >8% that were prescribed frequently (at least twice a month on average; Table 3). Notably, six out of 10 medication combinations involved vancomycin. Piperacillin-tazobactam was also represented in several combinations. These findings support the concern that others have reported regarding these two medications particularly when prescribed together.^22,23

Interestingly, enalapril was identified as a higher-risk medication both alone and in combination with another medication. We do not suspect that enalapril carries a higher risk than other angiotensin-converting enzyme (ACE) inhibitors to increase a patient’s serum creatinine. Rather, we suspect that in our hospitalized patients, this relatively short-acting ACE inhibitor is commonly used in several of our vulnerable populations such as in cardiac and bone marrow transplant patients.

The alert threshold of our model can be adjusted to increase either the sensitivity or the specificity of AKI detection. Our detection sensitivity increased by >1.5-fold with the alert trigger threshold set at a predicted AKI risk of >8%. As a screening tool, our alert limits could be set such that our sensitivity would be greater; however, balancing the potential for alert fatigue is important in determining the acceptance and, ultimately, the success of a working surveillance system.²⁴

A patient’s overall risk of AKI is influenced by many factors such as the presence of underlying chronic comorbidities and the nature or severity of the acute illness as this may affect the patient’s intravascular volume status, systemic blood pressures, or drug metabolism. Our study is limited as we are a children’s hospital and our patients may have fewer comorbidities than seen in the adult population. One could argue that this permits a perspective not clouded by the confounders of chronic disease and allows for the effect of the medications prescribed to be more apparent. However, our study includes critically ill patients and patients who may have been hemodynamically unstable. This may explain why the NINJA algorithm did not improve the sensitivity of our AKI detection as the NINJA collaborative excludes critically ill patients.

Dose and dosing frequency of the prescribed medications could not be taken into account, which could explain the finding that nonsteroidal anti-inflammatory drugs (NSAIDs) such as aspirin, ibuprofen, or ketorolac when used alone were associated with a low (<1%) rate of AKI despite being frequently prescribed. Additionally, as many providers are aware of the AKI risk of NSAIDs, these medications may have been used intermittently (as needed) or in select, perhaps healthier, patients or in patients that take these medications chronically who were admitted for reasons that did not alter their outpatient medication regimen.

Our study also reflects the prescribing habits of our institution and may not be directly applicable to nontertiary care hospitals or centers that do not have large cystic fibrosis, bone marrow, or solid organ transplant populations. Despite our study’s limitations, we feel that there are several findings that are relevant across centers and populations. Our data were derived from the systematic ordering of daily serum creatinines when a patient is at risk for nephrotoxin-related AKI. This is in step with the philosophy advocated by others that AKI identification can only occur if the providers are aware of this risk and are vigilant.²⁵ In this vigilance, we also recognize that not all risks are of the same magnitude and may not deserve the same attention when resources are limited. Our identification of those medication combinations most associated with AKI at our institution has helped us narrow our focus and identify specific areas of potential education and intervention. The specific combinations identified may also be relevant to similar institutions serving similarly complex patients. Those with dissimilar populations could use this methodology to identify those medication combinations most relevant for their patient population and their prescriber’s habits. More studies of this type would be beneficial to the medical community as a whole as certain medication combinations may be found to be high risk regardless of the institution and the age or demographics of the populations they serve.

 

Acknowledgments

Dr. Karyn E. Yonekawa conceptualized and designed the study, directed the data analysis, interpreted the data, drafted, revised and gave final approval of the manuscript. Dr. Chuan Zhou contributed to the study design, acquired data, conducted the data analysis, critically reviewed, and gave final approval of the manuscript. Ms. Wren L. Haaland contributed to the study design, acquired data, conducted the data analysis, critically reviewed, and gave final approval of the manuscript. Dr. Davene R. Wright contributed to the study design, data analysis, critically reviewed, revised, and gave final approval of the manuscript.

The authors would like to thank Holly Clifton and Suzanne Spencer for their assistance with data acquisition and Drs. Derya Caglar, Corrie McDaniel, and Thida Ong for their writing support.

All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

Disclosures

The authors have no conflicts of interest to report.

Acute kidney injury (AKI) is increasingly common in the hospitalized patient^1,2 with recent adult and pediatric multinational studies reporting AKI rates of 57% and 27%, respectively.^3,4 The development of AKI is associated with significant adverse outcomes including an increased risk of mortality.^5-7 For those that survive, the history of AKI may contribute to a lifetime of impaired health with chronic kidney disease.^8,9 This is particularly concerning for pediatric patients as AKI may impact morbidity for many decades, influence available therapies for these morbidities, and ultimately contribute to a shortened lifespan.¹⁰

AKI in the hospitalized patient is no longer accepted as an unfortunate and unavoidable consequence of illness or the indicated therapy. Currently, there is strong interest in this hospital-acquired condition with global initiatives aimed at increased prevention and early detection and treatment of AKI.^11,12 To this objective, risk stratification tools or prediction models could assist clinicians in decision making. Numerous studies have tested AKI prediction models either in particular high-risk populations or based on associated comorbidities, biomarkers, and critical illness scores. These studies are predominantly in adult populations, and few have been externally validated.¹³ While associations between certain medications and AKI are well known, an AKI prediction model that is applicable to pediatric or adult populations and is based on medication exposure is difficult. However, there is a growing recognition of the potential to develop such a model using the electronic health record (EHR).¹⁴

In 2013, Seattle Children’s Hospital (SCH) implemented a nephrotoxin and AKI detection system to assist in clinical decision making within the EHR. This system instituted the automatic ordering of serum creatinines to screen for AKI when the provider ordered three or more medications that were suspected to be nephrotoxic. Other clinical factors such as the diagnoses or preexisting conditions were not considered in the decision-tool algorithm. This original algorithm (Algorithm 1) was later modified and the list of suspected nephrotoxins was expanded (Table 1) in order to align with a national pediatric AKI collaborative (Algorithm 2). However, it was unclear whether the algorithm modification would improve AKI detection.

The present study had two objectives. The first was to evaluate the impact of the modifications on the sensitivity and specificity of our system. The second objective, if either the sensitivity or specificity was determined to be suboptimal, was to develop an improved model for nephrotoxin-related AKI detection. Having either the sensitivity or the specificity under 50% would be equivalent to or worse than a random guess, which we would consider unacceptable.

METHODS

Context

SCH is a tertiary care academic teaching hospital affiliated with the University of Washington School of Medicine, Harborview Medical Center, and the Seattle Cancer Care Alliance. The hospital has 371 licensed beds and approximately 18 medical subspecialty services.

 

Study Population

This was a retrospective cohort study examining all patients ages 0-21 years admitted to SCH between December 1, 2013 and November 30, 2015. The detection system was modified to align with the national pediatric AKI collaborative, Nephrotoxic Injury Negated by Just-in-Time Action (NINJA) in November 2014. Both acute care and intensive care patients were included (data not separated by location). Patients who had end-stage kidney disease and were receiving dialysis and patients who were evaluated in the emergency department without being admitted or admitted as observation status were excluded from analysis. Patients were also excluded if they did not have a baseline serum creatinine as defined below.

Study Measures

AKI is defined at SCH using the Kidney Disease: Improving Global Outcomes Stage 1 criteria as a guideline. The diagnosis of AKI is based on an increase in the baseline serum creatinine by 0.3 mg/dL or an increase in the serum creatinine by >1.5 times the baseline assuming the incoming creatinine is 0.5 mg/dL or higher. For our definition, the increase in serum creatinine needs to have occurred within a one-week timeframe and urine output is not a diagnostic criterion.¹⁵ Baseline serum creatinine is defined as the lowest serum creatinine in the previous six months. Forty medications were classified as nephrotoxins based on previous analysis¹⁶ and adapted for our institutional formulary.

Statistical Analysis

To evaluate the efficacy of our systems in detecting nephrotoxin-related AKI, the sensitivity and the specificity using both our original algorithm (Algorithm 1) and the modified algorithm (Algorithm 2) were generated on our complete data set. To test sensitivity, the proportion of AKI patients who would trigger alert using Algorithm 1 and then with Algorithm 2 was identified. Similarly, to test specificity, the proportion of non-AKI patients who did not trigger an alert by the surveillance systems was identified. The differences in sensitivity and specificity between the two algorithms were evaluated using two-sample tests of proportion.

The statistical method of Combinatorial Inference has been utilized in studies of cancer biology¹⁷ and in genomics.¹⁸ A variation of this approach was used in this study to identify the specific medication combinations most associated with AKI. First, all of the nephrotoxic medications and medication combinations that were prescribed during our study period were identified from a data set (ie, a training set) containing 75% of all encounters selected at random without replacement. Using this training set, the prevalence of each medication combination and the rate of AKI associated with each combination were identified. The predicted overall AKI risk of an individual medication is the average of all the AKI rates associated with each combination containing that specific medication. Also incorporated into the determination of the predicted AKI risk was the prevalence of that medication combination.

To test our model’s predictive capability, the algorithm was applied to the remaining 25% of the total patient data (ie, the test set). The predicted AKI risk was compared with the actual AKI rate in the test data set. Our model’s predictive capability was represented in a receiver operator characteristic (ROC) analysis. The goal was to achieve an area under the ROC curve (AUC) approaching one as this would reflect 100% sensitivity and 100% specificity, whereas an AUC of 0.5 would represent a random guess (50% chance of being correct).

Lastly, our final step was to use our model’s ROC curve to determine an optimal threshold of AKI risk for which to trigger an alert. This predicted risk threshold was based on our goal to increase our surveillance system’s sensitivity balanced with maintaining an acceptable specificity.

An a priori threshold of P = .05 was used to determine statistical significance of all results. Analyses were conducted in Stata 12.1 (StataCorp LP, College Station, Texas) and R 3.3.2 (R Foundation for Statistical Computing, Vienna, Austria). A sample data set containing replication code for our model can be found in an online repository (https://dataverse.harvard.edu/dataverse/chuan). This study was approved by the Seattle Children’s Institutional Review Board.

 

RESULTS

Sensitivity and Specificity

Of the patient encounters, 14,779 were eligible during the study period. The sensitivity of the system’s ability to identify nephrotoxin-related AKI decreased from 46.9% using Algorithm 1 to 43.3% using Algorithm 2, a change of 3.6% (P = .22). The specificity increased from 73.6% to 89.3%, a change of 15.7% (P < .001; Table 2).

Improvement of Our Nephrotoxin-Related AKI Detection System Using a Novel AKI Prediction Strategy

A total of 838 medication combinations were identified in our training set and the predicted AKI risk for every medication combination was determined. By comparing the predicted risk of AKI to the actual AKI occurrence, an ROC curve with an AUC of 0.756 (Figure) was generated. An increase in system sensitivity was prioritized when determining the optimal AKI risk at which the model would trigger an alert. Setting an alert threshold at a predicted AKI risk of >8%, our model performed with a sensitivity of 74% while decreasing the specificity to 70%.

Identification of High-Risk Nephrotoxic Medications and Medication Combinations

Approximately 200 medication combinations were associated with >8% AKI risk, our new AKI prediction model’s alert threshold. Medication combinations consisting of up to 11 concomitantly prescribed medications were present in our data set. However, many of these combinations were infrequently prescribed. Further analysis, conducted in order to increase the clinical relevance of our findings, identified 10 medications or medication combinations that were both associated with a predicted AKI risk of >8% and that were prescribed on average greater than twice a month (Table 3).

DISCUSSION

The nephrotoxin-related AKI detection system at SCH automatically places orders for serum creatinines on patients who have met criteria for concomitant nephrotoxin exposure. This has given us a robust database from which to develop our clinical decision-making tool. Both our original and updated systems were based on the absolute number of concomitant nephrotoxic medications prescribed.¹⁶ This is a reasonable approach given the complexity of building a surveillance system¹⁹ and resource limitations. However, a system based on observed rather than theoretical or in vitro data, adaptable to the institution and designed for ongoing refinement, would be more valuable.

The interest in AKI prediction tools continues to be high. Bedford et al. employed numerous variables and diagnostic codes to predict the development of AKI in adults during hospitalization. They were able to produce a prediction model with a reasonable fit (AUC 0.72) to identify patients at higher risk for AKI but were less successful in their attempts to predict progression to severe AKI.²⁰ Hodgson et al. recently developed an adult AKI prediction score (AUC 0.65-0.72) also based on numerous clinical factors that was able to positively impact inpatient mortality.²¹ To our knowledge, our model is unique in that it focuses on nephrotoxins using a predicted AKI risk algorithm based on observed AKI rates of previously ordered medications/medication combinations (two to 11 medications). Having a decision tool targeting medications gives the clinician guidance that can be used to make a specific intervention rather than identifying a patient at risk due to a diagnosis code or other difficult to modify factors.

There are abundant case studies and reports using logistic regression models identifying specific medications associated with AKI. Our choice of methodology was based on our assessment that logistic regression models would be inadequate for the development of a real-time clinical decision-making tool for several reasons. Using logistic regression to explore every medication combination based on our medication list would be challenging as there are approximately 5.5 × 10¹⁰ potential medication combinations. Additionally, logistic regression ignores any potential interactions between the medications. This is an important point as medication interactions can be synergistic, neutral, or antagonist. Consequently, the outcome generated from a set of combined variables may be different from one generated from the sum of each variable taken independently. Logistic regression also does not account for the potential prescribing trends among providers as it assumes that all medications or medication combinations are equally available at the same time. However, in practice, depending on numerous factors, such as hospital culture (eg, the presence of clinical standard work pathways), local bacterial resistance patterns, or medication shortages; certain medication combinations may occur more frequently while others not at all. Finally, logistic regression cannot account for the possibility of a medication combination occurring; therefore, logistic regression may identify a combination strongly associated with AKI that is rarely prescribed.

We theorized that AKI detection would improve with the Algorithm 2 modifications, including the expanded nephrotoxin list, which accompanied alignment with the national pediatric AKI collaborative, NINJA. The finding that our surveillance sensitivity did not improve with this system update supported our subsequent objective to develop a novel nephrotoxin-related AKI decision tool or detection system using our EHR data to identify which specific medications and/or medication combinations were associated with a higher rate of AKI. However, it should be noted that two factors related to measurement bias introduce limitations to our sensitivity and specificity analyses. First, regarding the presence of the alert system, our system will order serum creatinines on patients when they have been exposed to nephrotoxins. Consequently, the proportion of patients with creatinines measured will increase in the nephrotoxin-exposed patients. Unexposed patients may have AKI that is not detected because creatinines may not be ordered. Therefore, there is the potential for a relative increase in AKI detection among nephrotoxin-exposed patients as compared with unexposed patients, which would then affect the measured sensitivity and specificity of the alert. Second, the automated alerts require a baseline creatinine in order to trigger therefore are unable to identify AKI among patients who do not have a baseline serum creatinine measurement.

Our new nephrotoxin-related AKI detection model performed best when an alert was triggered for those medications or medication combinations with a predicted AKI risk of >8%. Forty-six medication combinations consisting of exactly two medications were determined to have a predicted AKI risk of >8% therefore would trigger an alert in our new model system. These medication combinations would not have triggered an alert using either of the previous system algorithms as both algorithms are based on the presence of three or more concomitant nephrotoxic medications.

From the list of suspected nephrotoxins, we identified 11 unique medications in 10 different combinations with a predicted AKI risk of >8% that were prescribed frequently (at least twice a month on average; Table 3). Notably, six out of 10 medication combinations involved vancomycin. Piperacillin-tazobactam was also represented in several combinations. These findings support the concern that others have reported regarding these two medications particularly when prescribed together.^22,23

Interestingly, enalapril was identified as a higher-risk medication both alone and in combination with another medication. We do not suspect that enalapril carries a higher risk than other angiotensin-converting enzyme (ACE) inhibitors to increase a patient’s serum creatinine. Rather, we suspect that in our hospitalized patients, this relatively short-acting ACE inhibitor is commonly used in several of our vulnerable populations such as in cardiac and bone marrow transplant patients.

The alert threshold of our model can be adjusted to increase either the sensitivity or the specificity of AKI detection. Our detection sensitivity increased by >1.5-fold with the alert trigger threshold set at a predicted AKI risk of >8%. As a screening tool, our alert limits could be set such that our sensitivity would be greater; however, balancing the potential for alert fatigue is important in determining the acceptance and, ultimately, the success of a working surveillance system.²⁴

A patient’s overall risk of AKI is influenced by many factors such as the presence of underlying chronic comorbidities and the nature or severity of the acute illness as this may affect the patient’s intravascular volume status, systemic blood pressures, or drug metabolism. Our study is limited as we are a children’s hospital and our patients may have fewer comorbidities than seen in the adult population. One could argue that this permits a perspective not clouded by the confounders of chronic disease and allows for the effect of the medications prescribed to be more apparent. However, our study includes critically ill patients and patients who may have been hemodynamically unstable. This may explain why the NINJA algorithm did not improve the sensitivity of our AKI detection as the NINJA collaborative excludes critically ill patients.

Dose and dosing frequency of the prescribed medications could not be taken into account, which could explain the finding that nonsteroidal anti-inflammatory drugs (NSAIDs) such as aspirin, ibuprofen, or ketorolac when used alone were associated with a low (<1%) rate of AKI despite being frequently prescribed. Additionally, as many providers are aware of the AKI risk of NSAIDs, these medications may have been used intermittently (as needed) or in select, perhaps healthier, patients or in patients that take these medications chronically who were admitted for reasons that did not alter their outpatient medication regimen.

Our study also reflects the prescribing habits of our institution and may not be directly applicable to nontertiary care hospitals or centers that do not have large cystic fibrosis, bone marrow, or solid organ transplant populations. Despite our study’s limitations, we feel that there are several findings that are relevant across centers and populations. Our data were derived from the systematic ordering of daily serum creatinines when a patient is at risk for nephrotoxin-related AKI. This is in step with the philosophy advocated by others that AKI identification can only occur if the providers are aware of this risk and are vigilant.²⁵ In this vigilance, we also recognize that not all risks are of the same magnitude and may not deserve the same attention when resources are limited. Our identification of those medication combinations most associated with AKI at our institution has helped us narrow our focus and identify specific areas of potential education and intervention. The specific combinations identified may also be relevant to similar institutions serving similarly complex patients. Those with dissimilar populations could use this methodology to identify those medication combinations most relevant for their patient population and their prescriber’s habits. More studies of this type would be beneficial to the medical community as a whole as certain medication combinations may be found to be high risk regardless of the institution and the age or demographics of the populations they serve.

 

Acknowledgments

Dr. Karyn E. Yonekawa conceptualized and designed the study, directed the data analysis, interpreted the data, drafted, revised and gave final approval of the manuscript. Dr. Chuan Zhou contributed to the study design, acquired data, conducted the data analysis, critically reviewed, and gave final approval of the manuscript. Ms. Wren L. Haaland contributed to the study design, acquired data, conducted the data analysis, critically reviewed, and gave final approval of the manuscript. Dr. Davene R. Wright contributed to the study design, data analysis, critically reviewed, revised, and gave final approval of the manuscript.

The authors would like to thank Holly Clifton and Suzanne Spencer for their assistance with data acquisition and Drs. Derya Caglar, Corrie McDaniel, and Thida Ong for their writing support.

All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

Disclosures

The authors have no conflicts of interest to report.

Patients frequently present to dermatologists with nail disorders as their chief concern. Alternatively, nail conditions may be encountered by the examining physician as an incidental finding that may be a clue to underlying systemic disease. Competence in the diagnosis and treatment of nail diseases can drastically improve patient quality of life and can be lifesaving,¹ but many dermatologists find management of nail diseases challenging.² Bridging this educational gap begins with dermatology resident and medical student education. In a collaboration with dermatology educators, the American Academy of Dermatology (AAD) prepared a free online core curriculum for medical students that covers the essential concepts of dermatology. We sought to determine the integration of nail education in the AAD Basic Dermatology Curriculum.

Methods

A cross-sectional study of the AAD Basic Dermatology Curriculum was conducted to determine nail disease content. The curriculum modules were downloaded in June 2018,³ and mentions of nails were recorded and evaluated for overall quantities and relevant content. References to nail procedures and diagnostic techniques including nail biopsies, fungal cultures, microscopy on nail scrapings, nail clippings, and nail-related cancers also were assessed in the analysis.

Results

Of 342 patients discussed in cases and quizzes, nails were mentioned for 19 patients (89 times total)(Table 1). Additionally, there were 2 mentions each of nail clippings and nail tumors, 0 mentions of nail biopsies, and 1 mention each of fungal cultures and microscopy on nail scrapings (Table 1). Of the 40 modules, nails were mentioned in 12 modules (Table 2) and 6 introductions to the modules (Table 1). There were no mentions of the terms nails, subungual, or onychomycosis in the learning objectives.³

Comment

Our study demonstrates a paucity of content relevant to nails in the AAD Basic Dermatology Curriculum. Medical students are missing an important opportunity to learn about diagnosis and management of nail conditions and may incorrectly conclude that nail expertise is not essential to becoming a competent board-certified dermatologist.

Particularly concerning is the exclusion of nail examinations in the skin exam module addressing full-body skin examinations (0 mentions in 31 slides). This curriculum may negatively influence medical students and may then follow at the resident level, with a study reporting that 50.3% (69/137) of residents examine nails only when the patient brings it to their attention.⁴

Most concerning was the inadequate coverage of nail unit melanoma in the melanoma module (1 mention in 53 slides). Furthermore, the ABCDE—asymmetry, border, color, diameter, and evolving—mnemonic for cutaneous melanoma was covered in 6 slides in this module, and the ABCDEF—family history added—mnemonic for nail unit melanoma was completely excluded. Not surprisingly, resident knowledge of melanonychia diagnosis is deficient, with a prior study demonstrating that 62% (88/142) of residents were not confident diagnosing and managing patients with melanonychia, and only 88% (125/142) of residents were aware of the nail melanoma mnemonic.⁴

Similarly, nail biopsy for melanonychia diagnosis was excluded from the curriculum, whereas skin biopsy was thoroughly discussed in the context of a cutaneous melanoma diagnosis. This deficient teaching may track to the dermatology resident curriculum, as a survey of third-year dermatology residents (N=240) showed that 58% performed 10 or fewer nail procedures, and one-third of residents felt incompetent in nail surgery.⁵

We acknowledge that the AAD Basic Dermatology Curriculum is simply an introduction to dermatology. However, given that dermatologists are among the major specialists who care for nail patients, we advocate for more content on nail diseases in this curriculum. Nails can easily be incorporated into existing modules, and a new module specifically dedicated to nail disease should be added. Moreover, we envision that our findings will positively reflect on competence in treating nail disease for dermatology residents.

Patients frequently present to dermatologists with nail disorders as their chief concern. Alternatively, nail conditions may be encountered by the examining physician as an incidental finding that may be a clue to underlying systemic disease. Competence in the diagnosis and treatment of nail diseases can drastically improve patient quality of life and can be lifesaving,¹ but many dermatologists find management of nail diseases challenging.² Bridging this educational gap begins with dermatology resident and medical student education. In a collaboration with dermatology educators, the American Academy of Dermatology (AAD) prepared a free online core curriculum for medical students that covers the essential concepts of dermatology. We sought to determine the integration of nail education in the AAD Basic Dermatology Curriculum.

Methods

A cross-sectional study of the AAD Basic Dermatology Curriculum was conducted to determine nail disease content. The curriculum modules were downloaded in June 2018,³ and mentions of nails were recorded and evaluated for overall quantities and relevant content. References to nail procedures and diagnostic techniques including nail biopsies, fungal cultures, microscopy on nail scrapings, nail clippings, and nail-related cancers also were assessed in the analysis.

Results

Of 342 patients discussed in cases and quizzes, nails were mentioned for 19 patients (89 times total)(Table 1). Additionally, there were 2 mentions each of nail clippings and nail tumors, 0 mentions of nail biopsies, and 1 mention each of fungal cultures and microscopy on nail scrapings (Table 1). Of the 40 modules, nails were mentioned in 12 modules (Table 2) and 6 introductions to the modules (Table 1). There were no mentions of the terms nails, subungual, or onychomycosis in the learning objectives.³

Comment

Our study demonstrates a paucity of content relevant to nails in the AAD Basic Dermatology Curriculum. Medical students are missing an important opportunity to learn about diagnosis and management of nail conditions and may incorrectly conclude that nail expertise is not essential to becoming a competent board-certified dermatologist.

Particularly concerning is the exclusion of nail examinations in the skin exam module addressing full-body skin examinations (0 mentions in 31 slides). This curriculum may negatively influence medical students and may then follow at the resident level, with a study reporting that 50.3% (69/137) of residents examine nails only when the patient brings it to their attention.⁴

Most concerning was the inadequate coverage of nail unit melanoma in the melanoma module (1 mention in 53 slides). Furthermore, the ABCDE—asymmetry, border, color, diameter, and evolving—mnemonic for cutaneous melanoma was covered in 6 slides in this module, and the ABCDEF—family history added—mnemonic for nail unit melanoma was completely excluded. Not surprisingly, resident knowledge of melanonychia diagnosis is deficient, with a prior study demonstrating that 62% (88/142) of residents were not confident diagnosing and managing patients with melanonychia, and only 88% (125/142) of residents were aware of the nail melanoma mnemonic.⁴

Similarly, nail biopsy for melanonychia diagnosis was excluded from the curriculum, whereas skin biopsy was thoroughly discussed in the context of a cutaneous melanoma diagnosis. This deficient teaching may track to the dermatology resident curriculum, as a survey of third-year dermatology residents (N=240) showed that 58% performed 10 or fewer nail procedures, and one-third of residents felt incompetent in nail surgery.⁵

We acknowledge that the AAD Basic Dermatology Curriculum is simply an introduction to dermatology. However, given that dermatologists are among the major specialists who care for nail patients, we advocate for more content on nail diseases in this curriculum. Nails can easily be incorporated into existing modules, and a new module specifically dedicated to nail disease should be added. Moreover, we envision that our findings will positively reflect on competence in treating nail disease for dermatology residents.

Patients frequently present to dermatologists with nail disorders as their chief concern. Alternatively, nail conditions may be encountered by the examining physician as an incidental finding that may be a clue to underlying systemic disease. Competence in the diagnosis and treatment of nail diseases can drastically improve patient quality of life and can be lifesaving,¹ but many dermatologists find management of nail diseases challenging.² Bridging this educational gap begins with dermatology resident and medical student education. In a collaboration with dermatology educators, the American Academy of Dermatology (AAD) prepared a free online core curriculum for medical students that covers the essential concepts of dermatology. We sought to determine the integration of nail education in the AAD Basic Dermatology Curriculum.

Methods

A cross-sectional study of the AAD Basic Dermatology Curriculum was conducted to determine nail disease content. The curriculum modules were downloaded in June 2018,³ and mentions of nails were recorded and evaluated for overall quantities and relevant content. References to nail procedures and diagnostic techniques including nail biopsies, fungal cultures, microscopy on nail scrapings, nail clippings, and nail-related cancers also were assessed in the analysis.

Results

Of 342 patients discussed in cases and quizzes, nails were mentioned for 19 patients (89 times total)(Table 1). Additionally, there were 2 mentions each of nail clippings and nail tumors, 0 mentions of nail biopsies, and 1 mention each of fungal cultures and microscopy on nail scrapings (Table 1). Of the 40 modules, nails were mentioned in 12 modules (Table 2) and 6 introductions to the modules (Table 1). There were no mentions of the terms nails, subungual, or onychomycosis in the learning objectives.³

Comment

Our study demonstrates a paucity of content relevant to nails in the AAD Basic Dermatology Curriculum. Medical students are missing an important opportunity to learn about diagnosis and management of nail conditions and may incorrectly conclude that nail expertise is not essential to becoming a competent board-certified dermatologist.

Particularly concerning is the exclusion of nail examinations in the skin exam module addressing full-body skin examinations (0 mentions in 31 slides). This curriculum may negatively influence medical students and may then follow at the resident level, with a study reporting that 50.3% (69/137) of residents examine nails only when the patient brings it to their attention.⁴

Most concerning was the inadequate coverage of nail unit melanoma in the melanoma module (1 mention in 53 slides). Furthermore, the ABCDE—asymmetry, border, color, diameter, and evolving—mnemonic for cutaneous melanoma was covered in 6 slides in this module, and the ABCDEF—family history added—mnemonic for nail unit melanoma was completely excluded. Not surprisingly, resident knowledge of melanonychia diagnosis is deficient, with a prior study demonstrating that 62% (88/142) of residents were not confident diagnosing and managing patients with melanonychia, and only 88% (125/142) of residents were aware of the nail melanoma mnemonic.⁴

Similarly, nail biopsy for melanonychia diagnosis was excluded from the curriculum, whereas skin biopsy was thoroughly discussed in the context of a cutaneous melanoma diagnosis. This deficient teaching may track to the dermatology resident curriculum, as a survey of third-year dermatology residents (N=240) showed that 58% performed 10 or fewer nail procedures, and one-third of residents felt incompetent in nail surgery.⁵

We acknowledge that the AAD Basic Dermatology Curriculum is simply an introduction to dermatology. However, given that dermatologists are among the major specialists who care for nail patients, we advocate for more content on nail diseases in this curriculum. Nails can easily be incorporated into existing modules, and a new module specifically dedicated to nail disease should be added. Moreover, we envision that our findings will positively reflect on competence in treating nail disease for dermatology residents.

Delirium is an acute change in mental status, affecting more than seven million hospitalized patients in the United States annually.¹ Several factors increase the risk of developing delirium, including advanced age,² cognitive dysfunction,³ hearing and vision impairment,^4-6 and severe illness or major surgery.⁷ Delirium may be precipitated during hospitalization by common inpatient interventions, such as the use of physical restraints, polypharmacy, or bladder catheters.^4,8 In-hospital delirium impacts an estimated 10%-15% of the general medical admissions and as many as 81% of patients in the intensive care unit (ICU).^9-11 Despite the relative frequency with which delirium is encountered in the hospital, subsequent emergency department (ED) presentations or hospital readmissions for these patients are poorly characterized.

 

The development of delirium is associated with several negative outcomes during the hospital stay. Delirium is an independent predictor of prolonged hospital stay,^7,9,12,13 prolonged mechanical ventilation,¹⁴ and mortality during admission.^14,15 Inpatient delirium is associated with functional decline at discharge, leading to a new nursing home placement.^16-19 Preexisting dementia is exacerbated by inpatient delirium, and a new diagnosis of cognitive impairment²⁰ or dementia becomes more common after an episode of delirium.²¹

These data suggest that people diagnosed with delirium may be particularly vulnerable in the posthospitalization period. Hospitals with high rates of unplanned readmissions face penalties from the Centers for Medicare and Medicaid Services.^22,23 However, few investigations have focused on postdischarge healthcare utilization, such as readmission rates and ED visits. Studies that address this topic are limited to postoperative patient populations.²⁴

Using a cohort of hospitalized patients, we examined whether those diagnosed with delirium experienced worse outcomes compared with patients with no such condition. We hypothesized that the patients diagnosed with delirium during hospitalization would experience more readmissions and ED visits within 30 days of discharge compared with those without delirium.

METHODS

Study Design

This single-center retrospective cohort study took place at the Kaiser Permanente San Rafael Medical Center (KP-SRF), a 116-bed general community medical and surgical hospital located in Northern California, from September 6, 2010 to March 31, 2015. The Kaiser Permanente Northern California institutional review board, in accordance with the provisions of the Declaration of the Helsinki and International Conference on Harmonization Guidelines for Good Clinical Practice (CN-15-2491-H), approved this study.

Participants and Eligibility Criteria

This study included Kaiser Permanente members at least 65 years old who were hospitalized at KP-SRF from September 2010 to March 2015. Patient data were obtained from the electronic medical records. Patients with delirium were identified from a delirium registry; all other patients served as controls.

Starting on September 6, 2010, a hospital-wide program was initiated to screen hospitalized medical and surgical patients using the Confusion Assessment Method (CAM).²⁵ As part of this program, nurses completed a four-hour training on delirium; the program included delirium identification and CAM administration. Patients deemed at risk for delirium by their nurse or displaying symptoms of delirium (fluctuation in attention or awareness, disorientation, restlessness, agitation, and psychomotor slowing) were screened by nurses one to two times within a 24-hour period. Physicians were notified by the nurse if their patient screened positive. Nurses were prohibited from performing CAMs in languages that they were not fluent in, thus resulting in screening of primarily English-speaking patients. Psychiatry was consulted at the discretion of the primary team physician to assist with diagnosis and management of delirium. As psychiatry consultation was left up to the discretion of the primary team physician, not all CAM-positive patients were evaluated. The psychiatrists conducted no routine evaluation on the CAM-negative patients unless requested by the primary team physician. The psychiatrist confirmed the delirium diagnosis with a clinical interview and assessment. The patients confirmed with delirium at any point during their hospitalization were prospectively added to a delirium registry. The patients assessed by the psychiatrist as not delirious were excluded from the registry. Only those patients added to the delirium registry during the study period were classified as delirious for this study. All other patients were included as controls. The presence of the nursing screening program using the CAM enriched the cohort, but a positive CAM was unnecessary nor was it sufficient for inclusion in the delirium group (Table 1).

To eliminate the influence of previous delirium episodes on readmission, the subjects were excluded if they reported a prior diagnosis of delirium in 2006 or later, which was the year the electronic medical record was initiated. This diagnosis was determined retrospectively using the following ICD-9 codes: 290.11, 290.3, 290.41, 292.0, 292.81, 292.89, 293.0, 293.0E, 293.0F, 293.1, 293.89, 294.10, 294.21, 304.00, 304.90, 305.50, 331.0, 437.0, 780.09, V11.8, and V15.89.²⁶ Subjects were also excluded if they were ever diagnosed with alcohol-related delirium, as defined by ICD-9 codes 291, 303.9, and 305. Subjects were excluded from the primary analysis if Kaiser Permanente membership lapsed to any degree within 30 days of discharge. Patients who died in the hospital were not excluded; however, the analyses of postdischarge outcomes were conducted on the subpopulation of study subjects who were discharged alive.

For subjects with multiple entries in the delirium registry, the earliest hospitalization during the study period in which a delirium diagnosis was recorded was selected. For eligible patients without a diagnosis of delirium, a single hospitalization was selected randomly from the individual patients during the time period. The analysis database included only one hospitalization for each subject. The flowchart of patient selection is outlined in the Figure.

 

Patient Characteristics

Patient demographics and clinical data were obtained from the electronic medical records. We used several scores to characterize illness severity, including the Charlson comorbidity index,²⁷ Laboratory-Based Acute Physiology, version 2 (LAPS2) score²⁸—an externally validated score for acute severity of illness—and disease categories as defined by the Healthcare Cost and Utilization Project (HCUP).²⁹

Outcomes

The primary outcome was the rate of readmission to the hospital within 30 days of discharge from the hospitalization in which delirium was first diagnosed. Readmissions and ED visits to any Kaiser Permanente hospital and to hospitals outside of the Kaiser Permanente network with Kaiser Permanente insurance were captured. To avoid incorrectly coding patients transferred from the index hospital to another hospital as readmissions, we excluded readmissions that occurred on the day of discharge or the following calendar day. This action was expected to lower the absolute number of readmissions but restrict the analysis to true readmissions. The models of postdischarge outcomes are based on the subset of patients discharged alive. The secondary outcome measures included discharge from the index hospitalization to a skilled nursing facility or hospice rather than to home and emergency room visits within 30 days of discharge. We also quantified rates of mortality during hospitalization and at 30 days postdischarge.

Statistical Analysis

Comparisons between patients with delirium and those without were performed using Pearson’s X² test for categorical variables and student t-test for continuous variables. The estimated odds of our outcome measures for delirious and nondelirious subjects were calculated from multivariable logistic regression models, which controlled for predictors of delirium and additional information obtained during the hospitalization. For inpatient outcomes (in-hospital mortality and discharge to skilled nursing facility or hospice), we adjusted only for admission characteristics: age, race/ethnicity, admission to ICU, Charlson comorbidity index, HCUP category, and admission category. To limit the number of variables in our model, we consolidated the initial 30 HCUP categories (Appendix Table 1) by illness type into 13 categories (Appendix Table 2). For postdischarge outcomes, we adjusted for all the variables, including disposition (Table 2). The average estimated odds were calculated based on the observed marginal distribution of the control variables. The P value indicates how likely the odds on each outcome for delirious subjects differed significantly from those for other subjects. All statistical analyses were performed using SAS 9.3 (SAS Institute Inc., Cary, North Carolina).

RESULTS

Demographics and Clinical Characteristics

A total of 718 patients with delirium and 7,927 patients without delirium were included in this study. The related demographic information is outlined in Table 2. On average, the patients with delirium were older (83 ± 8 years versus 77 ± 8 years, P < .0001) but no difference in gender distribution was observed between groups. A similar racial breakdown was noted between groups, with white patients accounting for 87% of both patients with delirium and those without. The majority of admissions were unplanned medical admissions. The delirium cohort included more emergent surgical admissions compared with patients who did not develop delirium. Patients who developed delirium exhibited higher levels of illness severity on admission, as measured by the Charlson and LAPS2 scores, and were more often admitted to the ICU. Significant differences were also observed between admission illness categories between patients with delirium and those without.

 

Primary Outcome

Delirium during admission was significantly associated with hospital readmission within 30 days of discharge (adjusted odds ratio [aOR] = 2.60, 95% CI: 1.96–3.44; P < .0001; Table 3).

Secondary Outcomes

Delirium during admission was significantly (P < .0001; Table 3) associated with an ED visit within 30 days of discharge (OR: 2.18; 95% CI: 1.77–2.69) and discharge to a skilled nursing facility or hospice rather than home (OR: 2.52; 95% CI: 2.09–3.01). Delirium was not associated (P > .1) with death during hospitalization nor death 30 days following discharge.

As the delirious patients were much more likely to be discharged to a skilled nursing facility than nondelirious patients, we tested whether discharge disposition influenced readmission rates and ED visits between delirious and nondelirious patients in an unadjusted univariate analysis. The association between delirium and readmission and ED utilization was present regardless of disposition. Among patients discharged to skilled nursing, readmission rates were 4.76% and 13.38% (P < .001), and ED visit rates were 12.29% and 23.24% (P < .001) for nondelirious and delirious patients, respectively. Among patients discharged home, readmission rates were 4.96% and 14.37% (P < .001), and ED visit rates were 11.93% and 29.04% (P < .001) for nondelirious and delirious patients, respectively.

DISCUSSION

In this study of patients in a community hospital in Northern California, we observed a significant association between inpatient delirium and risk of hospital readmission within 30 days of discharge. We also demonstrated increased skilled nursing facility placement and ED utilization after discharge among hospitalized patients with delirium compared with those without. Patients with delirium in this study were diagnosed by a psychiatrist—a gold standard³⁰—and the study was conducted in a health system database with near comprehensive ascertainment of readmissions. These results suggest that patients with delirium are particularly vulnerable in the posthospitalization period and are a key group to focusing on reducing readmission rates and postdischarge healthcare utilization.

Identifying the risk factors for hospital readmission is important for the benefit of both the patient and the hospital. In an analysis of Medicare claims data from 2003 to 2004, 19.6% of beneficiaries were readmitted within 30 days of discharge.³¹ There is a national effort to reduce unplanned hospital readmissions for both patient safety as hospitals with high readmission rates face penalties from the Centers for Medicare and Medicaid Services.^22,23 Why delirium is associated with readmission remains unclear. Delirium may precipitate aspiration events, reduce oral intake which complicates medication administration and nutrition, or reduced mobility, leading to pulmonary emboli and skin breakdown, any of which could lead to readmission.³² Delirium may also accelerate the progression of cognitive decline and overall loss of functional independence.²⁰ Delirious patients can be difficult to care for at home, and persistent delirium may lead to returns to the ED and readmission. Strategies to reduce readmissions associated with delirium may need to focus on both prevention of hospital-acquired delirium and targeted caregiver and patient support after discharge.

Hospital readmission and ED visits are not mutually exclusive experiences. In the United States, the majority of patients admitted to the hospital are admitted through the ED.³³ Thus, most of the readmissions in this cohort were also likely counted as 30-day ED visits. However, as ED utilization occurs regardless of whether a patient is discharged or admitted from the ED, we reported all ED visits in this analysis, similar to other studies.³⁴ More delirium patients returned to the ED 30 days postdischarge than were ultimately readmitted to the hospital, and delirious patients were more likely to visit the ED or be readmitted than nondelirious patients. These observations point toward the first 30 days after discharge as a crucial period for these patients.

Our study features several strengths. To our knowledge, this study is one of the largest investigations of inpatients with delirium. One distinguishing feature was that all cases of delirium in this study were diagnosed by a psychiatrist, which is considered a gold standard. Many studies rely solely on brief nursing-administered surveys for delirium diagnosis. Using Kaiser Permanente data allowed for more complete follow-up of patients, including vital status. Kaiser Permanente is both a medical system and an insurer, resulting in acquisition of detailed health information from all hospitalizations where Kaiser Permanente insurance was used for each patient. Therefore, patients were only lost to follow-up following discharge in the event of a membership lapse; these patients were excluded from analysis. The obtained data are also more generalizable than those of other studies examining readmission rates in delirious patients as the hospital where these data were collected is a 116-bed general community medical and surgical hospital. Thus, the patients enrolled in this study covered multiple hospital services with a variety of admission diagnoses. This condition contrasts with much of the existing literature on inpatient delirium; these studies mostly center on specific medical conditions or surgeries and are often conducted at academic medical centers. At the same time, Kaiser Permanente is a unique health maintenance organization focused on preventive care, and readmission rates are possibly lower than elsewhere given the universal access to primary care for Kaiser Permanente members. Our results may not generalize to patients hospitalized in other health systems.

The diagnosis of delirium is a clinical diagnosis without biomarkers or radiographic markers and is also underdiagnosed and poorly coded.³² For these reasons, delirium can be challenging to study in large administrative databases or data derived from electronic medical records. We addressed this limitation by classifying the delirium patients only when they had been diagnosed by a staff psychiatrist. However, not all patients who screened positive with the CAM were evaluated by the staff psychiatrist during the study period. Thus, several CAM-positive patients who were not evaluated by psychiatry were included in the control population. This situation may cause bias toward identification of more severe cases of delirium. Although the physicians were encouraged to consult the psychiatry department for any patients who screened positive for delirium with the CAM, the psychiatrist may not have been involved if patients were managed without consultation. These patients may have exhibited less severe delirium or hypoactive delirium. In addition, the CAM fails to detect all delirious patients; interrater variability may occur with CAM administration, and non-English speaking patients are more likely to be excluded.³⁵ These situations are another possible way for our control population to include some delirious patients and those patients with less severe or hypoactive subtypes. While this might bias toward the null hypothesis, it is also possible our results only indicate an association between more clinically apparent delirium and readmission. A major limitation of this study is that we were unable to quantify the number of cohort patients screened with the CAM or the results of screening, thus limiting our ability to quantify the impact of potential biases introduced by the screening program.

This study may have underestimated readmission rates. We defined readmissions as all hospitalizations at any Kaiser Permanente facility, or to an alternate facility where Kaiser Permanente insurance was used, within 30 days of discharge. We excluded the day of discharge or the following calendar day to avoid mischaracterizing transfers from the index hospital to another Kaiser Permanente facility as readmissions. This step was conducted to avoid biasing our comparison, as delirious patients are less frequently discharged home than nondelirious patients. Therefore, while the relative odds of readmission between delirious and nondelirious patients reported in this study should be generalizable to other community hospitals, the absolute readmission rates reported here may not be comparable to those reported in other studies.

Delirium may represent a marker of more severe illness or medical complications accrued during the hospitalization, which could lead to the associations observed in this study due to confounding.³² Patients with delirium are more likely to be admitted emergently, admitted to the ICU, and feature higher acuity conditions than patients without delirium. We attempted to mitigate this possibility by using a multivariable model to control for variables related to illness severity, including the Charlson comorbidity index, HCUP diagnostic categories, and ICU admission. Despite including HCUP diagnostic categories in our model, we were unable to capture the contribution of certain diseases with finer granularity, such as preexistent dementia, which may also affect clinical outcomes.³⁶ Similarly, although we incorporated markers of illness severity into our model, we were unable to adjust for baseline functional status or frailty, which were not reliably recorded in the electronic medical record but are potential confounders when investigating clinical outcomes including hospital readmission.

We also lacked information regarding the duration of delirium in our cohort. Therefore, we were unable to test whether longer episodes of delirium were more predictive of readmission than shorter episodes.

 

CONCLUSION

In-hospital delirium is associated with several negative patient outcomes. Our study demonstrates that delirium predicts 30-day readmission and emergency department utilization after hospital discharge. Bearing in mind that a third of hospital-acquired delirium cases may be preventable,³² hospitals should prioritize interventions to reduce postdischarge healthcare utilization and complications in this particularly vulnerable group.

Acknowledgments

The authors would like to acknowledge Dr. Andrew L. Avins for his guidance with the initial development of this project and Julie Fourie for contributing data to the overall study.

Disclosures

Dr. Liu receives funding from NIH K23GM112018 and NIGMS R35128672. Dr. Josephson receives compensation as the JAMA Neurology Editor in Chief and Continuum Audio Associate Editor. The remaining authors have no conflicts of interest.

Funding

This study was funded by Kaiser Permanente Graduate Medical Education, who approved the design, conduct, and reporting of this study.

Delirium is an acute change in mental status, affecting more than seven million hospitalized patients in the United States annually.¹ Several factors increase the risk of developing delirium, including advanced age,² cognitive dysfunction,³ hearing and vision impairment,^4-6 and severe illness or major surgery.⁷ Delirium may be precipitated during hospitalization by common inpatient interventions, such as the use of physical restraints, polypharmacy, or bladder catheters.^4,8 In-hospital delirium impacts an estimated 10%-15% of the general medical admissions and as many as 81% of patients in the intensive care unit (ICU).^9-11 Despite the relative frequency with which delirium is encountered in the hospital, subsequent emergency department (ED) presentations or hospital readmissions for these patients are poorly characterized.

 

The development of delirium is associated with several negative outcomes during the hospital stay. Delirium is an independent predictor of prolonged hospital stay,^7,9,12,13 prolonged mechanical ventilation,¹⁴ and mortality during admission.^14,15 Inpatient delirium is associated with functional decline at discharge, leading to a new nursing home placement.^16-19 Preexisting dementia is exacerbated by inpatient delirium, and a new diagnosis of cognitive impairment²⁰ or dementia becomes more common after an episode of delirium.²¹

These data suggest that people diagnosed with delirium may be particularly vulnerable in the posthospitalization period. Hospitals with high rates of unplanned readmissions face penalties from the Centers for Medicare and Medicaid Services.^22,23 However, few investigations have focused on postdischarge healthcare utilization, such as readmission rates and ED visits. Studies that address this topic are limited to postoperative patient populations.²⁴

Using a cohort of hospitalized patients, we examined whether those diagnosed with delirium experienced worse outcomes compared with patients with no such condition. We hypothesized that the patients diagnosed with delirium during hospitalization would experience more readmissions and ED visits within 30 days of discharge compared with those without delirium.

METHODS

Study Design

This single-center retrospective cohort study took place at the Kaiser Permanente San Rafael Medical Center (KP-SRF), a 116-bed general community medical and surgical hospital located in Northern California, from September 6, 2010 to March 31, 2015. The Kaiser Permanente Northern California institutional review board, in accordance with the provisions of the Declaration of the Helsinki and International Conference on Harmonization Guidelines for Good Clinical Practice (CN-15-2491-H), approved this study.

Participants and Eligibility Criteria

This study included Kaiser Permanente members at least 65 years old who were hospitalized at KP-SRF from September 2010 to March 2015. Patient data were obtained from the electronic medical records. Patients with delirium were identified from a delirium registry; all other patients served as controls.

Starting on September 6, 2010, a hospital-wide program was initiated to screen hospitalized medical and surgical patients using the Confusion Assessment Method (CAM).²⁵ As part of this program, nurses completed a four-hour training on delirium; the program included delirium identification and CAM administration. Patients deemed at risk for delirium by their nurse or displaying symptoms of delirium (fluctuation in attention or awareness, disorientation, restlessness, agitation, and psychomotor slowing) were screened by nurses one to two times within a 24-hour period. Physicians were notified by the nurse if their patient screened positive. Nurses were prohibited from performing CAMs in languages that they were not fluent in, thus resulting in screening of primarily English-speaking patients. Psychiatry was consulted at the discretion of the primary team physician to assist with diagnosis and management of delirium. As psychiatry consultation was left up to the discretion of the primary team physician, not all CAM-positive patients were evaluated. The psychiatrists conducted no routine evaluation on the CAM-negative patients unless requested by the primary team physician. The psychiatrist confirmed the delirium diagnosis with a clinical interview and assessment. The patients confirmed with delirium at any point during their hospitalization were prospectively added to a delirium registry. The patients assessed by the psychiatrist as not delirious were excluded from the registry. Only those patients added to the delirium registry during the study period were classified as delirious for this study. All other patients were included as controls. The presence of the nursing screening program using the CAM enriched the cohort, but a positive CAM was unnecessary nor was it sufficient for inclusion in the delirium group (Table 1).

To eliminate the influence of previous delirium episodes on readmission, the subjects were excluded if they reported a prior diagnosis of delirium in 2006 or later, which was the year the electronic medical record was initiated. This diagnosis was determined retrospectively using the following ICD-9 codes: 290.11, 290.3, 290.41, 292.0, 292.81, 292.89, 293.0, 293.0E, 293.0F, 293.1, 293.89, 294.10, 294.21, 304.00, 304.90, 305.50, 331.0, 437.0, 780.09, V11.8, and V15.89.²⁶ Subjects were also excluded if they were ever diagnosed with alcohol-related delirium, as defined by ICD-9 codes 291, 303.9, and 305. Subjects were excluded from the primary analysis if Kaiser Permanente membership lapsed to any degree within 30 days of discharge. Patients who died in the hospital were not excluded; however, the analyses of postdischarge outcomes were conducted on the subpopulation of study subjects who were discharged alive.

For subjects with multiple entries in the delirium registry, the earliest hospitalization during the study period in which a delirium diagnosis was recorded was selected. For eligible patients without a diagnosis of delirium, a single hospitalization was selected randomly from the individual patients during the time period. The analysis database included only one hospitalization for each subject. The flowchart of patient selection is outlined in the Figure.

 

Patient Characteristics

Patient demographics and clinical data were obtained from the electronic medical records. We used several scores to characterize illness severity, including the Charlson comorbidity index,²⁷ Laboratory-Based Acute Physiology, version 2 (LAPS2) score²⁸—an externally validated score for acute severity of illness—and disease categories as defined by the Healthcare Cost and Utilization Project (HCUP).²⁹

Outcomes

The primary outcome was the rate of readmission to the hospital within 30 days of discharge from the hospitalization in which delirium was first diagnosed. Readmissions and ED visits to any Kaiser Permanente hospital and to hospitals outside of the Kaiser Permanente network with Kaiser Permanente insurance were captured. To avoid incorrectly coding patients transferred from the index hospital to another hospital as readmissions, we excluded readmissions that occurred on the day of discharge or the following calendar day. This action was expected to lower the absolute number of readmissions but restrict the analysis to true readmissions. The models of postdischarge outcomes are based on the subset of patients discharged alive. The secondary outcome measures included discharge from the index hospitalization to a skilled nursing facility or hospice rather than to home and emergency room visits within 30 days of discharge. We also quantified rates of mortality during hospitalization and at 30 days postdischarge.

Statistical Analysis

Comparisons between patients with delirium and those without were performed using Pearson’s X² test for categorical variables and student t-test for continuous variables. The estimated odds of our outcome measures for delirious and nondelirious subjects were calculated from multivariable logistic regression models, which controlled for predictors of delirium and additional information obtained during the hospitalization. For inpatient outcomes (in-hospital mortality and discharge to skilled nursing facility or hospice), we adjusted only for admission characteristics: age, race/ethnicity, admission to ICU, Charlson comorbidity index, HCUP category, and admission category. To limit the number of variables in our model, we consolidated the initial 30 HCUP categories (Appendix Table 1) by illness type into 13 categories (Appendix Table 2). For postdischarge outcomes, we adjusted for all the variables, including disposition (Table 2). The average estimated odds were calculated based on the observed marginal distribution of the control variables. The P value indicates how likely the odds on each outcome for delirious subjects differed significantly from those for other subjects. All statistical analyses were performed using SAS 9.3 (SAS Institute Inc., Cary, North Carolina).

RESULTS

Demographics and Clinical Characteristics

A total of 718 patients with delirium and 7,927 patients without delirium were included in this study. The related demographic information is outlined in Table 2. On average, the patients with delirium were older (83 ± 8 years versus 77 ± 8 years, P < .0001) but no difference in gender distribution was observed between groups. A similar racial breakdown was noted between groups, with white patients accounting for 87% of both patients with delirium and those without. The majority of admissions were unplanned medical admissions. The delirium cohort included more emergent surgical admissions compared with patients who did not develop delirium. Patients who developed delirium exhibited higher levels of illness severity on admission, as measured by the Charlson and LAPS2 scores, and were more often admitted to the ICU. Significant differences were also observed between admission illness categories between patients with delirium and those without.

 

Primary Outcome

Delirium during admission was significantly associated with hospital readmission within 30 days of discharge (adjusted odds ratio [aOR] = 2.60, 95% CI: 1.96–3.44; P < .0001; Table 3).

Secondary Outcomes

Delirium during admission was significantly (P < .0001; Table 3) associated with an ED visit within 30 days of discharge (OR: 2.18; 95% CI: 1.77–2.69) and discharge to a skilled nursing facility or hospice rather than home (OR: 2.52; 95% CI: 2.09–3.01). Delirium was not associated (P > .1) with death during hospitalization nor death 30 days following discharge.

As the delirious patients were much more likely to be discharged to a skilled nursing facility than nondelirious patients, we tested whether discharge disposition influenced readmission rates and ED visits between delirious and nondelirious patients in an unadjusted univariate analysis. The association between delirium and readmission and ED utilization was present regardless of disposition. Among patients discharged to skilled nursing, readmission rates were 4.76% and 13.38% (P < .001), and ED visit rates were 12.29% and 23.24% (P < .001) for nondelirious and delirious patients, respectively. Among patients discharged home, readmission rates were 4.96% and 14.37% (P < .001), and ED visit rates were 11.93% and 29.04% (P < .001) for nondelirious and delirious patients, respectively.

DISCUSSION

In this study of patients in a community hospital in Northern California, we observed a significant association between inpatient delirium and risk of hospital readmission within 30 days of discharge. We also demonstrated increased skilled nursing facility placement and ED utilization after discharge among hospitalized patients with delirium compared with those without. Patients with delirium in this study were diagnosed by a psychiatrist—a gold standard³⁰—and the study was conducted in a health system database with near comprehensive ascertainment of readmissions. These results suggest that patients with delirium are particularly vulnerable in the posthospitalization period and are a key group to focusing on reducing readmission rates and postdischarge healthcare utilization.

Identifying the risk factors for hospital readmission is important for the benefit of both the patient and the hospital. In an analysis of Medicare claims data from 2003 to 2004, 19.6% of beneficiaries were readmitted within 30 days of discharge.³¹ There is a national effort to reduce unplanned hospital readmissions for both patient safety as hospitals with high readmission rates face penalties from the Centers for Medicare and Medicaid Services.^22,23 Why delirium is associated with readmission remains unclear. Delirium may precipitate aspiration events, reduce oral intake which complicates medication administration and nutrition, or reduced mobility, leading to pulmonary emboli and skin breakdown, any of which could lead to readmission.³² Delirium may also accelerate the progression of cognitive decline and overall loss of functional independence.²⁰ Delirious patients can be difficult to care for at home, and persistent delirium may lead to returns to the ED and readmission. Strategies to reduce readmissions associated with delirium may need to focus on both prevention of hospital-acquired delirium and targeted caregiver and patient support after discharge.

Hospital readmission and ED visits are not mutually exclusive experiences. In the United States, the majority of patients admitted to the hospital are admitted through the ED.³³ Thus, most of the readmissions in this cohort were also likely counted as 30-day ED visits. However, as ED utilization occurs regardless of whether a patient is discharged or admitted from the ED, we reported all ED visits in this analysis, similar to other studies.³⁴ More delirium patients returned to the ED 30 days postdischarge than were ultimately readmitted to the hospital, and delirious patients were more likely to visit the ED or be readmitted than nondelirious patients. These observations point toward the first 30 days after discharge as a crucial period for these patients.

Our study features several strengths. To our knowledge, this study is one of the largest investigations of inpatients with delirium. One distinguishing feature was that all cases of delirium in this study were diagnosed by a psychiatrist, which is considered a gold standard. Many studies rely solely on brief nursing-administered surveys for delirium diagnosis. Using Kaiser Permanente data allowed for more complete follow-up of patients, including vital status. Kaiser Permanente is both a medical system and an insurer, resulting in acquisition of detailed health information from all hospitalizations where Kaiser Permanente insurance was used for each patient. Therefore, patients were only lost to follow-up following discharge in the event of a membership lapse; these patients were excluded from analysis. The obtained data are also more generalizable than those of other studies examining readmission rates in delirious patients as the hospital where these data were collected is a 116-bed general community medical and surgical hospital. Thus, the patients enrolled in this study covered multiple hospital services with a variety of admission diagnoses. This condition contrasts with much of the existing literature on inpatient delirium; these studies mostly center on specific medical conditions or surgeries and are often conducted at academic medical centers. At the same time, Kaiser Permanente is a unique health maintenance organization focused on preventive care, and readmission rates are possibly lower than elsewhere given the universal access to primary care for Kaiser Permanente members. Our results may not generalize to patients hospitalized in other health systems.

The diagnosis of delirium is a clinical diagnosis without biomarkers or radiographic markers and is also underdiagnosed and poorly coded.³² For these reasons, delirium can be challenging to study in large administrative databases or data derived from electronic medical records. We addressed this limitation by classifying the delirium patients only when they had been diagnosed by a staff psychiatrist. However, not all patients who screened positive with the CAM were evaluated by the staff psychiatrist during the study period. Thus, several CAM-positive patients who were not evaluated by psychiatry were included in the control population. This situation may cause bias toward identification of more severe cases of delirium. Although the physicians were encouraged to consult the psychiatry department for any patients who screened positive for delirium with the CAM, the psychiatrist may not have been involved if patients were managed without consultation. These patients may have exhibited less severe delirium or hypoactive delirium. In addition, the CAM fails to detect all delirious patients; interrater variability may occur with CAM administration, and non-English speaking patients are more likely to be excluded.³⁵ These situations are another possible way for our control population to include some delirious patients and those patients with less severe or hypoactive subtypes. While this might bias toward the null hypothesis, it is also possible our results only indicate an association between more clinically apparent delirium and readmission. A major limitation of this study is that we were unable to quantify the number of cohort patients screened with the CAM or the results of screening, thus limiting our ability to quantify the impact of potential biases introduced by the screening program.

This study may have underestimated readmission rates. We defined readmissions as all hospitalizations at any Kaiser Permanente facility, or to an alternate facility where Kaiser Permanente insurance was used, within 30 days of discharge. We excluded the day of discharge or the following calendar day to avoid mischaracterizing transfers from the index hospital to another Kaiser Permanente facility as readmissions. This step was conducted to avoid biasing our comparison, as delirious patients are less frequently discharged home than nondelirious patients. Therefore, while the relative odds of readmission between delirious and nondelirious patients reported in this study should be generalizable to other community hospitals, the absolute readmission rates reported here may not be comparable to those reported in other studies.

Delirium may represent a marker of more severe illness or medical complications accrued during the hospitalization, which could lead to the associations observed in this study due to confounding.³² Patients with delirium are more likely to be admitted emergently, admitted to the ICU, and feature higher acuity conditions than patients without delirium. We attempted to mitigate this possibility by using a multivariable model to control for variables related to illness severity, including the Charlson comorbidity index, HCUP diagnostic categories, and ICU admission. Despite including HCUP diagnostic categories in our model, we were unable to capture the contribution of certain diseases with finer granularity, such as preexistent dementia, which may also affect clinical outcomes.³⁶ Similarly, although we incorporated markers of illness severity into our model, we were unable to adjust for baseline functional status or frailty, which were not reliably recorded in the electronic medical record but are potential confounders when investigating clinical outcomes including hospital readmission.

We also lacked information regarding the duration of delirium in our cohort. Therefore, we were unable to test whether longer episodes of delirium were more predictive of readmission than shorter episodes.

 

CONCLUSION

In-hospital delirium is associated with several negative patient outcomes. Our study demonstrates that delirium predicts 30-day readmission and emergency department utilization after hospital discharge. Bearing in mind that a third of hospital-acquired delirium cases may be preventable,³² hospitals should prioritize interventions to reduce postdischarge healthcare utilization and complications in this particularly vulnerable group.

Acknowledgments

The authors would like to acknowledge Dr. Andrew L. Avins for his guidance with the initial development of this project and Julie Fourie for contributing data to the overall study.

Disclosures

Dr. Liu receives funding from NIH K23GM112018 and NIGMS R35128672. Dr. Josephson receives compensation as the JAMA Neurology Editor in Chief and Continuum Audio Associate Editor. The remaining authors have no conflicts of interest.

Funding

This study was funded by Kaiser Permanente Graduate Medical Education, who approved the design, conduct, and reporting of this study.

Delirium is an acute change in mental status, affecting more than seven million hospitalized patients in the United States annually.¹ Several factors increase the risk of developing delirium, including advanced age,² cognitive dysfunction,³ hearing and vision impairment,^4-6 and severe illness or major surgery.⁷ Delirium may be precipitated during hospitalization by common inpatient interventions, such as the use of physical restraints, polypharmacy, or bladder catheters.^4,8 In-hospital delirium impacts an estimated 10%-15% of the general medical admissions and as many as 81% of patients in the intensive care unit (ICU).^9-11 Despite the relative frequency with which delirium is encountered in the hospital, subsequent emergency department (ED) presentations or hospital readmissions for these patients are poorly characterized.

 

The development of delirium is associated with several negative outcomes during the hospital stay. Delirium is an independent predictor of prolonged hospital stay,^7,9,12,13 prolonged mechanical ventilation,¹⁴ and mortality during admission.^14,15 Inpatient delirium is associated with functional decline at discharge, leading to a new nursing home placement.^16-19 Preexisting dementia is exacerbated by inpatient delirium, and a new diagnosis of cognitive impairment²⁰ or dementia becomes more common after an episode of delirium.²¹

These data suggest that people diagnosed with delirium may be particularly vulnerable in the posthospitalization period. Hospitals with high rates of unplanned readmissions face penalties from the Centers for Medicare and Medicaid Services.^22,23 However, few investigations have focused on postdischarge healthcare utilization, such as readmission rates and ED visits. Studies that address this topic are limited to postoperative patient populations.²⁴

Using a cohort of hospitalized patients, we examined whether those diagnosed with delirium experienced worse outcomes compared with patients with no such condition. We hypothesized that the patients diagnosed with delirium during hospitalization would experience more readmissions and ED visits within 30 days of discharge compared with those without delirium.

METHODS

Study Design

This single-center retrospective cohort study took place at the Kaiser Permanente San Rafael Medical Center (KP-SRF), a 116-bed general community medical and surgical hospital located in Northern California, from September 6, 2010 to March 31, 2015. The Kaiser Permanente Northern California institutional review board, in accordance with the provisions of the Declaration of the Helsinki and International Conference on Harmonization Guidelines for Good Clinical Practice (CN-15-2491-H), approved this study.

Participants and Eligibility Criteria

This study included Kaiser Permanente members at least 65 years old who were hospitalized at KP-SRF from September 2010 to March 2015. Patient data were obtained from the electronic medical records. Patients with delirium were identified from a delirium registry; all other patients served as controls.

Starting on September 6, 2010, a hospital-wide program was initiated to screen hospitalized medical and surgical patients using the Confusion Assessment Method (CAM).²⁵ As part of this program, nurses completed a four-hour training on delirium; the program included delirium identification and CAM administration. Patients deemed at risk for delirium by their nurse or displaying symptoms of delirium (fluctuation in attention or awareness, disorientation, restlessness, agitation, and psychomotor slowing) were screened by nurses one to two times within a 24-hour period. Physicians were notified by the nurse if their patient screened positive. Nurses were prohibited from performing CAMs in languages that they were not fluent in, thus resulting in screening of primarily English-speaking patients. Psychiatry was consulted at the discretion of the primary team physician to assist with diagnosis and management of delirium. As psychiatry consultation was left up to the discretion of the primary team physician, not all CAM-positive patients were evaluated. The psychiatrists conducted no routine evaluation on the CAM-negative patients unless requested by the primary team physician. The psychiatrist confirmed the delirium diagnosis with a clinical interview and assessment. The patients confirmed with delirium at any point during their hospitalization were prospectively added to a delirium registry. The patients assessed by the psychiatrist as not delirious were excluded from the registry. Only those patients added to the delirium registry during the study period were classified as delirious for this study. All other patients were included as controls. The presence of the nursing screening program using the CAM enriched the cohort, but a positive CAM was unnecessary nor was it sufficient for inclusion in the delirium group (Table 1).

To eliminate the influence of previous delirium episodes on readmission, the subjects were excluded if they reported a prior diagnosis of delirium in 2006 or later, which was the year the electronic medical record was initiated. This diagnosis was determined retrospectively using the following ICD-9 codes: 290.11, 290.3, 290.41, 292.0, 292.81, 292.89, 293.0, 293.0E, 293.0F, 293.1, 293.89, 294.10, 294.21, 304.00, 304.90, 305.50, 331.0, 437.0, 780.09, V11.8, and V15.89.²⁶ Subjects were also excluded if they were ever diagnosed with alcohol-related delirium, as defined by ICD-9 codes 291, 303.9, and 305. Subjects were excluded from the primary analysis if Kaiser Permanente membership lapsed to any degree within 30 days of discharge. Patients who died in the hospital were not excluded; however, the analyses of postdischarge outcomes were conducted on the subpopulation of study subjects who were discharged alive.

For subjects with multiple entries in the delirium registry, the earliest hospitalization during the study period in which a delirium diagnosis was recorded was selected. For eligible patients without a diagnosis of delirium, a single hospitalization was selected randomly from the individual patients during the time period. The analysis database included only one hospitalization for each subject. The flowchart of patient selection is outlined in the Figure.

 

Patient Characteristics

Patient demographics and clinical data were obtained from the electronic medical records. We used several scores to characterize illness severity, including the Charlson comorbidity index,²⁷ Laboratory-Based Acute Physiology, version 2 (LAPS2) score²⁸—an externally validated score for acute severity of illness—and disease categories as defined by the Healthcare Cost and Utilization Project (HCUP).²⁹

Outcomes

The primary outcome was the rate of readmission to the hospital within 30 days of discharge from the hospitalization in which delirium was first diagnosed. Readmissions and ED visits to any Kaiser Permanente hospital and to hospitals outside of the Kaiser Permanente network with Kaiser Permanente insurance were captured. To avoid incorrectly coding patients transferred from the index hospital to another hospital as readmissions, we excluded readmissions that occurred on the day of discharge or the following calendar day. This action was expected to lower the absolute number of readmissions but restrict the analysis to true readmissions. The models of postdischarge outcomes are based on the subset of patients discharged alive. The secondary outcome measures included discharge from the index hospitalization to a skilled nursing facility or hospice rather than to home and emergency room visits within 30 days of discharge. We also quantified rates of mortality during hospitalization and at 30 days postdischarge.

Statistical Analysis

Comparisons between patients with delirium and those without were performed using Pearson’s X² test for categorical variables and student t-test for continuous variables. The estimated odds of our outcome measures for delirious and nondelirious subjects were calculated from multivariable logistic regression models, which controlled for predictors of delirium and additional information obtained during the hospitalization. For inpatient outcomes (in-hospital mortality and discharge to skilled nursing facility or hospice), we adjusted only for admission characteristics: age, race/ethnicity, admission to ICU, Charlson comorbidity index, HCUP category, and admission category. To limit the number of variables in our model, we consolidated the initial 30 HCUP categories (Appendix Table 1) by illness type into 13 categories (Appendix Table 2). For postdischarge outcomes, we adjusted for all the variables, including disposition (Table 2). The average estimated odds were calculated based on the observed marginal distribution of the control variables. The P value indicates how likely the odds on each outcome for delirious subjects differed significantly from those for other subjects. All statistical analyses were performed using SAS 9.3 (SAS Institute Inc., Cary, North Carolina).

RESULTS

Demographics and Clinical Characteristics

A total of 718 patients with delirium and 7,927 patients without delirium were included in this study. The related demographic information is outlined in Table 2. On average, the patients with delirium were older (83 ± 8 years versus 77 ± 8 years, P < .0001) but no difference in gender distribution was observed between groups. A similar racial breakdown was noted between groups, with white patients accounting for 87% of both patients with delirium and those without. The majority of admissions were unplanned medical admissions. The delirium cohort included more emergent surgical admissions compared with patients who did not develop delirium. Patients who developed delirium exhibited higher levels of illness severity on admission, as measured by the Charlson and LAPS2 scores, and were more often admitted to the ICU. Significant differences were also observed between admission illness categories between patients with delirium and those without.

 

Primary Outcome

Delirium during admission was significantly associated with hospital readmission within 30 days of discharge (adjusted odds ratio [aOR] = 2.60, 95% CI: 1.96–3.44; P < .0001; Table 3).

Secondary Outcomes

Delirium during admission was significantly (P < .0001; Table 3) associated with an ED visit within 30 days of discharge (OR: 2.18; 95% CI: 1.77–2.69) and discharge to a skilled nursing facility or hospice rather than home (OR: 2.52; 95% CI: 2.09–3.01). Delirium was not associated (P > .1) with death during hospitalization nor death 30 days following discharge.

As the delirious patients were much more likely to be discharged to a skilled nursing facility than nondelirious patients, we tested whether discharge disposition influenced readmission rates and ED visits between delirious and nondelirious patients in an unadjusted univariate analysis. The association between delirium and readmission and ED utilization was present regardless of disposition. Among patients discharged to skilled nursing, readmission rates were 4.76% and 13.38% (P < .001), and ED visit rates were 12.29% and 23.24% (P < .001) for nondelirious and delirious patients, respectively. Among patients discharged home, readmission rates were 4.96% and 14.37% (P < .001), and ED visit rates were 11.93% and 29.04% (P < .001) for nondelirious and delirious patients, respectively.

DISCUSSION

In this study of patients in a community hospital in Northern California, we observed a significant association between inpatient delirium and risk of hospital readmission within 30 days of discharge. We also demonstrated increased skilled nursing facility placement and ED utilization after discharge among hospitalized patients with delirium compared with those without. Patients with delirium in this study were diagnosed by a psychiatrist—a gold standard³⁰—and the study was conducted in a health system database with near comprehensive ascertainment of readmissions. These results suggest that patients with delirium are particularly vulnerable in the posthospitalization period and are a key group to focusing on reducing readmission rates and postdischarge healthcare utilization.

Identifying the risk factors for hospital readmission is important for the benefit of both the patient and the hospital. In an analysis of Medicare claims data from 2003 to 2004, 19.6% of beneficiaries were readmitted within 30 days of discharge.³¹ There is a national effort to reduce unplanned hospital readmissions for both patient safety as hospitals with high readmission rates face penalties from the Centers for Medicare and Medicaid Services.^22,23 Why delirium is associated with readmission remains unclear. Delirium may precipitate aspiration events, reduce oral intake which complicates medication administration and nutrition, or reduced mobility, leading to pulmonary emboli and skin breakdown, any of which could lead to readmission.³² Delirium may also accelerate the progression of cognitive decline and overall loss of functional independence.²⁰ Delirious patients can be difficult to care for at home, and persistent delirium may lead to returns to the ED and readmission. Strategies to reduce readmissions associated with delirium may need to focus on both prevention of hospital-acquired delirium and targeted caregiver and patient support after discharge.

Hospital readmission and ED visits are not mutually exclusive experiences. In the United States, the majority of patients admitted to the hospital are admitted through the ED.³³ Thus, most of the readmissions in this cohort were also likely counted as 30-day ED visits. However, as ED utilization occurs regardless of whether a patient is discharged or admitted from the ED, we reported all ED visits in this analysis, similar to other studies.³⁴ More delirium patients returned to the ED 30 days postdischarge than were ultimately readmitted to the hospital, and delirious patients were more likely to visit the ED or be readmitted than nondelirious patients. These observations point toward the first 30 days after discharge as a crucial period for these patients.

Our study features several strengths. To our knowledge, this study is one of the largest investigations of inpatients with delirium. One distinguishing feature was that all cases of delirium in this study were diagnosed by a psychiatrist, which is considered a gold standard. Many studies rely solely on brief nursing-administered surveys for delirium diagnosis. Using Kaiser Permanente data allowed for more complete follow-up of patients, including vital status. Kaiser Permanente is both a medical system and an insurer, resulting in acquisition of detailed health information from all hospitalizations where Kaiser Permanente insurance was used for each patient. Therefore, patients were only lost to follow-up following discharge in the event of a membership lapse; these patients were excluded from analysis. The obtained data are also more generalizable than those of other studies examining readmission rates in delirious patients as the hospital where these data were collected is a 116-bed general community medical and surgical hospital. Thus, the patients enrolled in this study covered multiple hospital services with a variety of admission diagnoses. This condition contrasts with much of the existing literature on inpatient delirium; these studies mostly center on specific medical conditions or surgeries and are often conducted at academic medical centers. At the same time, Kaiser Permanente is a unique health maintenance organization focused on preventive care, and readmission rates are possibly lower than elsewhere given the universal access to primary care for Kaiser Permanente members. Our results may not generalize to patients hospitalized in other health systems.

The diagnosis of delirium is a clinical diagnosis without biomarkers or radiographic markers and is also underdiagnosed and poorly coded.³² For these reasons, delirium can be challenging to study in large administrative databases or data derived from electronic medical records. We addressed this limitation by classifying the delirium patients only when they had been diagnosed by a staff psychiatrist. However, not all patients who screened positive with the CAM were evaluated by the staff psychiatrist during the study period. Thus, several CAM-positive patients who were not evaluated by psychiatry were included in the control population. This situation may cause bias toward identification of more severe cases of delirium. Although the physicians were encouraged to consult the psychiatry department for any patients who screened positive for delirium with the CAM, the psychiatrist may not have been involved if patients were managed without consultation. These patients may have exhibited less severe delirium or hypoactive delirium. In addition, the CAM fails to detect all delirious patients; interrater variability may occur with CAM administration, and non-English speaking patients are more likely to be excluded.³⁵ These situations are another possible way for our control population to include some delirious patients and those patients with less severe or hypoactive subtypes. While this might bias toward the null hypothesis, it is also possible our results only indicate an association between more clinically apparent delirium and readmission. A major limitation of this study is that we were unable to quantify the number of cohort patients screened with the CAM or the results of screening, thus limiting our ability to quantify the impact of potential biases introduced by the screening program.

This study may have underestimated readmission rates. We defined readmissions as all hospitalizations at any Kaiser Permanente facility, or to an alternate facility where Kaiser Permanente insurance was used, within 30 days of discharge. We excluded the day of discharge or the following calendar day to avoid mischaracterizing transfers from the index hospital to another Kaiser Permanente facility as readmissions. This step was conducted to avoid biasing our comparison, as delirious patients are less frequently discharged home than nondelirious patients. Therefore, while the relative odds of readmission between delirious and nondelirious patients reported in this study should be generalizable to other community hospitals, the absolute readmission rates reported here may not be comparable to those reported in other studies.

Delirium may represent a marker of more severe illness or medical complications accrued during the hospitalization, which could lead to the associations observed in this study due to confounding.³² Patients with delirium are more likely to be admitted emergently, admitted to the ICU, and feature higher acuity conditions than patients without delirium. We attempted to mitigate this possibility by using a multivariable model to control for variables related to illness severity, including the Charlson comorbidity index, HCUP diagnostic categories, and ICU admission. Despite including HCUP diagnostic categories in our model, we were unable to capture the contribution of certain diseases with finer granularity, such as preexistent dementia, which may also affect clinical outcomes.³⁶ Similarly, although we incorporated markers of illness severity into our model, we were unable to adjust for baseline functional status or frailty, which were not reliably recorded in the electronic medical record but are potential confounders when investigating clinical outcomes including hospital readmission.

We also lacked information regarding the duration of delirium in our cohort. Therefore, we were unable to test whether longer episodes of delirium were more predictive of readmission than shorter episodes.

 

CONCLUSION

In-hospital delirium is associated with several negative patient outcomes. Our study demonstrates that delirium predicts 30-day readmission and emergency department utilization after hospital discharge. Bearing in mind that a third of hospital-acquired delirium cases may be preventable,³² hospitals should prioritize interventions to reduce postdischarge healthcare utilization and complications in this particularly vulnerable group.

Acknowledgments

The authors would like to acknowledge Dr. Andrew L. Avins for his guidance with the initial development of this project and Julie Fourie for contributing data to the overall study.

Disclosures

Dr. Liu receives funding from NIH K23GM112018 and NIGMS R35128672. Dr. Josephson receives compensation as the JAMA Neurology Editor in Chief and Continuum Audio Associate Editor. The remaining authors have no conflicts of interest.

Funding

This study was funded by Kaiser Permanente Graduate Medical Education, who approved the design, conduct, and reporting of this study.

Almost all specialties in internal medicine have a sound scientific research base through which clinical practice is informed.¹ For the field of Hospital Medicine (HM), this evidence has largely comprised research generated from fields outside of the specialty. The need to develop, invest, and grow investigators in hospital-based medicine remains unmet as HM and its footprint in hospital systems continue to grow.^2,3

Despite this fact, little is known about the current state of research in HM. A 2014 survey of the members of the Society of Hospital Medicine (SHM) found that research output across the field of HM, as measured on the basis of peer-reviewed publications, was growing.⁴ Since then, however, the numbers of individuals engaged in research activities, their background and training, publication output, or funding sources have not been quantified. Similarly, little is known about which institutions support the development of junior investigators (ie, HM research fellowships), how these programs are funded, and whether or not matriculants enter the field as investigators. These gaps must be measured, evaluated, and ideally addressed through strategic policy and funding initiatives to advance the state of science within HM.

Members of the SHM Research Committee developed, designed, and deployed a survey to improve the understanding of the state of research in HM. In this study, we aimed to establish the baseline of research in HM to enable the measurement of progress through periodic waves of data collection. Specifically, we sought to quantify and describe the characteristics of existing research programs, the sources and types of funding, the number and background of faculty, and the availability of resources for training researchers in HM.

 

METHODS

Study Setting and Participants

Given that no defined list, database, or external resource that identifies research programs and contacts in HM exists, we began by creating a strategy to identify and sample adult HM programs and their leaders engaged in research activity. We iteratively developed a two-step approach to maximize inclusivity. First, we partnered with SHM to identify programs and leaders actively engaging in research activities. SHM is the largest professional organization within HM and maintains an extensive membership database that includes the titles, e-mail addresses, and affiliations of hospitalists in the United States, including academic and nonacademic sites. This list was manually scanned, and the leaders of academic and research programs in adult HM were identified by examining their titles (eg, Division Chief, Research Lead, etc.) and academic affiliations. During this step, members of the committee noticed that certain key individuals were either missing, no longer occupying their role/title, or had been replaced by others. Therefore, we performed a second step and asked the members of the SHM Research Committee to identify academic and research leaders by using current personal contacts, publication history, and social networks. We asked members to identify individuals and programs that had received grant funding, were actively presenting research at SHM (or other major national venues), and/or were producing peer-reviewed publications related to HM. These programs were purposefully chosen (ie, over HM programs known for clinical activities) to create an enriched sample of those engaged in research in HM. The research committee performed the “second pass” to ensure that established investigators who may not be accurately captured within the SHM database were included to maximize yield for the survey. Finally, these two sources were merged to ensure the absence of duplicate contacts and the identification of a primary respondent for each affiliate. As a result, a convenience sample of 100 programs and corresponding individuals was compiled for the purposes of this survey.

Survey Development

A workgroup within the SHM Research Committee was tasked to create a survey that would achieve four distinct goals: (1) identify institutions currently engaging in hospital-based research; (2) define the characteristics, including sources of research funding, training opportunities, criteria for promotion, and grant support, of research programs within institutions; (3) understand the prevalence of research fellowship programs, including size, training curricula, and funding sources; and (4) evaluate the productivity and funding sources of HM investigators at each site.

Survey questions that target each of these domains were drafted by the workgroup. Questions were pretested with colleagues outside the workgroup focused on this project (ie, from the main research committee). The instrument was refined and edited to improve the readability and clarity of questions on the basis of the feedback obtained through the iterative process. The revised instrument was then programmed into an online survey administration tool (SurveyMonkey^®) to facilitate electronic dissemination. Finally, the members of the workgroup tested the online survey to ensure functionality. No identifiable information was collected from respondents, and no monetary incentive was offered for the completion of the survey. An invitation to participate in the survey was sent via e-mail to each of the program contacts identified.

 

Statistical Analysis

Descriptive statistics, including proportions, means, and percentages, were used to tabulate results. All analyses were conducted using Stata 13 MP/SE (StataCorp, College Station, Texas).

Ethical and Regulatory Considerations

The study was reviewed and deemed exempt from regulation by the University of Michigan Institutional Review Board (HUM000138628).

RESULTS

General Characteristics of Research Programs and Faculty

Out of 100 program contacts, 28 (representing 1,586 faculty members) responded and were included in the survey (program response rate = 28%). When comparing programs that did respond with those that did not, a greater proportion of programs in university settings were noted among respondents (79% vs 21%). Respondents represented programs from all regions of the United States, with most representing university-based (79%), university-affiliated (14%) or Veterans Health Administration (VHA; 11%) programs. Most respondents were in leadership roles, including division chiefs (32%), research directors/leads (21%), section chiefs (18%), and related titles, such as program director. Respondents indicated that the total number of faculty members in their programs (including nonclinicians and advance practice providers) varied from eight to 152 (mean [SD] = 57 [36]) members, with physicians representing the majority of faculty members (Table 1).

Among the 1,586 faculty members within the 28 programs, respondents identified 192 faculty members (12%) as currently receiving extra- or intramural support for research activities. Of these faculty, over half (58%) received <25% of effort from intra or extramural sources, and 28 (15%) and 52 (27%) faculty members received 25%-50% or >50% of support for their effort, respectively. The number of investigators who received funding across programs ranged from 0 to 28 faculty members. Compared with the 192 funded investigators, respondents indicated that a larger number of faculty in their programs (n = 656 or 41%) were involved in local quality improvement (QI) efforts. Of the 656 faculty members involved in QI efforts, 241 individuals (37%) were internally funded and received protected time/effort for their work.

Key Attributes of Research Programs

In the evaluation of the amount of total grant funding, respondents from 17 programs indicated that they received $500,000 in annual extra and intramural funding, and those from three programs stated that they received $500,000 to $999,999 in funding. Five respondents indicated that their programs currently received $1 million to $5 million in grant funding, and three reported >$5 million in research support. The sources of research funding included several divisions within the National Institute of Health (NIH, 12 programs), Agency for Healthcare Research and Quality (AHRQ, four programs), foundations (four programs), and internal grants (six programs). Additionally, six programs indicated “other” sources of funding that included the VHA, Patient-Centered Outcomes Research Institute (PCORI), Centers for Medicare and Medicaid Services, Centers for Disease Control (CDC), and industry sources.

A range of grants, including career development awards (11 programs); small grants, such as R21 and R03s (eight programs); R-level grants, including VA merit awards (five programs); program series grants, such as P and U grants (five programs), and foundation grants (eight programs), were reported as types of awards. Respondents from 16 programs indicated that they provided internal pilot grants. Amounts for such grants ranged from <$50,000 (14 programs) to $50,000-$100,000 (two programs).

 

Research Fellowship Programs/Training Programs

Only five of the 28 surveyed programs indicated that they currently had a research training or fellowship program for developing hospitalist investigators. The age of these programs varied from <1 year to 10 years. Three of the five programs stated that they had two fellows per year, and two stated they had spots for one trainee annually. All respondents indicated that fellows received training on study design, research methods, quantitative (eg, large database and secondary analyses) and qualitative data analysis. In addition, two programs included training in systematic review and meta-analyses, and three included focused courses on healthcare policy. Four of the five programs included training in QI tools, such as LEAN and Six Sigma. Funding for four of the five fellowship programs came from internal sources (eg, department and CTSA). However, two programs added they received some support from extramural funding and philanthropy. Following training, respondents from programs indicated that the majority of their graduates (60%) went on to hybrid research/QI roles (50/50 research/clinical effort), whereas 40% obtained dedicated research investigator (80/20) positions (Table 2).

The 23 institutions without research training programs cited that the most important barrier for establishing such programs was lack of funding (12 programs) and the lack of a pipeline of hospitalists seeking such training (six programs). However, 15 programs indicated that opportunities for hospitalists to gain research training in the form of courses were available internally (eg, courses in the department or medical school) or externally (eg, School of Public Health). Seven programs indicated that they were planning to start a HM research fellowship within the next five years.

Research Faculty

Among the 28 respondents, 15 stated that they have faculty members who conduct research as their main professional activity (ie, >50% effort). The number of faculty members in each program in such roles varied from one to 10. Respondents indicated that faculty members in this category were most often midcareer assistant or associate professors with few full professors. All programs indicated that scholarship in the form of peer-reviewed publications was required for the promotion of faculty. Faculty members who performed research as their main activity had all received formal fellowship training and consequently had dual degrees (MD with MPH or MD, with MSc being the two most common combinations). With respect to clinical activities, most respondents indicated that research faculty spent 10% to 49% of their effort on clinical work. However, five respondents indicated that research faculty had <10% effort on clinical duties (Table 3).

Eleven respondents (39%) identified the main focus of faculty as health service research, where four (14%) identified their main focus as clinical trials. Regardless of funding status, all respondents stated that their faculty were interested in studying quality and process improvement efforts (eg, transitions or readmissions, n = 19), patient safety initiatives (eg, hospital-acquired complications, n = 17), and disease-specific areas (eg, thrombosis, n = 15).

In terms of research output, 12 respondents stated that their research/QI faculty collectively published 11-50 peer-reviewed papers during the academic year, and 10 programs indicated that their faculty published 0-10 papers per year. Only three programs reported that their faculty collectively published 50-99 peer-reviewed papers per year. With respect to abstract presentations at national conferences, 13 programs indicated that they presented 0-10 abstracts, and 12 indicated that they presented 11-50.

 

DISCUSSION

In this first survey quantifying research activities in HM, respondents from 28 programs shared important insights into research activities at their institutions. Although our sample size was small, substantial variation in the size, composition, and structure of research programs in HM among respondents was observed. For example, few respondents indicated the availability of training programs for research in HM at their institutions. Similarly, among faculty who focused mainly on research, variation in funding streams and effort protection was observed. A preponderance of midcareer faculty with a range of funding sources, including NIH, AHRQ, VHA, CMS, and CDC was reported. Collectively, these data not only provide a unique glimpse into the state of research in HM but also help establish a baseline of the status of the field at large.

Some findings of our study are intuitive given our sampling strategy and the types of programs that responded. For example, the fact that most respondents for research programs represented university-based or affiliated institutions is expected given the tripartite academic mission. However, even within our sample of highly motivated programs, some findings are surprising and merit further exploration. For example, the observation that some respondents identified HM investigators within their program with <25% in intra- or extramural funding was unexpected. On the other extreme, we were surprised to find that three programs reported >$5 million in research funding. Understanding whether specific factors, such as the availability of experienced mentors within and outside departments or assistance from support staff (eg, statisticians and project managers), are associated with success and funding within these programs are important questions to answer. By focusing on these issues, we will be well poised as a field to understand what works, what does not work, and why.

Likewise, the finding that few programs within our sample offer formal training in the form of fellowships to research investigators represents an improvement opportunity. A pipeline for growing investigators is critical for the specialty that is HM. Notably, this call is not new; rather, previous investigators have highlighted the importance of developing academically oriented hospitalists for the future of the field.⁵ The implementation of faculty scholarship development programs has improved the scholarly output, mentoring activities, and succession planning of academics within HM.^6,7 Conversely, lack of adequate mentorship and support for academic activities remains a challenge and as a factor associated with the failure to produce academic work.⁸ Without a cadre of investigators asking critical questions related to care delivery, the legitimacy of our field may be threatened.

While extrapolating to the field is difficult given the small number of our respondents, highlighting the progress that has been made is important. For example, while misalignment between funding and clinical and research mission persists, our survey found that several programs have been successful in securing extramural funding for their investigators. Additionally, internal funding for QI work appears to be increasing, with hospitalists receiving dedicated effort for much of this work. Innovation in how best to support and develop these types of efforts have also emerged. For example, the University of Michigan Specialist Hospitalist Allied Research Program offers dedicated effort and funding for hospitalists tackling projects germane to HM (eg, ordering of blood cultures for febrile inpatients) that overlap with subspecialists (eg, infectious diseases).⁹ Thus, hospitalists are linked with other specialties in the development of research agendas and academic products. Similarly, the launch of the HOMERUN network, a coalition of investigators who bridge health systems to study problems central to HM, has helped usher in a new era of research opportunities in the specialty.¹⁰ Fundamentally, the culture of HM has begun to place an emphasis on academic and scholarly productivity in addition to clinical prowess.^11-13 Increased support and funding for training programs geared toward innovation and research in HM is needed to continue this mission. The Society for General Internal Medicine, American College of Physicians, and SHM have important roles to play as the largest professional organizations for generalists in this respect. Support for research, QI, and investigators in HM remains an urgent and largely unmet need.

Our study has limitations. First, our response rate was low at 28% but is consistent with the response rates of other surveys of physician groups.¹⁴ Caution in making inferences to the field at large is necessary given the potential for selection and nonresponse bias. However, we expect that respondents are likely biased toward programs actively conducting research and engaged in QI, thus better reflecting the state of these activities in HM. Second, given that we did not ask for any identifying information, we have no way of establishing the accuracy of the data provided by respondents. However, we have no reason to believe that responses would be altered in a systematic fashion. Future studies that link our findings to publicly available data (eg, databases of active grants and funding) might be useful. Third, while our survey instrument was created and internally validated by hospitalist researchers, its lack of external validation could limit findings. Finally, our results vary on the basis of how respondents answered questions related to effort and time allocation given that these measures differ across programs.

In summary, the findings from this study highlight substantial variations in the number, training, and funding of research faculty across HM programs. Understanding the factors behind the success of some programs and the failures of others appears important in informing and growing the research in the field. Future studies that aim to expand survey participation, raise the awareness of the state of research in HM, and identify barriers and facilitators to academic success in HM are needed.

 

Disclosures

Dr. Chopra discloses grant funding from the Agency for Healthcare Research and Quality (AHRQ), VA Health Services and Research Department, and Centers for Disease Control. Dr. Jones discloses grant funding from AHRQ. All other authors disclose no conflicts of interest.

Almost all specialties in internal medicine have a sound scientific research base through which clinical practice is informed.¹ For the field of Hospital Medicine (HM), this evidence has largely comprised research generated from fields outside of the specialty. The need to develop, invest, and grow investigators in hospital-based medicine remains unmet as HM and its footprint in hospital systems continue to grow.^2,3

Despite this fact, little is known about the current state of research in HM. A 2014 survey of the members of the Society of Hospital Medicine (SHM) found that research output across the field of HM, as measured on the basis of peer-reviewed publications, was growing.⁴ Since then, however, the numbers of individuals engaged in research activities, their background and training, publication output, or funding sources have not been quantified. Similarly, little is known about which institutions support the development of junior investigators (ie, HM research fellowships), how these programs are funded, and whether or not matriculants enter the field as investigators. These gaps must be measured, evaluated, and ideally addressed through strategic policy and funding initiatives to advance the state of science within HM.

Members of the SHM Research Committee developed, designed, and deployed a survey to improve the understanding of the state of research in HM. In this study, we aimed to establish the baseline of research in HM to enable the measurement of progress through periodic waves of data collection. Specifically, we sought to quantify and describe the characteristics of existing research programs, the sources and types of funding, the number and background of faculty, and the availability of resources for training researchers in HM.

 

METHODS

Study Setting and Participants

Given that no defined list, database, or external resource that identifies research programs and contacts in HM exists, we began by creating a strategy to identify and sample adult HM programs and their leaders engaged in research activity. We iteratively developed a two-step approach to maximize inclusivity. First, we partnered with SHM to identify programs and leaders actively engaging in research activities. SHM is the largest professional organization within HM and maintains an extensive membership database that includes the titles, e-mail addresses, and affiliations of hospitalists in the United States, including academic and nonacademic sites. This list was manually scanned, and the leaders of academic and research programs in adult HM were identified by examining their titles (eg, Division Chief, Research Lead, etc.) and academic affiliations. During this step, members of the committee noticed that certain key individuals were either missing, no longer occupying their role/title, or had been replaced by others. Therefore, we performed a second step and asked the members of the SHM Research Committee to identify academic and research leaders by using current personal contacts, publication history, and social networks. We asked members to identify individuals and programs that had received grant funding, were actively presenting research at SHM (or other major national venues), and/or were producing peer-reviewed publications related to HM. These programs were purposefully chosen (ie, over HM programs known for clinical activities) to create an enriched sample of those engaged in research in HM. The research committee performed the “second pass” to ensure that established investigators who may not be accurately captured within the SHM database were included to maximize yield for the survey. Finally, these two sources were merged to ensure the absence of duplicate contacts and the identification of a primary respondent for each affiliate. As a result, a convenience sample of 100 programs and corresponding individuals was compiled for the purposes of this survey.

Survey Development

A workgroup within the SHM Research Committee was tasked to create a survey that would achieve four distinct goals: (1) identify institutions currently engaging in hospital-based research; (2) define the characteristics, including sources of research funding, training opportunities, criteria for promotion, and grant support, of research programs within institutions; (3) understand the prevalence of research fellowship programs, including size, training curricula, and funding sources; and (4) evaluate the productivity and funding sources of HM investigators at each site.

Survey questions that target each of these domains were drafted by the workgroup. Questions were pretested with colleagues outside the workgroup focused on this project (ie, from the main research committee). The instrument was refined and edited to improve the readability and clarity of questions on the basis of the feedback obtained through the iterative process. The revised instrument was then programmed into an online survey administration tool (SurveyMonkey^®) to facilitate electronic dissemination. Finally, the members of the workgroup tested the online survey to ensure functionality. No identifiable information was collected from respondents, and no monetary incentive was offered for the completion of the survey. An invitation to participate in the survey was sent via e-mail to each of the program contacts identified.

 

Statistical Analysis

Descriptive statistics, including proportions, means, and percentages, were used to tabulate results. All analyses were conducted using Stata 13 MP/SE (StataCorp, College Station, Texas).

Ethical and Regulatory Considerations

The study was reviewed and deemed exempt from regulation by the University of Michigan Institutional Review Board (HUM000138628).

RESULTS

General Characteristics of Research Programs and Faculty

Out of 100 program contacts, 28 (representing 1,586 faculty members) responded and were included in the survey (program response rate = 28%). When comparing programs that did respond with those that did not, a greater proportion of programs in university settings were noted among respondents (79% vs 21%). Respondents represented programs from all regions of the United States, with most representing university-based (79%), university-affiliated (14%) or Veterans Health Administration (VHA; 11%) programs. Most respondents were in leadership roles, including division chiefs (32%), research directors/leads (21%), section chiefs (18%), and related titles, such as program director. Respondents indicated that the total number of faculty members in their programs (including nonclinicians and advance practice providers) varied from eight to 152 (mean [SD] = 57 [36]) members, with physicians representing the majority of faculty members (Table 1).

Among the 1,586 faculty members within the 28 programs, respondents identified 192 faculty members (12%) as currently receiving extra- or intramural support for research activities. Of these faculty, over half (58%) received <25% of effort from intra or extramural sources, and 28 (15%) and 52 (27%) faculty members received 25%-50% or >50% of support for their effort, respectively. The number of investigators who received funding across programs ranged from 0 to 28 faculty members. Compared with the 192 funded investigators, respondents indicated that a larger number of faculty in their programs (n = 656 or 41%) were involved in local quality improvement (QI) efforts. Of the 656 faculty members involved in QI efforts, 241 individuals (37%) were internally funded and received protected time/effort for their work.

Key Attributes of Research Programs

In the evaluation of the amount of total grant funding, respondents from 17 programs indicated that they received $500,000 in annual extra and intramural funding, and those from three programs stated that they received $500,000 to $999,999 in funding. Five respondents indicated that their programs currently received $1 million to $5 million in grant funding, and three reported >$5 million in research support. The sources of research funding included several divisions within the National Institute of Health (NIH, 12 programs), Agency for Healthcare Research and Quality (AHRQ, four programs), foundations (four programs), and internal grants (six programs). Additionally, six programs indicated “other” sources of funding that included the VHA, Patient-Centered Outcomes Research Institute (PCORI), Centers for Medicare and Medicaid Services, Centers for Disease Control (CDC), and industry sources.

A range of grants, including career development awards (11 programs); small grants, such as R21 and R03s (eight programs); R-level grants, including VA merit awards (five programs); program series grants, such as P and U grants (five programs), and foundation grants (eight programs), were reported as types of awards. Respondents from 16 programs indicated that they provided internal pilot grants. Amounts for such grants ranged from <$50,000 (14 programs) to $50,000-$100,000 (two programs).

 

Research Fellowship Programs/Training Programs

Only five of the 28 surveyed programs indicated that they currently had a research training or fellowship program for developing hospitalist investigators. The age of these programs varied from <1 year to 10 years. Three of the five programs stated that they had two fellows per year, and two stated they had spots for one trainee annually. All respondents indicated that fellows received training on study design, research methods, quantitative (eg, large database and secondary analyses) and qualitative data analysis. In addition, two programs included training in systematic review and meta-analyses, and three included focused courses on healthcare policy. Four of the five programs included training in QI tools, such as LEAN and Six Sigma. Funding for four of the five fellowship programs came from internal sources (eg, department and CTSA). However, two programs added they received some support from extramural funding and philanthropy. Following training, respondents from programs indicated that the majority of their graduates (60%) went on to hybrid research/QI roles (50/50 research/clinical effort), whereas 40% obtained dedicated research investigator (80/20) positions (Table 2).

The 23 institutions without research training programs cited that the most important barrier for establishing such programs was lack of funding (12 programs) and the lack of a pipeline of hospitalists seeking such training (six programs). However, 15 programs indicated that opportunities for hospitalists to gain research training in the form of courses were available internally (eg, courses in the department or medical school) or externally (eg, School of Public Health). Seven programs indicated that they were planning to start a HM research fellowship within the next five years.

Research Faculty

Among the 28 respondents, 15 stated that they have faculty members who conduct research as their main professional activity (ie, >50% effort). The number of faculty members in each program in such roles varied from one to 10. Respondents indicated that faculty members in this category were most often midcareer assistant or associate professors with few full professors. All programs indicated that scholarship in the form of peer-reviewed publications was required for the promotion of faculty. Faculty members who performed research as their main activity had all received formal fellowship training and consequently had dual degrees (MD with MPH or MD, with MSc being the two most common combinations). With respect to clinical activities, most respondents indicated that research faculty spent 10% to 49% of their effort on clinical work. However, five respondents indicated that research faculty had <10% effort on clinical duties (Table 3).

Eleven respondents (39%) identified the main focus of faculty as health service research, where four (14%) identified their main focus as clinical trials. Regardless of funding status, all respondents stated that their faculty were interested in studying quality and process improvement efforts (eg, transitions or readmissions, n = 19), patient safety initiatives (eg, hospital-acquired complications, n = 17), and disease-specific areas (eg, thrombosis, n = 15).

In terms of research output, 12 respondents stated that their research/QI faculty collectively published 11-50 peer-reviewed papers during the academic year, and 10 programs indicated that their faculty published 0-10 papers per year. Only three programs reported that their faculty collectively published 50-99 peer-reviewed papers per year. With respect to abstract presentations at national conferences, 13 programs indicated that they presented 0-10 abstracts, and 12 indicated that they presented 11-50.

 

DISCUSSION

In this first survey quantifying research activities in HM, respondents from 28 programs shared important insights into research activities at their institutions. Although our sample size was small, substantial variation in the size, composition, and structure of research programs in HM among respondents was observed. For example, few respondents indicated the availability of training programs for research in HM at their institutions. Similarly, among faculty who focused mainly on research, variation in funding streams and effort protection was observed. A preponderance of midcareer faculty with a range of funding sources, including NIH, AHRQ, VHA, CMS, and CDC was reported. Collectively, these data not only provide a unique glimpse into the state of research in HM but also help establish a baseline of the status of the field at large.

Some findings of our study are intuitive given our sampling strategy and the types of programs that responded. For example, the fact that most respondents for research programs represented university-based or affiliated institutions is expected given the tripartite academic mission. However, even within our sample of highly motivated programs, some findings are surprising and merit further exploration. For example, the observation that some respondents identified HM investigators within their program with <25% in intra- or extramural funding was unexpected. On the other extreme, we were surprised to find that three programs reported >$5 million in research funding. Understanding whether specific factors, such as the availability of experienced mentors within and outside departments or assistance from support staff (eg, statisticians and project managers), are associated with success and funding within these programs are important questions to answer. By focusing on these issues, we will be well poised as a field to understand what works, what does not work, and why.

Likewise, the finding that few programs within our sample offer formal training in the form of fellowships to research investigators represents an improvement opportunity. A pipeline for growing investigators is critical for the specialty that is HM. Notably, this call is not new; rather, previous investigators have highlighted the importance of developing academically oriented hospitalists for the future of the field.⁵ The implementation of faculty scholarship development programs has improved the scholarly output, mentoring activities, and succession planning of academics within HM.^6,7 Conversely, lack of adequate mentorship and support for academic activities remains a challenge and as a factor associated with the failure to produce academic work.⁸ Without a cadre of investigators asking critical questions related to care delivery, the legitimacy of our field may be threatened.

While extrapolating to the field is difficult given the small number of our respondents, highlighting the progress that has been made is important. For example, while misalignment between funding and clinical and research mission persists, our survey found that several programs have been successful in securing extramural funding for their investigators. Additionally, internal funding for QI work appears to be increasing, with hospitalists receiving dedicated effort for much of this work. Innovation in how best to support and develop these types of efforts have also emerged. For example, the University of Michigan Specialist Hospitalist Allied Research Program offers dedicated effort and funding for hospitalists tackling projects germane to HM (eg, ordering of blood cultures for febrile inpatients) that overlap with subspecialists (eg, infectious diseases).⁹ Thus, hospitalists are linked with other specialties in the development of research agendas and academic products. Similarly, the launch of the HOMERUN network, a coalition of investigators who bridge health systems to study problems central to HM, has helped usher in a new era of research opportunities in the specialty.¹⁰ Fundamentally, the culture of HM has begun to place an emphasis on academic and scholarly productivity in addition to clinical prowess.^11-13 Increased support and funding for training programs geared toward innovation and research in HM is needed to continue this mission. The Society for General Internal Medicine, American College of Physicians, and SHM have important roles to play as the largest professional organizations for generalists in this respect. Support for research, QI, and investigators in HM remains an urgent and largely unmet need.

Our study has limitations. First, our response rate was low at 28% but is consistent with the response rates of other surveys of physician groups.¹⁴ Caution in making inferences to the field at large is necessary given the potential for selection and nonresponse bias. However, we expect that respondents are likely biased toward programs actively conducting research and engaged in QI, thus better reflecting the state of these activities in HM. Second, given that we did not ask for any identifying information, we have no way of establishing the accuracy of the data provided by respondents. However, we have no reason to believe that responses would be altered in a systematic fashion. Future studies that link our findings to publicly available data (eg, databases of active grants and funding) might be useful. Third, while our survey instrument was created and internally validated by hospitalist researchers, its lack of external validation could limit findings. Finally, our results vary on the basis of how respondents answered questions related to effort and time allocation given that these measures differ across programs.

In summary, the findings from this study highlight substantial variations in the number, training, and funding of research faculty across HM programs. Understanding the factors behind the success of some programs and the failures of others appears important in informing and growing the research in the field. Future studies that aim to expand survey participation, raise the awareness of the state of research in HM, and identify barriers and facilitators to academic success in HM are needed.

 

Disclosures

Dr. Chopra discloses grant funding from the Agency for Healthcare Research and Quality (AHRQ), VA Health Services and Research Department, and Centers for Disease Control. Dr. Jones discloses grant funding from AHRQ. All other authors disclose no conflicts of interest.

Almost all specialties in internal medicine have a sound scientific research base through which clinical practice is informed.¹ For the field of Hospital Medicine (HM), this evidence has largely comprised research generated from fields outside of the specialty. The need to develop, invest, and grow investigators in hospital-based medicine remains unmet as HM and its footprint in hospital systems continue to grow.^2,3

Despite this fact, little is known about the current state of research in HM. A 2014 survey of the members of the Society of Hospital Medicine (SHM) found that research output across the field of HM, as measured on the basis of peer-reviewed publications, was growing.⁴ Since then, however, the numbers of individuals engaged in research activities, their background and training, publication output, or funding sources have not been quantified. Similarly, little is known about which institutions support the development of junior investigators (ie, HM research fellowships), how these programs are funded, and whether or not matriculants enter the field as investigators. These gaps must be measured, evaluated, and ideally addressed through strategic policy and funding initiatives to advance the state of science within HM.

Members of the SHM Research Committee developed, designed, and deployed a survey to improve the understanding of the state of research in HM. In this study, we aimed to establish the baseline of research in HM to enable the measurement of progress through periodic waves of data collection. Specifically, we sought to quantify and describe the characteristics of existing research programs, the sources and types of funding, the number and background of faculty, and the availability of resources for training researchers in HM.

 

METHODS

Study Setting and Participants

Given that no defined list, database, or external resource that identifies research programs and contacts in HM exists, we began by creating a strategy to identify and sample adult HM programs and their leaders engaged in research activity. We iteratively developed a two-step approach to maximize inclusivity. First, we partnered with SHM to identify programs and leaders actively engaging in research activities. SHM is the largest professional organization within HM and maintains an extensive membership database that includes the titles, e-mail addresses, and affiliations of hospitalists in the United States, including academic and nonacademic sites. This list was manually scanned, and the leaders of academic and research programs in adult HM were identified by examining their titles (eg, Division Chief, Research Lead, etc.) and academic affiliations. During this step, members of the committee noticed that certain key individuals were either missing, no longer occupying their role/title, or had been replaced by others. Therefore, we performed a second step and asked the members of the SHM Research Committee to identify academic and research leaders by using current personal contacts, publication history, and social networks. We asked members to identify individuals and programs that had received grant funding, were actively presenting research at SHM (or other major national venues), and/or were producing peer-reviewed publications related to HM. These programs were purposefully chosen (ie, over HM programs known for clinical activities) to create an enriched sample of those engaged in research in HM. The research committee performed the “second pass” to ensure that established investigators who may not be accurately captured within the SHM database were included to maximize yield for the survey. Finally, these two sources were merged to ensure the absence of duplicate contacts and the identification of a primary respondent for each affiliate. As a result, a convenience sample of 100 programs and corresponding individuals was compiled for the purposes of this survey.

Survey Development

A workgroup within the SHM Research Committee was tasked to create a survey that would achieve four distinct goals: (1) identify institutions currently engaging in hospital-based research; (2) define the characteristics, including sources of research funding, training opportunities, criteria for promotion, and grant support, of research programs within institutions; (3) understand the prevalence of research fellowship programs, including size, training curricula, and funding sources; and (4) evaluate the productivity and funding sources of HM investigators at each site.

Survey questions that target each of these domains were drafted by the workgroup. Questions were pretested with colleagues outside the workgroup focused on this project (ie, from the main research committee). The instrument was refined and edited to improve the readability and clarity of questions on the basis of the feedback obtained through the iterative process. The revised instrument was then programmed into an online survey administration tool (SurveyMonkey^®) to facilitate electronic dissemination. Finally, the members of the workgroup tested the online survey to ensure functionality. No identifiable information was collected from respondents, and no monetary incentive was offered for the completion of the survey. An invitation to participate in the survey was sent via e-mail to each of the program contacts identified.

 

Statistical Analysis

Descriptive statistics, including proportions, means, and percentages, were used to tabulate results. All analyses were conducted using Stata 13 MP/SE (StataCorp, College Station, Texas).

Ethical and Regulatory Considerations

The study was reviewed and deemed exempt from regulation by the University of Michigan Institutional Review Board (HUM000138628).

RESULTS

General Characteristics of Research Programs and Faculty

Out of 100 program contacts, 28 (representing 1,586 faculty members) responded and were included in the survey (program response rate = 28%). When comparing programs that did respond with those that did not, a greater proportion of programs in university settings were noted among respondents (79% vs 21%). Respondents represented programs from all regions of the United States, with most representing university-based (79%), university-affiliated (14%) or Veterans Health Administration (VHA; 11%) programs. Most respondents were in leadership roles, including division chiefs (32%), research directors/leads (21%), section chiefs (18%), and related titles, such as program director. Respondents indicated that the total number of faculty members in their programs (including nonclinicians and advance practice providers) varied from eight to 152 (mean [SD] = 57 [36]) members, with physicians representing the majority of faculty members (Table 1).

Among the 1,586 faculty members within the 28 programs, respondents identified 192 faculty members (12%) as currently receiving extra- or intramural support for research activities. Of these faculty, over half (58%) received <25% of effort from intra or extramural sources, and 28 (15%) and 52 (27%) faculty members received 25%-50% or >50% of support for their effort, respectively. The number of investigators who received funding across programs ranged from 0 to 28 faculty members. Compared with the 192 funded investigators, respondents indicated that a larger number of faculty in their programs (n = 656 or 41%) were involved in local quality improvement (QI) efforts. Of the 656 faculty members involved in QI efforts, 241 individuals (37%) were internally funded and received protected time/effort for their work.

Key Attributes of Research Programs

In the evaluation of the amount of total grant funding, respondents from 17 programs indicated that they received $500,000 in annual extra and intramural funding, and those from three programs stated that they received $500,000 to $999,999 in funding. Five respondents indicated that their programs currently received $1 million to $5 million in grant funding, and three reported >$5 million in research support. The sources of research funding included several divisions within the National Institute of Health (NIH, 12 programs), Agency for Healthcare Research and Quality (AHRQ, four programs), foundations (four programs), and internal grants (six programs). Additionally, six programs indicated “other” sources of funding that included the VHA, Patient-Centered Outcomes Research Institute (PCORI), Centers for Medicare and Medicaid Services, Centers for Disease Control (CDC), and industry sources.

A range of grants, including career development awards (11 programs); small grants, such as R21 and R03s (eight programs); R-level grants, including VA merit awards (five programs); program series grants, such as P and U grants (five programs), and foundation grants (eight programs), were reported as types of awards. Respondents from 16 programs indicated that they provided internal pilot grants. Amounts for such grants ranged from <$50,000 (14 programs) to $50,000-$100,000 (two programs).

 

Research Fellowship Programs/Training Programs

Only five of the 28 surveyed programs indicated that they currently had a research training or fellowship program for developing hospitalist investigators. The age of these programs varied from <1 year to 10 years. Three of the five programs stated that they had two fellows per year, and two stated they had spots for one trainee annually. All respondents indicated that fellows received training on study design, research methods, quantitative (eg, large database and secondary analyses) and qualitative data analysis. In addition, two programs included training in systematic review and meta-analyses, and three included focused courses on healthcare policy. Four of the five programs included training in QI tools, such as LEAN and Six Sigma. Funding for four of the five fellowship programs came from internal sources (eg, department and CTSA). However, two programs added they received some support from extramural funding and philanthropy. Following training, respondents from programs indicated that the majority of their graduates (60%) went on to hybrid research/QI roles (50/50 research/clinical effort), whereas 40% obtained dedicated research investigator (80/20) positions (Table 2).

The 23 institutions without research training programs cited that the most important barrier for establishing such programs was lack of funding (12 programs) and the lack of a pipeline of hospitalists seeking such training (six programs). However, 15 programs indicated that opportunities for hospitalists to gain research training in the form of courses were available internally (eg, courses in the department or medical school) or externally (eg, School of Public Health). Seven programs indicated that they were planning to start a HM research fellowship within the next five years.

Research Faculty

Among the 28 respondents, 15 stated that they have faculty members who conduct research as their main professional activity (ie, >50% effort). The number of faculty members in each program in such roles varied from one to 10. Respondents indicated that faculty members in this category were most often midcareer assistant or associate professors with few full professors. All programs indicated that scholarship in the form of peer-reviewed publications was required for the promotion of faculty. Faculty members who performed research as their main activity had all received formal fellowship training and consequently had dual degrees (MD with MPH or MD, with MSc being the two most common combinations). With respect to clinical activities, most respondents indicated that research faculty spent 10% to 49% of their effort on clinical work. However, five respondents indicated that research faculty had <10% effort on clinical duties (Table 3).

Eleven respondents (39%) identified the main focus of faculty as health service research, where four (14%) identified their main focus as clinical trials. Regardless of funding status, all respondents stated that their faculty were interested in studying quality and process improvement efforts (eg, transitions or readmissions, n = 19), patient safety initiatives (eg, hospital-acquired complications, n = 17), and disease-specific areas (eg, thrombosis, n = 15).

In terms of research output, 12 respondents stated that their research/QI faculty collectively published 11-50 peer-reviewed papers during the academic year, and 10 programs indicated that their faculty published 0-10 papers per year. Only three programs reported that their faculty collectively published 50-99 peer-reviewed papers per year. With respect to abstract presentations at national conferences, 13 programs indicated that they presented 0-10 abstracts, and 12 indicated that they presented 11-50.

 

DISCUSSION

In this first survey quantifying research activities in HM, respondents from 28 programs shared important insights into research activities at their institutions. Although our sample size was small, substantial variation in the size, composition, and structure of research programs in HM among respondents was observed. For example, few respondents indicated the availability of training programs for research in HM at their institutions. Similarly, among faculty who focused mainly on research, variation in funding streams and effort protection was observed. A preponderance of midcareer faculty with a range of funding sources, including NIH, AHRQ, VHA, CMS, and CDC was reported. Collectively, these data not only provide a unique glimpse into the state of research in HM but also help establish a baseline of the status of the field at large.

Some findings of our study are intuitive given our sampling strategy and the types of programs that responded. For example, the fact that most respondents for research programs represented university-based or affiliated institutions is expected given the tripartite academic mission. However, even within our sample of highly motivated programs, some findings are surprising and merit further exploration. For example, the observation that some respondents identified HM investigators within their program with <25% in intra- or extramural funding was unexpected. On the other extreme, we were surprised to find that three programs reported >$5 million in research funding. Understanding whether specific factors, such as the availability of experienced mentors within and outside departments or assistance from support staff (eg, statisticians and project managers), are associated with success and funding within these programs are important questions to answer. By focusing on these issues, we will be well poised as a field to understand what works, what does not work, and why.

Likewise, the finding that few programs within our sample offer formal training in the form of fellowships to research investigators represents an improvement opportunity. A pipeline for growing investigators is critical for the specialty that is HM. Notably, this call is not new; rather, previous investigators have highlighted the importance of developing academically oriented hospitalists for the future of the field.⁵ The implementation of faculty scholarship development programs has improved the scholarly output, mentoring activities, and succession planning of academics within HM.^6,7 Conversely, lack of adequate mentorship and support for academic activities remains a challenge and as a factor associated with the failure to produce academic work.⁸ Without a cadre of investigators asking critical questions related to care delivery, the legitimacy of our field may be threatened.

While extrapolating to the field is difficult given the small number of our respondents, highlighting the progress that has been made is important. For example, while misalignment between funding and clinical and research mission persists, our survey found that several programs have been successful in securing extramural funding for their investigators. Additionally, internal funding for QI work appears to be increasing, with hospitalists receiving dedicated effort for much of this work. Innovation in how best to support and develop these types of efforts have also emerged. For example, the University of Michigan Specialist Hospitalist Allied Research Program offers dedicated effort and funding for hospitalists tackling projects germane to HM (eg, ordering of blood cultures for febrile inpatients) that overlap with subspecialists (eg, infectious diseases).⁹ Thus, hospitalists are linked with other specialties in the development of research agendas and academic products. Similarly, the launch of the HOMERUN network, a coalition of investigators who bridge health systems to study problems central to HM, has helped usher in a new era of research opportunities in the specialty.¹⁰ Fundamentally, the culture of HM has begun to place an emphasis on academic and scholarly productivity in addition to clinical prowess.^11-13 Increased support and funding for training programs geared toward innovation and research in HM is needed to continue this mission. The Society for General Internal Medicine, American College of Physicians, and SHM have important roles to play as the largest professional organizations for generalists in this respect. Support for research, QI, and investigators in HM remains an urgent and largely unmet need.

Our study has limitations. First, our response rate was low at 28% but is consistent with the response rates of other surveys of physician groups.¹⁴ Caution in making inferences to the field at large is necessary given the potential for selection and nonresponse bias. However, we expect that respondents are likely biased toward programs actively conducting research and engaged in QI, thus better reflecting the state of these activities in HM. Second, given that we did not ask for any identifying information, we have no way of establishing the accuracy of the data provided by respondents. However, we have no reason to believe that responses would be altered in a systematic fashion. Future studies that link our findings to publicly available data (eg, databases of active grants and funding) might be useful. Third, while our survey instrument was created and internally validated by hospitalist researchers, its lack of external validation could limit findings. Finally, our results vary on the basis of how respondents answered questions related to effort and time allocation given that these measures differ across programs.

In summary, the findings from this study highlight substantial variations in the number, training, and funding of research faculty across HM programs. Understanding the factors behind the success of some programs and the failures of others appears important in informing and growing the research in the field. Future studies that aim to expand survey participation, raise the awareness of the state of research in HM, and identify barriers and facilitators to academic success in HM are needed.

 

Disclosures

Dr. Chopra discloses grant funding from the Agency for Healthcare Research and Quality (AHRQ), VA Health Services and Research Department, and Centers for Disease Control. Dr. Jones discloses grant funding from AHRQ. All other authors disclose no conflicts of interest.

Acute respiratory illnesses (ARIs), including acute exacerbations of asthma, croup, pneumonia, and bronchiolitis, are among the most common illnesses in childhood.¹ Although most ARIs can be managed in the outpatient setting, hospitalization is common with respiratory illnesses accounting for >425,000 hospitalizations annually.¹ Pneumonia, asthma, and bronchiolitis each rank among the top five reasons for pediatric hospitalization in the United States.¹ Successful efforts to prevent or mitigate the severity of ARIs could have a major impact on child health.

Exposure to secondhand smoke (SHS) is a preventable risk factor for ARI in children, particularly when there is regular exposure in the home.² Chronic exposure to SHS impacts systemic inflammation by suppressing serum interferon-gamma,³ which can lead to increased susceptibility to viral and bacterial infections,⁴ and increasing Th2 (atopic) cytokine expression, which is associated with asthma.⁵ SHS exposure in children has also been linked to diminished lung function.⁶ As a result, SHS exposure is associated with increased ARI susceptibility and severity in children.^7-10

Much research has focused on the clinical impact of SHS exposure on respiratory health in children, but little is known about the impact on patient-reported outcomes, such as health-related quality of life (HRQOL). Patient-reported outcomes help provide a comprehensive evaluation of the effectiveness of healthcare delivery systems. These outcomes are increasingly used by health service researchers to better understand patient and caregiver perspectives.¹¹ Given the known associations between SHS exposure and ARI morbidity, we postulated that regular SHS exposure would also impact HRQOL in children. In this study, we assessed the relationship between SHS exposure and HRQOL within a large, multicenter, prospective cohort of children presenting to the emergency department (ED) and/or hospital with ARI.

 

METHODS

Study Population

This study was nested within the Pediatric Respiratory Illness Measurement System (PRIMES) study, a prospective cohort study of children with ARI in the ED and inpatient settings at five tertiary care children’s hospitals within the Pediatric Research in Inpatient Settings Network in Colorado, Pennsylvania, Tennessee, Texas, and Washington. Eligible children were two weeks to 16 years of age hospitalized after presenting to the ED with a primary diagnosis of asthma, croup, bronchiolitis, or pneumonia between July 1, 2014 and June 30, 2016. Because of an anticipated low frequency of croup hospitalizations, we also included children presenting to the ED and then discharged to home with this diagnosis. Children were assigned to a PRIMES diagnosis group based on their final discharge diagnosis. If there was a discrepancy between admission and discharge diagnoses, the discharge diagnosis was used. If a child had more than one discharge diagnosis for a PRIMES condition (eg, acute asthma and pneumonia), we chose the PRIMES condition with the lowest total enrollments overall. If the final discharge diagnosis was not a PRIMES condition, the case was excluded from further analysis. Patients with immunodeficiency, cystic fibrosis, a history of prematurity <32 weeks, chronic neuromuscular disease, cardiovascular disease, pulmonary diseases (other than asthma), and moderate to severe developmental delay were also excluded. Children admitted to intensive care were eligible only if they were transferred to an acute care ward <72 hours following admission. A survey was administered at the time of enrollment that collected information on SHS exposure, HRQOL, healthcare utilization, and demographics. All study procedures were reviewed and approved by the institutional review boards at each of the participating hospitals.

SECONDHAND SMOKE EXPOSURE

To ascertain SHS exposure, we asked caregivers, “How many persons living in the child’s home smoke?” Responses were dichotomized into non-SHS exposed (0 smokers) and SHS exposed (≥1 smokers). Children with missing data on home SHS exposure were excluded.

Health-Related Quality of Life Outcomes

We estimated HRQOL using the Pediatric Quality of Life (PedsQL^TM) 4.0 Generic Core and Infant Scales. The PedsQL instruments are validated, population HRQOL measures that evaluate the physical, mental, emotional, and social functioning of children two to 18 years old based on self- or caregiver-proxy report.^12-15 These instruments have also shown responsiveness as well as construct and predictive validity in hospitalized children.¹¹ For this study, we focused on the PedsQL physical functioning subscale, which assesses for problems with physical activities (eg, sports activity or exercise, low energy, and hurts or aches) on a five-point Likert scale (never to almost always a problem). Scores range from 0 to 100 with higher scores indicating a better HRQOL. The reported minimal clinically important difference (MCID), defined as the smallest difference in which individuals would perceive a benefit or would necessitate a change in management, for this scale is 4.5 points.^16,17

Children >8 years old were invited to complete the self-report version of the PedsQL. For children <8 years old, and for older children who were unable to complete them, surveys were completed by a parent or legal guardian. Respondents were asked to assess perceptions of their (or their child’s) HRQOL during periods of baseline health (the child’s usual state of health in the month preceding the current illness) and during the acute illness (the child’s state of health at the time of admission) as SHS exposure may influence perceptions of general health and/or contribute to worse outcomes during periods of acute illness.

 

Covariates collected at the time of enrollment included sociodemographics (child age, gender, race/ethnicity, and caregiver education), and healthcare utilization (caregiver-reported patient visits to a healthcare provider in the preceding six months). Insurance status and level of medical complexity (using the Pediatric Medical Complexity Algorithm)¹⁸ were obtained using the Pediatric Hospital Information System database, an administrative database containing clinical and resource utilization data from >45 children’s hospitals in the United States including all of the PRIMES study hospitals.¹³

Analysis

Descriptive statistics included frequency (%) and mean (standard deviation). Bivariate comparisons according to SHS exposure status were analyzed using chi-squared tests for categorical variables and analysis of variance for continuous variables. Multivariable linear mixed regression models were used to examine associations between home SHS exposure and HRQOL for baseline health and during admission, overall and stratified by diagnosis. Covariates in each model included age, sex, race/ethnicity, caregiver education, and healthcare visits in the preceding six months. We also included a hospital random effect to account for clustering of patients within hospitals and used robust standard errors for inference.

In a secondary analysis to explore potential dose-response effects of SHS exposure, we examined associations between an ordinal exposure variable (0 smokers, 1 smoker, ≥2 smokers) and HRQOL for baseline health and during admission for the acute illness. Because of sample size limitations, diagnosis-specific analyses examining dose-response effects were not conducted.

RESULTS

Study Population

Of the 2,334 children enrolled in the PRIMES study, 25 (1%) respondents did not report on home SHS exposure and were excluded, yielding a final study population of 2,309 children, of whom 728 (32%) had reported home SHS exposure. The study population included 664 children with asthma (mean age seven years [3.5]; 38% with home SHS exposure), 740 with bronchiolitis (mean age 0.7 years [0.5]; 32% with home SHS exposure), 342 with croup (mean age 1.7 [1.1]; 25% with home SHS exposure), and 563 with pneumonia (mean age 4.4 [3.8]; 27% with home SHS exposure; Table 1). Compared with non-SHS-exposed children, those with home SHS exposure tend to be slightly older (3.9 vs 3.4 years, P = .01), more likely to be non-Hispanic Black (29% vs 19%, P < .001), to have a chronic condition (52% vs 41%, P < .001), to come from a household where caregiver(s) did not graduate from college (45% vs 29%, P < .001), and to have public insurance (73% vs 49%, P < .001).

Home SHS Exposure and Health-related Quality of Life

The overall mean HRQOL score for baseline health was 83 (15), with a range across diagnoses of 82 to 87. Compared with non-SHS-exposed children, children with home SHS exposure had a lower mean HRQOL score for baseline health (adjusted mean difference –3.04 [95% CI -4.34, –1.74]). In analyses stratified by diagnosis, baseline health scores were lower for SHS-exposed children for all four conditions, but differences were statistically significant only for bronchiolitis (adjusted mean difference –2.94 [–5.0, –0.89]) and pneumonia (adjusted mean value –4.13 [–6.82, –1.44]; Table 2); none of these differences met the MCID threshold.

 

The overall mean HRQOL score at the time of admission was 56 (23), with a range across diagnoses of 49 to 61, with lower scores noted among SHS-exposed children compared with non-SHS-exposed children (adjusted mean difference –2.16 [–4.22, –0.10]). Similar to scores representing baseline health, admission scores were lower across all four conditions for SHS-exposed children. Only children with croup, however, had significantly lower admission scores that also met the MCID threshold (adjusted mean difference –5.71 [–10.67, –0.75]; Table 2).

To assess for potential dose-response effects of SHS exposure on HRQOL, we stratified SHS-exposed children into those with one smoker in the home (n = 513) and those with ≥2 smokers in the home (n = 215). Compared with non-SHS-exposed children, both HRQOL scores (baseline health and admission) were lower for SHS-exposed children. Consistent with a dose-response association, scores were lowest for children with ≥2 smokers in the home, both at baseline health (adjusted mean difference –3.92 [–6.03, –1.81]) and on admission (adjusted mean difference –3.67 [–6.98, –0.36]; Table 3).

DISCUSSION

Within a multicenter cohort of 2,309 children hospitalized with ARI, we noted significantly lower HRQOL scores among children exposed to SHS in the home as compared with nonexposed children. Differences were greatest for children living with ≥2 smokers in the home. In analyses stratified by diagnosis, differences in baseline health HRQOL scores were greatest for children with bronchiolitis and pneumonia. Differences in acute illness scores were greatest for children with croup.¹⁶

Our study provides evidence for acute and chronic impacts of SHS on HRQOL in children hospitalized with ARI. Although several studies have linked SHS exposure to reduced HRQOL in adults,^19,20 few similar studies have been conducted in children. Nonetheless, a wealth of studies have documented the negative impact of SHS exposure on clinical outcomes among children with ARI.^8,10,21-23 Our findings that home SHS exposure was associated with reduced HRQOL among our cohort of children with ARI are therefore consistent with related findings in adults and children. The observation that the effects of SHS exposure on HRQOL were greatest among children living with ≥2 smokers provides further evidence of a potential causal link between regular SHS exposure and HRQOL.

Although the magnitude and significance of associations between SHS exposure and HRQOL varied for each of the four diagnoses for baseline health and the acute illness, it is important to note that the point estimates for the adjusted mean differences were uniformly lower for the SHS-exposed children in each subgroup. Even so, only acute illness scores for croup exceeded the MCID threshold.¹⁶ Croup is the only included condition of the upper airway and is characterized by laryngotracheal inflammation leading to the typical cough and, in moderate to severe cases, stridor. Given that chronic SHS exposure induces a proinflammatory state,³ it is possible that SHS-exposed children with croup had more severe illness compared with nonexposed children with croup resulting in lower HRQOL scores on admission. Further, perceived differences in illness severity and HRQOL may be more readily apparent in children with croup (eg, stridor at rest vs intermittent or no stridor) as compared with children with lower respiratory tract diseases.

Of the four included diagnoses, the link between SHS exposure and asthma outcomes has been most studied. Prior work has demonstrated more frequent and severe acute exacerbations, as well as worse long-term lung function among SHS-exposed children as compared with nonexposed children.^22-24 It was, therefore, surprising that our study failed to demonstrate associations between SHS exposure and HRQOL among children with asthma. Reasons for this finding are unclear. One hypothesis is that caregivers of SHS-exposed children with asthma may be more aware of the impacts of SHS exposure on respiratory health (through prior education) and, thus, more likely to modify their smoking behaviors, or for their children to be on daily asthma controller therapy. Alternatively, caregivers of children with asthma may be more likely to underreport home SHS exposure. Thirty-eight percent of children with asthma, however, were classified as SHS-exposed. This percentage was greater than the other three conditions studied (25%-32%), suggesting that differential bias in underreporting was minimal. Given that children with asthma were older, on average, than children with the other three conditions, it may also be that these children spent more time in smoke-free environments (eg, school).

Nearly one-third of children in our study were exposed to SHS in the home. This is similar to the prevalence of exposure in other studies conducted among hospitalized children^8,10,21,25 but higher than the national prevalence of home SHS exposure among children in the United States.²⁶ Thus, hospitalized children represent a particularly vulnerable population and an important target for interventions aiming to reduce exposure to SHS. Although longitudinal interventions are likely necessary to affect long-term success, hospitalization for ARI may serve as a powerful teachable moment to begin cessation efforts. Hospitalization also offers time beyond a typical primary care outpatient encounter to focus on cessation counseling and may be the only opportunity to engage in counseling activities for some families with limited time or access. Further, prior studies have demonstrated both the feasibility and the effectiveness of smoking cessation interventions in hospitalized children.^27-30 Unfortunately, however, SHS exposure is often not documented at the time of hospitalization, and many opportunities to intervene are missed.^25,31 Thus, there is a need for improved strategies to reliably identify and intervene on SHS-exposed children in the hospital setting.

These findings should be considered in the context of several limitations. The observational nature of our study raises the potential for confounding, specifically with regard to socioeconomic status, as this is associated with both SHS exposure and lower HRQOL. Our modeling approach attempted to control for several factors associated with socioeconomic status, including caregiver education and insurance coverage, but there is potential for residual confounding. No single question is sufficient to fully assess SHS exposure as the intensity of home SHS exposure likely varies widely, and some children may be exposed to SHS outside of the home environment.³² The home, however, is often the most likely source of regular SHS exposure,^33,34 especially among young children (our cohort’s mean age was 3.6 years). Misclassification of SHS exposure is also possible due to underreporting of smoking.^35,36 As a result, some children regularly exposed to SHS may have been misclassified as nonexposed, and the observed associations between SHS exposure and HRQOL may be underestimated. Confirming our study’s findings using objective assessments of SHS exposure, such as cotinine, are warranted. Given the young age of our cohort, the PedsQL surveys were completed by the parent or legal guardian only in >90% of the enrolled subjects, and caregiver perceptions may not accurately reflect the child’s perceptions. Prior work, however, has demonstrated the validity of parent-proxy reporting of the PedsQL, including correlation with child self-report.³⁷ In our study, correlation between child and caregiver reporting (when available) was also very good (r = 0.72, 95% CI 0.64, 0.77). It is also possible that the timing of the HRQOL assessments (on admission) may have biased perceptions of baseline HRQOL, although we anticipate any bias would likely be nondifferential between SHS-exposed and nonexposed children and across diagnoses.

Nearly one-third of children in our study were exposed to SHS exposure in the home, and SHS exposure was associated with lower HRQOL for baseline health and during acute illness, providing further evidence of the dangers of SHS. Much work is needed in order to eliminate the impact of SHS on child health and families of children hospitalized for respiratory illness should be considered a priority population for smoking cessation efforts.

 

Acknowledgment

The authors wish to acknowledge the efforts of PRIS-PRIMES study team. The authors also wish to thank the children and families who consented to be a part of the PRIMES study.

Disclosures

The authors have no conflicts of interest relevant to this article to disclose.

Funding

This study was supported by NIH-NHLBI 1R01HL121067 to RMS.

Acute respiratory illnesses (ARIs), including acute exacerbations of asthma, croup, pneumonia, and bronchiolitis, are among the most common illnesses in childhood.¹ Although most ARIs can be managed in the outpatient setting, hospitalization is common with respiratory illnesses accounting for >425,000 hospitalizations annually.¹ Pneumonia, asthma, and bronchiolitis each rank among the top five reasons for pediatric hospitalization in the United States.¹ Successful efforts to prevent or mitigate the severity of ARIs could have a major impact on child health.

Exposure to secondhand smoke (SHS) is a preventable risk factor for ARI in children, particularly when there is regular exposure in the home.² Chronic exposure to SHS impacts systemic inflammation by suppressing serum interferon-gamma,³ which can lead to increased susceptibility to viral and bacterial infections,⁴ and increasing Th2 (atopic) cytokine expression, which is associated with asthma.⁵ SHS exposure in children has also been linked to diminished lung function.⁶ As a result, SHS exposure is associated with increased ARI susceptibility and severity in children.^7-10

Much research has focused on the clinical impact of SHS exposure on respiratory health in children, but little is known about the impact on patient-reported outcomes, such as health-related quality of life (HRQOL). Patient-reported outcomes help provide a comprehensive evaluation of the effectiveness of healthcare delivery systems. These outcomes are increasingly used by health service researchers to better understand patient and caregiver perspectives.¹¹ Given the known associations between SHS exposure and ARI morbidity, we postulated that regular SHS exposure would also impact HRQOL in children. In this study, we assessed the relationship between SHS exposure and HRQOL within a large, multicenter, prospective cohort of children presenting to the emergency department (ED) and/or hospital with ARI.

 

METHODS

Study Population

This study was nested within the Pediatric Respiratory Illness Measurement System (PRIMES) study, a prospective cohort study of children with ARI in the ED and inpatient settings at five tertiary care children’s hospitals within the Pediatric Research in Inpatient Settings Network in Colorado, Pennsylvania, Tennessee, Texas, and Washington. Eligible children were two weeks to 16 years of age hospitalized after presenting to the ED with a primary diagnosis of asthma, croup, bronchiolitis, or pneumonia between July 1, 2014 and June 30, 2016. Because of an anticipated low frequency of croup hospitalizations, we also included children presenting to the ED and then discharged to home with this diagnosis. Children were assigned to a PRIMES diagnosis group based on their final discharge diagnosis. If there was a discrepancy between admission and discharge diagnoses, the discharge diagnosis was used. If a child had more than one discharge diagnosis for a PRIMES condition (eg, acute asthma and pneumonia), we chose the PRIMES condition with the lowest total enrollments overall. If the final discharge diagnosis was not a PRIMES condition, the case was excluded from further analysis. Patients with immunodeficiency, cystic fibrosis, a history of prematurity <32 weeks, chronic neuromuscular disease, cardiovascular disease, pulmonary diseases (other than asthma), and moderate to severe developmental delay were also excluded. Children admitted to intensive care were eligible only if they were transferred to an acute care ward <72 hours following admission. A survey was administered at the time of enrollment that collected information on SHS exposure, HRQOL, healthcare utilization, and demographics. All study procedures were reviewed and approved by the institutional review boards at each of the participating hospitals.

SECONDHAND SMOKE EXPOSURE

To ascertain SHS exposure, we asked caregivers, “How many persons living in the child’s home smoke?” Responses were dichotomized into non-SHS exposed (0 smokers) and SHS exposed (≥1 smokers). Children with missing data on home SHS exposure were excluded.

Health-Related Quality of Life Outcomes

We estimated HRQOL using the Pediatric Quality of Life (PedsQL^TM) 4.0 Generic Core and Infant Scales. The PedsQL instruments are validated, population HRQOL measures that evaluate the physical, mental, emotional, and social functioning of children two to 18 years old based on self- or caregiver-proxy report.^12-15 These instruments have also shown responsiveness as well as construct and predictive validity in hospitalized children.¹¹ For this study, we focused on the PedsQL physical functioning subscale, which assesses for problems with physical activities (eg, sports activity or exercise, low energy, and hurts or aches) on a five-point Likert scale (never to almost always a problem). Scores range from 0 to 100 with higher scores indicating a better HRQOL. The reported minimal clinically important difference (MCID), defined as the smallest difference in which individuals would perceive a benefit or would necessitate a change in management, for this scale is 4.5 points.^16,17

Children >8 years old were invited to complete the self-report version of the PedsQL. For children <8 years old, and for older children who were unable to complete them, surveys were completed by a parent or legal guardian. Respondents were asked to assess perceptions of their (or their child’s) HRQOL during periods of baseline health (the child’s usual state of health in the month preceding the current illness) and during the acute illness (the child’s state of health at the time of admission) as SHS exposure may influence perceptions of general health and/or contribute to worse outcomes during periods of acute illness.

 

Covariates collected at the time of enrollment included sociodemographics (child age, gender, race/ethnicity, and caregiver education), and healthcare utilization (caregiver-reported patient visits to a healthcare provider in the preceding six months). Insurance status and level of medical complexity (using the Pediatric Medical Complexity Algorithm)¹⁸ were obtained using the Pediatric Hospital Information System database, an administrative database containing clinical and resource utilization data from >45 children’s hospitals in the United States including all of the PRIMES study hospitals.¹³

Analysis

Descriptive statistics included frequency (%) and mean (standard deviation). Bivariate comparisons according to SHS exposure status were analyzed using chi-squared tests for categorical variables and analysis of variance for continuous variables. Multivariable linear mixed regression models were used to examine associations between home SHS exposure and HRQOL for baseline health and during admission, overall and stratified by diagnosis. Covariates in each model included age, sex, race/ethnicity, caregiver education, and healthcare visits in the preceding six months. We also included a hospital random effect to account for clustering of patients within hospitals and used robust standard errors for inference.

In a secondary analysis to explore potential dose-response effects of SHS exposure, we examined associations between an ordinal exposure variable (0 smokers, 1 smoker, ≥2 smokers) and HRQOL for baseline health and during admission for the acute illness. Because of sample size limitations, diagnosis-specific analyses examining dose-response effects were not conducted.

RESULTS

Study Population

Of the 2,334 children enrolled in the PRIMES study, 25 (1%) respondents did not report on home SHS exposure and were excluded, yielding a final study population of 2,309 children, of whom 728 (32%) had reported home SHS exposure. The study population included 664 children with asthma (mean age seven years [3.5]; 38% with home SHS exposure), 740 with bronchiolitis (mean age 0.7 years [0.5]; 32% with home SHS exposure), 342 with croup (mean age 1.7 [1.1]; 25% with home SHS exposure), and 563 with pneumonia (mean age 4.4 [3.8]; 27% with home SHS exposure; Table 1). Compared with non-SHS-exposed children, those with home SHS exposure tend to be slightly older (3.9 vs 3.4 years, P = .01), more likely to be non-Hispanic Black (29% vs 19%, P < .001), to have a chronic condition (52% vs 41%, P < .001), to come from a household where caregiver(s) did not graduate from college (45% vs 29%, P < .001), and to have public insurance (73% vs 49%, P < .001).

Home SHS Exposure and Health-related Quality of Life

The overall mean HRQOL score for baseline health was 83 (15), with a range across diagnoses of 82 to 87. Compared with non-SHS-exposed children, children with home SHS exposure had a lower mean HRQOL score for baseline health (adjusted mean difference –3.04 [95% CI -4.34, –1.74]). In analyses stratified by diagnosis, baseline health scores were lower for SHS-exposed children for all four conditions, but differences were statistically significant only for bronchiolitis (adjusted mean difference –2.94 [–5.0, –0.89]) and pneumonia (adjusted mean value –4.13 [–6.82, –1.44]; Table 2); none of these differences met the MCID threshold.

 

The overall mean HRQOL score at the time of admission was 56 (23), with a range across diagnoses of 49 to 61, with lower scores noted among SHS-exposed children compared with non-SHS-exposed children (adjusted mean difference –2.16 [–4.22, –0.10]). Similar to scores representing baseline health, admission scores were lower across all four conditions for SHS-exposed children. Only children with croup, however, had significantly lower admission scores that also met the MCID threshold (adjusted mean difference –5.71 [–10.67, –0.75]; Table 2).

To assess for potential dose-response effects of SHS exposure on HRQOL, we stratified SHS-exposed children into those with one smoker in the home (n = 513) and those with ≥2 smokers in the home (n = 215). Compared with non-SHS-exposed children, both HRQOL scores (baseline health and admission) were lower for SHS-exposed children. Consistent with a dose-response association, scores were lowest for children with ≥2 smokers in the home, both at baseline health (adjusted mean difference –3.92 [–6.03, –1.81]) and on admission (adjusted mean difference –3.67 [–6.98, –0.36]; Table 3).

DISCUSSION

Within a multicenter cohort of 2,309 children hospitalized with ARI, we noted significantly lower HRQOL scores among children exposed to SHS in the home as compared with nonexposed children. Differences were greatest for children living with ≥2 smokers in the home. In analyses stratified by diagnosis, differences in baseline health HRQOL scores were greatest for children with bronchiolitis and pneumonia. Differences in acute illness scores were greatest for children with croup.¹⁶

Our study provides evidence for acute and chronic impacts of SHS on HRQOL in children hospitalized with ARI. Although several studies have linked SHS exposure to reduced HRQOL in adults,^19,20 few similar studies have been conducted in children. Nonetheless, a wealth of studies have documented the negative impact of SHS exposure on clinical outcomes among children with ARI.^8,10,21-23 Our findings that home SHS exposure was associated with reduced HRQOL among our cohort of children with ARI are therefore consistent with related findings in adults and children. The observation that the effects of SHS exposure on HRQOL were greatest among children living with ≥2 smokers provides further evidence of a potential causal link between regular SHS exposure and HRQOL.

Although the magnitude and significance of associations between SHS exposure and HRQOL varied for each of the four diagnoses for baseline health and the acute illness, it is important to note that the point estimates for the adjusted mean differences were uniformly lower for the SHS-exposed children in each subgroup. Even so, only acute illness scores for croup exceeded the MCID threshold.¹⁶ Croup is the only included condition of the upper airway and is characterized by laryngotracheal inflammation leading to the typical cough and, in moderate to severe cases, stridor. Given that chronic SHS exposure induces a proinflammatory state,³ it is possible that SHS-exposed children with croup had more severe illness compared with nonexposed children with croup resulting in lower HRQOL scores on admission. Further, perceived differences in illness severity and HRQOL may be more readily apparent in children with croup (eg, stridor at rest vs intermittent or no stridor) as compared with children with lower respiratory tract diseases.

Of the four included diagnoses, the link between SHS exposure and asthma outcomes has been most studied. Prior work has demonstrated more frequent and severe acute exacerbations, as well as worse long-term lung function among SHS-exposed children as compared with nonexposed children.^22-24 It was, therefore, surprising that our study failed to demonstrate associations between SHS exposure and HRQOL among children with asthma. Reasons for this finding are unclear. One hypothesis is that caregivers of SHS-exposed children with asthma may be more aware of the impacts of SHS exposure on respiratory health (through prior education) and, thus, more likely to modify their smoking behaviors, or for their children to be on daily asthma controller therapy. Alternatively, caregivers of children with asthma may be more likely to underreport home SHS exposure. Thirty-eight percent of children with asthma, however, were classified as SHS-exposed. This percentage was greater than the other three conditions studied (25%-32%), suggesting that differential bias in underreporting was minimal. Given that children with asthma were older, on average, than children with the other three conditions, it may also be that these children spent more time in smoke-free environments (eg, school).

Nearly one-third of children in our study were exposed to SHS in the home. This is similar to the prevalence of exposure in other studies conducted among hospitalized children^8,10,21,25 but higher than the national prevalence of home SHS exposure among children in the United States.²⁶ Thus, hospitalized children represent a particularly vulnerable population and an important target for interventions aiming to reduce exposure to SHS. Although longitudinal interventions are likely necessary to affect long-term success, hospitalization for ARI may serve as a powerful teachable moment to begin cessation efforts. Hospitalization also offers time beyond a typical primary care outpatient encounter to focus on cessation counseling and may be the only opportunity to engage in counseling activities for some families with limited time or access. Further, prior studies have demonstrated both the feasibility and the effectiveness of smoking cessation interventions in hospitalized children.^27-30 Unfortunately, however, SHS exposure is often not documented at the time of hospitalization, and many opportunities to intervene are missed.^25,31 Thus, there is a need for improved strategies to reliably identify and intervene on SHS-exposed children in the hospital setting.

These findings should be considered in the context of several limitations. The observational nature of our study raises the potential for confounding, specifically with regard to socioeconomic status, as this is associated with both SHS exposure and lower HRQOL. Our modeling approach attempted to control for several factors associated with socioeconomic status, including caregiver education and insurance coverage, but there is potential for residual confounding. No single question is sufficient to fully assess SHS exposure as the intensity of home SHS exposure likely varies widely, and some children may be exposed to SHS outside of the home environment.³² The home, however, is often the most likely source of regular SHS exposure,^33,34 especially among young children (our cohort’s mean age was 3.6 years). Misclassification of SHS exposure is also possible due to underreporting of smoking.^35,36 As a result, some children regularly exposed to SHS may have been misclassified as nonexposed, and the observed associations between SHS exposure and HRQOL may be underestimated. Confirming our study’s findings using objective assessments of SHS exposure, such as cotinine, are warranted. Given the young age of our cohort, the PedsQL surveys were completed by the parent or legal guardian only in >90% of the enrolled subjects, and caregiver perceptions may not accurately reflect the child’s perceptions. Prior work, however, has demonstrated the validity of parent-proxy reporting of the PedsQL, including correlation with child self-report.³⁷ In our study, correlation between child and caregiver reporting (when available) was also very good (r = 0.72, 95% CI 0.64, 0.77). It is also possible that the timing of the HRQOL assessments (on admission) may have biased perceptions of baseline HRQOL, although we anticipate any bias would likely be nondifferential between SHS-exposed and nonexposed children and across diagnoses.

Nearly one-third of children in our study were exposed to SHS exposure in the home, and SHS exposure was associated with lower HRQOL for baseline health and during acute illness, providing further evidence of the dangers of SHS. Much work is needed in order to eliminate the impact of SHS on child health and families of children hospitalized for respiratory illness should be considered a priority population for smoking cessation efforts.

 

Acknowledgment

The authors wish to acknowledge the efforts of PRIS-PRIMES study team. The authors also wish to thank the children and families who consented to be a part of the PRIMES study.

Disclosures

The authors have no conflicts of interest relevant to this article to disclose.

Funding

This study was supported by NIH-NHLBI 1R01HL121067 to RMS.

Acute respiratory illnesses (ARIs), including acute exacerbations of asthma, croup, pneumonia, and bronchiolitis, are among the most common illnesses in childhood.¹ Although most ARIs can be managed in the outpatient setting, hospitalization is common with respiratory illnesses accounting for >425,000 hospitalizations annually.¹ Pneumonia, asthma, and bronchiolitis each rank among the top five reasons for pediatric hospitalization in the United States.¹ Successful efforts to prevent or mitigate the severity of ARIs could have a major impact on child health.

Exposure to secondhand smoke (SHS) is a preventable risk factor for ARI in children, particularly when there is regular exposure in the home.² Chronic exposure to SHS impacts systemic inflammation by suppressing serum interferon-gamma,³ which can lead to increased susceptibility to viral and bacterial infections,⁴ and increasing Th2 (atopic) cytokine expression, which is associated with asthma.⁵ SHS exposure in children has also been linked to diminished lung function.⁶ As a result, SHS exposure is associated with increased ARI susceptibility and severity in children.^7-10

Much research has focused on the clinical impact of SHS exposure on respiratory health in children, but little is known about the impact on patient-reported outcomes, such as health-related quality of life (HRQOL). Patient-reported outcomes help provide a comprehensive evaluation of the effectiveness of healthcare delivery systems. These outcomes are increasingly used by health service researchers to better understand patient and caregiver perspectives.¹¹ Given the known associations between SHS exposure and ARI morbidity, we postulated that regular SHS exposure would also impact HRQOL in children. In this study, we assessed the relationship between SHS exposure and HRQOL within a large, multicenter, prospective cohort of children presenting to the emergency department (ED) and/or hospital with ARI.

 

METHODS

Study Population

This study was nested within the Pediatric Respiratory Illness Measurement System (PRIMES) study, a prospective cohort study of children with ARI in the ED and inpatient settings at five tertiary care children’s hospitals within the Pediatric Research in Inpatient Settings Network in Colorado, Pennsylvania, Tennessee, Texas, and Washington. Eligible children were two weeks to 16 years of age hospitalized after presenting to the ED with a primary diagnosis of asthma, croup, bronchiolitis, or pneumonia between July 1, 2014 and June 30, 2016. Because of an anticipated low frequency of croup hospitalizations, we also included children presenting to the ED and then discharged to home with this diagnosis. Children were assigned to a PRIMES diagnosis group based on their final discharge diagnosis. If there was a discrepancy between admission and discharge diagnoses, the discharge diagnosis was used. If a child had more than one discharge diagnosis for a PRIMES condition (eg, acute asthma and pneumonia), we chose the PRIMES condition with the lowest total enrollments overall. If the final discharge diagnosis was not a PRIMES condition, the case was excluded from further analysis. Patients with immunodeficiency, cystic fibrosis, a history of prematurity <32 weeks, chronic neuromuscular disease, cardiovascular disease, pulmonary diseases (other than asthma), and moderate to severe developmental delay were also excluded. Children admitted to intensive care were eligible only if they were transferred to an acute care ward <72 hours following admission. A survey was administered at the time of enrollment that collected information on SHS exposure, HRQOL, healthcare utilization, and demographics. All study procedures were reviewed and approved by the institutional review boards at each of the participating hospitals.

SECONDHAND SMOKE EXPOSURE

To ascertain SHS exposure, we asked caregivers, “How many persons living in the child’s home smoke?” Responses were dichotomized into non-SHS exposed (0 smokers) and SHS exposed (≥1 smokers). Children with missing data on home SHS exposure were excluded.

Health-Related Quality of Life Outcomes

We estimated HRQOL using the Pediatric Quality of Life (PedsQL^TM) 4.0 Generic Core and Infant Scales. The PedsQL instruments are validated, population HRQOL measures that evaluate the physical, mental, emotional, and social functioning of children two to 18 years old based on self- or caregiver-proxy report.^12-15 These instruments have also shown responsiveness as well as construct and predictive validity in hospitalized children.¹¹ For this study, we focused on the PedsQL physical functioning subscale, which assesses for problems with physical activities (eg, sports activity or exercise, low energy, and hurts or aches) on a five-point Likert scale (never to almost always a problem). Scores range from 0 to 100 with higher scores indicating a better HRQOL. The reported minimal clinically important difference (MCID), defined as the smallest difference in which individuals would perceive a benefit or would necessitate a change in management, for this scale is 4.5 points.^16,17

Children >8 years old were invited to complete the self-report version of the PedsQL. For children <8 years old, and for older children who were unable to complete them, surveys were completed by a parent or legal guardian. Respondents were asked to assess perceptions of their (or their child’s) HRQOL during periods of baseline health (the child’s usual state of health in the month preceding the current illness) and during the acute illness (the child’s state of health at the time of admission) as SHS exposure may influence perceptions of general health and/or contribute to worse outcomes during periods of acute illness.

 

Covariates collected at the time of enrollment included sociodemographics (child age, gender, race/ethnicity, and caregiver education), and healthcare utilization (caregiver-reported patient visits to a healthcare provider in the preceding six months). Insurance status and level of medical complexity (using the Pediatric Medical Complexity Algorithm)¹⁸ were obtained using the Pediatric Hospital Information System database, an administrative database containing clinical and resource utilization data from >45 children’s hospitals in the United States including all of the PRIMES study hospitals.¹³

Analysis

Descriptive statistics included frequency (%) and mean (standard deviation). Bivariate comparisons according to SHS exposure status were analyzed using chi-squared tests for categorical variables and analysis of variance for continuous variables. Multivariable linear mixed regression models were used to examine associations between home SHS exposure and HRQOL for baseline health and during admission, overall and stratified by diagnosis. Covariates in each model included age, sex, race/ethnicity, caregiver education, and healthcare visits in the preceding six months. We also included a hospital random effect to account for clustering of patients within hospitals and used robust standard errors for inference.

In a secondary analysis to explore potential dose-response effects of SHS exposure, we examined associations between an ordinal exposure variable (0 smokers, 1 smoker, ≥2 smokers) and HRQOL for baseline health and during admission for the acute illness. Because of sample size limitations, diagnosis-specific analyses examining dose-response effects were not conducted.

RESULTS

Study Population

Of the 2,334 children enrolled in the PRIMES study, 25 (1%) respondents did not report on home SHS exposure and were excluded, yielding a final study population of 2,309 children, of whom 728 (32%) had reported home SHS exposure. The study population included 664 children with asthma (mean age seven years [3.5]; 38% with home SHS exposure), 740 with bronchiolitis (mean age 0.7 years [0.5]; 32% with home SHS exposure), 342 with croup (mean age 1.7 [1.1]; 25% with home SHS exposure), and 563 with pneumonia (mean age 4.4 [3.8]; 27% with home SHS exposure; Table 1). Compared with non-SHS-exposed children, those with home SHS exposure tend to be slightly older (3.9 vs 3.4 years, P = .01), more likely to be non-Hispanic Black (29% vs 19%, P < .001), to have a chronic condition (52% vs 41%, P < .001), to come from a household where caregiver(s) did not graduate from college (45% vs 29%, P < .001), and to have public insurance (73% vs 49%, P < .001).

Home SHS Exposure and Health-related Quality of Life

The overall mean HRQOL score for baseline health was 83 (15), with a range across diagnoses of 82 to 87. Compared with non-SHS-exposed children, children with home SHS exposure had a lower mean HRQOL score for baseline health (adjusted mean difference –3.04 [95% CI -4.34, –1.74]). In analyses stratified by diagnosis, baseline health scores were lower for SHS-exposed children for all four conditions, but differences were statistically significant only for bronchiolitis (adjusted mean difference –2.94 [–5.0, –0.89]) and pneumonia (adjusted mean value –4.13 [–6.82, –1.44]; Table 2); none of these differences met the MCID threshold.

 

The overall mean HRQOL score at the time of admission was 56 (23), with a range across diagnoses of 49 to 61, with lower scores noted among SHS-exposed children compared with non-SHS-exposed children (adjusted mean difference –2.16 [–4.22, –0.10]). Similar to scores representing baseline health, admission scores were lower across all four conditions for SHS-exposed children. Only children with croup, however, had significantly lower admission scores that also met the MCID threshold (adjusted mean difference –5.71 [–10.67, –0.75]; Table 2).

To assess for potential dose-response effects of SHS exposure on HRQOL, we stratified SHS-exposed children into those with one smoker in the home (n = 513) and those with ≥2 smokers in the home (n = 215). Compared with non-SHS-exposed children, both HRQOL scores (baseline health and admission) were lower for SHS-exposed children. Consistent with a dose-response association, scores were lowest for children with ≥2 smokers in the home, both at baseline health (adjusted mean difference –3.92 [–6.03, –1.81]) and on admission (adjusted mean difference –3.67 [–6.98, –0.36]; Table 3).

DISCUSSION

Within a multicenter cohort of 2,309 children hospitalized with ARI, we noted significantly lower HRQOL scores among children exposed to SHS in the home as compared with nonexposed children. Differences were greatest for children living with ≥2 smokers in the home. In analyses stratified by diagnosis, differences in baseline health HRQOL scores were greatest for children with bronchiolitis and pneumonia. Differences in acute illness scores were greatest for children with croup.¹⁶

Our study provides evidence for acute and chronic impacts of SHS on HRQOL in children hospitalized with ARI. Although several studies have linked SHS exposure to reduced HRQOL in adults,^19,20 few similar studies have been conducted in children. Nonetheless, a wealth of studies have documented the negative impact of SHS exposure on clinical outcomes among children with ARI.^8,10,21-23 Our findings that home SHS exposure was associated with reduced HRQOL among our cohort of children with ARI are therefore consistent with related findings in adults and children. The observation that the effects of SHS exposure on HRQOL were greatest among children living with ≥2 smokers provides further evidence of a potential causal link between regular SHS exposure and HRQOL.

Although the magnitude and significance of associations between SHS exposure and HRQOL varied for each of the four diagnoses for baseline health and the acute illness, it is important to note that the point estimates for the adjusted mean differences were uniformly lower for the SHS-exposed children in each subgroup. Even so, only acute illness scores for croup exceeded the MCID threshold.¹⁶ Croup is the only included condition of the upper airway and is characterized by laryngotracheal inflammation leading to the typical cough and, in moderate to severe cases, stridor. Given that chronic SHS exposure induces a proinflammatory state,³ it is possible that SHS-exposed children with croup had more severe illness compared with nonexposed children with croup resulting in lower HRQOL scores on admission. Further, perceived differences in illness severity and HRQOL may be more readily apparent in children with croup (eg, stridor at rest vs intermittent or no stridor) as compared with children with lower respiratory tract diseases.

Of the four included diagnoses, the link between SHS exposure and asthma outcomes has been most studied. Prior work has demonstrated more frequent and severe acute exacerbations, as well as worse long-term lung function among SHS-exposed children as compared with nonexposed children.^22-24 It was, therefore, surprising that our study failed to demonstrate associations between SHS exposure and HRQOL among children with asthma. Reasons for this finding are unclear. One hypothesis is that caregivers of SHS-exposed children with asthma may be more aware of the impacts of SHS exposure on respiratory health (through prior education) and, thus, more likely to modify their smoking behaviors, or for their children to be on daily asthma controller therapy. Alternatively, caregivers of children with asthma may be more likely to underreport home SHS exposure. Thirty-eight percent of children with asthma, however, were classified as SHS-exposed. This percentage was greater than the other three conditions studied (25%-32%), suggesting that differential bias in underreporting was minimal. Given that children with asthma were older, on average, than children with the other three conditions, it may also be that these children spent more time in smoke-free environments (eg, school).

Nearly one-third of children in our study were exposed to SHS in the home. This is similar to the prevalence of exposure in other studies conducted among hospitalized children^8,10,21,25 but higher than the national prevalence of home SHS exposure among children in the United States.²⁶ Thus, hospitalized children represent a particularly vulnerable population and an important target for interventions aiming to reduce exposure to SHS. Although longitudinal interventions are likely necessary to affect long-term success, hospitalization for ARI may serve as a powerful teachable moment to begin cessation efforts. Hospitalization also offers time beyond a typical primary care outpatient encounter to focus on cessation counseling and may be the only opportunity to engage in counseling activities for some families with limited time or access. Further, prior studies have demonstrated both the feasibility and the effectiveness of smoking cessation interventions in hospitalized children.^27-30 Unfortunately, however, SHS exposure is often not documented at the time of hospitalization, and many opportunities to intervene are missed.^25,31 Thus, there is a need for improved strategies to reliably identify and intervene on SHS-exposed children in the hospital setting.

These findings should be considered in the context of several limitations. The observational nature of our study raises the potential for confounding, specifically with regard to socioeconomic status, as this is associated with both SHS exposure and lower HRQOL. Our modeling approach attempted to control for several factors associated with socioeconomic status, including caregiver education and insurance coverage, but there is potential for residual confounding. No single question is sufficient to fully assess SHS exposure as the intensity of home SHS exposure likely varies widely, and some children may be exposed to SHS outside of the home environment.³² The home, however, is often the most likely source of regular SHS exposure,^33,34 especially among young children (our cohort’s mean age was 3.6 years). Misclassification of SHS exposure is also possible due to underreporting of smoking.^35,36 As a result, some children regularly exposed to SHS may have been misclassified as nonexposed, and the observed associations between SHS exposure and HRQOL may be underestimated. Confirming our study’s findings using objective assessments of SHS exposure, such as cotinine, are warranted. Given the young age of our cohort, the PedsQL surveys were completed by the parent or legal guardian only in >90% of the enrolled subjects, and caregiver perceptions may not accurately reflect the child’s perceptions. Prior work, however, has demonstrated the validity of parent-proxy reporting of the PedsQL, including correlation with child self-report.³⁷ In our study, correlation between child and caregiver reporting (when available) was also very good (r = 0.72, 95% CI 0.64, 0.77). It is also possible that the timing of the HRQOL assessments (on admission) may have biased perceptions of baseline HRQOL, although we anticipate any bias would likely be nondifferential between SHS-exposed and nonexposed children and across diagnoses.

Nearly one-third of children in our study were exposed to SHS exposure in the home, and SHS exposure was associated with lower HRQOL for baseline health and during acute illness, providing further evidence of the dangers of SHS. Much work is needed in order to eliminate the impact of SHS on child health and families of children hospitalized for respiratory illness should be considered a priority population for smoking cessation efforts.

 

Acknowledgment

The authors wish to acknowledge the efforts of PRIS-PRIMES study team. The authors also wish to thank the children and families who consented to be a part of the PRIMES study.

Disclosures

The authors have no conflicts of interest relevant to this article to disclose.

Funding

This study was supported by NIH-NHLBI 1R01HL121067 to RMS.

Internal Medicine (IM) residency graduates are expected to manage a wide range of acute clinical events.¹ Urgent and emergent inpatient situations require a broad knowledge base for rapid bedside diagnosis, yet the essential clinical skills required to manage acute clinical events pose a unique training challenge given the rarity and high-stakes nature of several such emergencies. For example, in three years of residency, a trainee may never have the opportunity to manage anaphylaxis, yet IM graduates must be able to recognize and quickly initiate proper lifesaving treatment for this relatively rare event² when it does occur.

In an era of work-hour limitations and heightened trainee supervision, residents perceive diminished familiarity with several clinical situations^3-5 and may feel unprepared to handle crisis events such as cardiac arrest.⁶ Given the sporadic nature of clinical medicine, many residents may not be exposed to certain acute inpatient clinical scenarios by the end of their training, a potentially critical education gap. To our knowledge, IM residents’ level of exposure to acute clinical events has not previously been studied. The aims of this study were to develop an instrument aimed at assessing IM residents’ exposure to hospital acute clinical events at a large academic medical center and to investigate the relationship between exposure and confidence in managing these events.

 

METHODS

Survey Development

We reviewed the Massachusetts General Hospital (MGH) IM residency program curriculum (including simulation, conferences, and other didactics), the American Board of Internal Medicine certification requirements (primarily related to Advanced Cardiac Life Support [ACLS]), and the MGH inpatient rapid response events and gained input from the IM program leadership to develop a list of 50 acute clinical events that a graduating resident may be expected to manage independently (Box 1, Supplementary Appendix).^7-9 We then developed a survey assessing residents’ exposure to and confidence in managing such events. To classify the level of exposure, residents were asked to distinguish whether they had managed these events during a simulation session, inpatient as a part of a team, or inpatient independently. At our institution, IM postgraduate year 1 (PGY-1) interns manage a floor of patients overnight under a senior resident’s supervision, PGY-2 residents manage a team of several interns often without attending presence on ward rounds,¹⁰ and senior PGY-3 or -4 residents are expected to lead the hospital’s rapid response and code team and triage decompensating patients to the intensive care unit. Therefore, there are ample opportunities for IM residents to manage conditions independently (ie, in a direct leadership role) with attending supervision. House officers’ role in medical management, including calling appropriate subspecialty consultation, depends on the clinical condition; for example, a graduating senior resident would be expected to evaluate comprehensively a hypotensive patient and diagnose tension pneumothorax (while calling interventional pulmonary support for needle decompression and chest tube placement) and independently run an ACLS algorithm in the case of an unstable arrhythmia or cardiac arrest.

Residents were also asked to rate their perceived confidence in managing each condition independently on a five-point scale (ranging from “definitely cannot manage this condition independently” to “definitely can manage this condition independently”). We refined the survey instrument through a collaborative, iterative review process, including cognitive interviews and piloting with IM subspecialty fellows.

Participants and Data Collection

All IM residents at the Massachusetts General Hospital were invited to participate in the study. The study was conducted in May 2015 to reflect training throughout the prior academic year(s) and allow us to evaluate graduating residents’ exposures across all prior years of training. The instrument was administered anonymously via a web-based survey tool, Qualtrics (Provo, Utah). The study was approved as exempt by the Partners Institutional Review Board.

Data Analysis

Residents’ self-reported exposure to hospital acute events was classified into the following six ordinal categories: (1) never seen (have never seen the condition under any circumstances); (2) simulation alone (have managed the condition only during a mannequin-simulated patient case); (3) team alone (have managed the condition inpatient as a part of a team of providers, not in a primary leadership role); (4) team plus simulation; (5) independently (have managed the condition inpatient alone or in a primary leadership role); and (6) independently plus simulation. Residents’ self-reported exposure was examined for each postgraduate year (PGY) class both in aggregate and for each individual acute event. We sought to identify events that the majority of residents had managed independently (85% of residents or greater) and less common events that at least 15% of residents had never experienced.

 

We also examined residents’ self-reported confidence for each PGY class in aggregate and for each clinical acute scenario. Confidence was investigated in a dichotomized manner with a “definitely can” rating indicating “Confident” and with “probably can,” “neutral,” “probably cannot,” or “definitely cannot” ratings indicating “Not Confident” to manage the condition independently. Dichotomization thus allowed us to set a high bar for confidence, reflecting the self-perceived ability of the residents to manage the conditions as future independent physicians.

We used logistic regression models with the generalized estimating equations (GEE) approach to take into account the repeated measures of 50 clinical acute clinical events assessed for each resident. We compared the distribution of self-reported exposure and confidence among different PGY classes and examined the relationship between confidence and self-reported exposure stratified by level of training. We also assessed the independent effect of exposure on confidence controlling for level of training in a multivariable logistic regression model.

RESULTS

A total of 140 of 170 IM residents completed the survey (82% overall response rate: 72% of all PGY-1 residents, 86% of PGY-2 residents, and 89% of PGY-3/4 residents). In total, 41 PGY-1 residents (29% of respondents), 50 PGY-2 residents (36%), and 49 PGY-3 or PGY-4 residents (35%) participated. The majority of residents were in the Categorical IM training track (106 residents, 76% of respondents), whereas the remainder of respondents were in various subspecialty training tracks within our IM residency program, including Primary Care (14 residents, 10%), and four-year tracks, including Global Health (six residents, 4%), and Medicine-Pediatrics (14 residents, 10%).

Assessment of Exposure

Residents reported increasingly independent exposures as they progressed through residency training. PGY-1 residents on average had never seen 16.3% of the 50 acute events, whereas PGY-3/4 residents had never seen only 4.0% of the events (P < .0001). PGY-1 residents had managed 31.3% of events independently (or both independently and in simulation) as opposed to 71.7% of events for PGY-3/4 residents (P < .0001). Simulation alone accounted for a substantial proportion of exposures (16.4%) for PGY-1 residents, but this was significantly lower for PGY-2 or PGY-3/4 residents (P < .0001), who reported a greater percentage of exposures in nonsimulation clinical scenarios either independently or as a part of an inpatient team. There were no outlier residents who reported lower exposure compared with their PGY peers.

There was a wide spectrum of resident-reported exposures when individual acute events were examined (Table, full data in Supplementary Appendix Table 1). Events with the highest levels of exposure, which >85% of PGY-1 residents had managed independently, included alcohol withdrawal, chronic obstructive pulmonary disease exacerbation, rapid atrial fibrillation, agitated delirium, hypertensive urgency, and hyperkalemia. Events with the lowest levels of exposure, which at least 15% of graduating residents had never encountered in the hospital, included the following eight of 50 events (16%): torsades de pointes (51% of PGY-3/4 residents), acute mechanical valve failure (49%), tension pneumothorax (38.8%), use of emergency transcutaneous pacing (38.8%), elevated intracranial pressure (ICP)/herniation (24.5%), aortic dissection (22.4%), cord compression (16.3%), and use of emergency cardioversion (16.3%). Several PGY-3/4 residents had managed several of these events only in mannequin simulations, including torsades de pointes (41%), transcutaneous pacing (33%), and tension pneumothorax (24%).

 

Assessment of Confidence

Both levels of training and exposure to acute events were associated with increased confidence in managing such events. PGY-1 residents felt confident in managing 24.9% of acute events independently, compared to 48.4% of events for PGY-2 residents and 72.5% of events for PGY-3/4 residents (P < .0001). There was considerable variation in confidence among the individual acute events (Supplementary Appendix Table 2). A majority of graduating PGY-3/4 residents did not feel confident in managing the following 10 of the 50 events (20%): use of emergency cardioversion, aortic dissection, thrombotic thrombocytopenic purpura/hemolytic uremic syndrome (TTP/HUS), torsades de pointes, posterior reversible encephalopathy syndrome (PRES), intracranial hemorrhage, use of emergency transcutaneous pacing, tension pneumothorax, elevated ICP/herniation, and acute mechanical valve failure.

Residents’ self-reported confidence also correlated with level of exposure. There was a significant increase in resident confidence with increasingly independent exposure stratified by level of training (Figure; all with P < .0001). In the multivariable logistic regression model, increasing exposure correlated with increased resident confidence (P < .0001) while controlling for PGY year (P = .001).

DISCUSSION

We developed an instrument to assess resident exposure to and confidence in managing 50 inpatient acute clinical events. Both exposure and level of training were associated with increasing resident confidence. We identified specific events with low levels of exposure and confidence that could be targeted for educational interventions.

To our knowledge, this is the first study to examine IM residents’ exposure to and confidence in managing a wide range of inpatient acute clinical events. A primary goal of residency is to provide physicians-in-training graduated responsibility to prepare them for eventual independent practice. Although our survey confirmed that IM residents’ exposure and confidence significantly increased as they advanced through training (a not unexpected finding), our data also show that even after controlling for year in training, independent exposures significantly correlated with increased confidence. This speaks to the importance of preserving opportunities for residents to manage critical events in a supported manner, an admittedly challenging prospect given the oft-competing calls for supervision of and mentored feedback for trainees.¹¹

Despite identifying independent exposure as an important factor that impacts resident confidence, we found that there was still a substantial proportion of events (28.3%) that senior medical residents near the end of their training had not managed independently in a primary leadership role. Although our study was not designed to determine the reasons for this varied resident exposure, possible explanations may include the relative rarity of certain acute clinical events compared with others, or less likely the effect of duty hour limitations, attending supervision of trainees, or programmatic changes in resident leadership responsibilities. Whatever the cause, this finding uniquely identifies an area for improvement to prevent new attending physicians from feeling unprepared to manage potentially critical emergencies.

An important goal of our study was to develop an instrument that would enable training programs to identify their learning needs. Both program-wide and individual assessments of resident case exposure and confidence are essential for identifying such learning needs and areas for curricular development. Program-wide assessments can spur an important debate about program goals and requirements with respect to what scenarios residents must be able to manage competently by graduation.¹² In addition, such assessments can help individualize learning exposures based on a specific learner’s needs and career goals. The administration of our survey instrument required minimal resources, and the high response rate in our study suggests that other programs can implement our instrument to accomplish these goals.

Alternative methods, such as electronic learning portfolios (efolios), can be utilized to assess resident case exposure. In comparison to our survey instrument, efolios limit recall bias by utilizing case logs and have additional capabilities such as compiling evaluations and enabling trainees to set learning goals. However, there are considerable barriers to the effective use of efolios, including software cost, learner attitudes, and time constraints.¹³ Tools such as our end-of-year assessment offer an alternative method that limits these barriers.

Once educational growth opportunities have been identified through survey-based or other methods, residency programs must determine how to optimize curricula for the needs and career goals of their trainees. We found considerable overlap among conditions that graduating residents had both limited exposure to and low confidence in managing (eg, torsades de pointes, tension pneumothorax, and emergency cardioversion), which are logical topics for future curriculum development. We also identified a few conditions (including PRES, TTP/HUS, and intracranial hemorrhage) that graduating residents did not feel confident in managing despite a relatively higher reported level of exposure. Whether to focus specific educational interventions on the most rare or most commonly encountered acute clinical events is likely to be a topic of debate among individual training programs, but the results of our survey indicate that there is likely to be educational benefit to both strategies.

Residency programs can employ a variety of modalities to enhance learner exposure and confidence in managing clinical scenarios that are deemed important by the program, including didactics, simulation, and changes in program structure. There is a substantial literature on the use of dedicated curricula for crisis management and the use of simulation as a training tool for responding to acute clinical events in multiple specialties^14-24 and in nonmedical domains such as aviation.^25-27 Simulation has been shown to improve residents’ clinical skills and comfort level with some acute events^28-30 and may even be superior to traditional clinical medical education.³¹ In addition, programs can utilize targeted clinical experiences such as intensive care unit and subspecialty rotations^32,33 in an effort to customize educational interventions to fill identified gaps in learner exposure or confidence.

Our study has several limitations. First, we investigated a single large IM residency program at a quaternary academic medical center, and therefore, our findings may not be externally generalizable to all IM residencies or other medical specialties. Our unique peer-led simulation curriculum, including 16 PGY-1 and 8 PGY-2 cases chosen based on clinical rotations at Massachusetts General Hospital,⁷ likely impacted residents’ exposure to simulation that is specific to our institution. However, although specific inpatient acute events may vary among other institutions, our finding that graduating residents still reported gaps in their clinical experience is likely generalizable to other programs given the varied and unpredictable nature of ward medicine training. In addition, our survey tool was simple to administer and could be tailored to reflect the acute events and training needs relevant to other residency programs, specialties, and institutions. Second, the retrospective nature of our study may be subject to participants’ recall bias. We did not restrict our survey questions to urgent conditions managed only on IM hospital wards and some may have been experienced in the emergency room or intensive care units; however, these exposures are still relevant as key components of IM training. Third, our list of 50 acute clinical events was intentionally broad and included several conditions that require multidisciplinary subspecialist consultation, which could have impacted residents’ self-report of “independent” exposures. However, these scenarios are ones that hospitalists may independently recognize and stabilize, engaging appropriate specialists. Fourth, we were not able to validate residents’ self-reported exposures against other measures of the frequency of housestaff management of acute events (such as billing data or patient logs) as this information is not routinely collected. We also did not attempt to identify the reasons underlying the variation seen in resident exposure and confidence for individual acute events, but as a needs assessment, this was beyond the scope of our study. Finally, our assessment of resident confidence was subjective and we were not able to assess competence, with prior studies demonstrating conflicting results regarding the relationship between self-reported proficiency and observed competence.^34-36 Future studies are needed to investigate whether case exposure assessment leads to changes in residency curricula and whether such curricula increase resident confidence and competence in managing hospital acute clinical events.

 

CONCLUSION

We developed an easy-to-administer tool to assess IM residents’ exposure to and confidence in managing inpatient acute events. We found that both significantly increased as residents advanced through training, and self-reported confidence additionally correlated with level of exposure independent of PGY class. We identified several specific inpatient acute clinical events with low levels of resident exposure and confidence that can serve as targets for future IM residency curriculum development. Future studies assessing the impact of such curricula on resident confidence and competence are needed.

Disclosures

The authors declare no conflict of interest.

Internal Medicine (IM) residency graduates are expected to manage a wide range of acute clinical events.¹ Urgent and emergent inpatient situations require a broad knowledge base for rapid bedside diagnosis, yet the essential clinical skills required to manage acute clinical events pose a unique training challenge given the rarity and high-stakes nature of several such emergencies. For example, in three years of residency, a trainee may never have the opportunity to manage anaphylaxis, yet IM graduates must be able to recognize and quickly initiate proper lifesaving treatment for this relatively rare event² when it does occur.

In an era of work-hour limitations and heightened trainee supervision, residents perceive diminished familiarity with several clinical situations^3-5 and may feel unprepared to handle crisis events such as cardiac arrest.⁶ Given the sporadic nature of clinical medicine, many residents may not be exposed to certain acute inpatient clinical scenarios by the end of their training, a potentially critical education gap. To our knowledge, IM residents’ level of exposure to acute clinical events has not previously been studied. The aims of this study were to develop an instrument aimed at assessing IM residents’ exposure to hospital acute clinical events at a large academic medical center and to investigate the relationship between exposure and confidence in managing these events.

 

METHODS

Survey Development

We reviewed the Massachusetts General Hospital (MGH) IM residency program curriculum (including simulation, conferences, and other didactics), the American Board of Internal Medicine certification requirements (primarily related to Advanced Cardiac Life Support [ACLS]), and the MGH inpatient rapid response events and gained input from the IM program leadership to develop a list of 50 acute clinical events that a graduating resident may be expected to manage independently (Box 1, Supplementary Appendix).^7-9 We then developed a survey assessing residents’ exposure to and confidence in managing such events. To classify the level of exposure, residents were asked to distinguish whether they had managed these events during a simulation session, inpatient as a part of a team, or inpatient independently. At our institution, IM postgraduate year 1 (PGY-1) interns manage a floor of patients overnight under a senior resident’s supervision, PGY-2 residents manage a team of several interns often without attending presence on ward rounds,¹⁰ and senior PGY-3 or -4 residents are expected to lead the hospital’s rapid response and code team and triage decompensating patients to the intensive care unit. Therefore, there are ample opportunities for IM residents to manage conditions independently (ie, in a direct leadership role) with attending supervision. House officers’ role in medical management, including calling appropriate subspecialty consultation, depends on the clinical condition; for example, a graduating senior resident would be expected to evaluate comprehensively a hypotensive patient and diagnose tension pneumothorax (while calling interventional pulmonary support for needle decompression and chest tube placement) and independently run an ACLS algorithm in the case of an unstable arrhythmia or cardiac arrest.

Residents were also asked to rate their perceived confidence in managing each condition independently on a five-point scale (ranging from “definitely cannot manage this condition independently” to “definitely can manage this condition independently”). We refined the survey instrument through a collaborative, iterative review process, including cognitive interviews and piloting with IM subspecialty fellows.

Participants and Data Collection

All IM residents at the Massachusetts General Hospital were invited to participate in the study. The study was conducted in May 2015 to reflect training throughout the prior academic year(s) and allow us to evaluate graduating residents’ exposures across all prior years of training. The instrument was administered anonymously via a web-based survey tool, Qualtrics (Provo, Utah). The study was approved as exempt by the Partners Institutional Review Board.

Data Analysis

Residents’ self-reported exposure to hospital acute events was classified into the following six ordinal categories: (1) never seen (have never seen the condition under any circumstances); (2) simulation alone (have managed the condition only during a mannequin-simulated patient case); (3) team alone (have managed the condition inpatient as a part of a team of providers, not in a primary leadership role); (4) team plus simulation; (5) independently (have managed the condition inpatient alone or in a primary leadership role); and (6) independently plus simulation. Residents’ self-reported exposure was examined for each postgraduate year (PGY) class both in aggregate and for each individual acute event. We sought to identify events that the majority of residents had managed independently (85% of residents or greater) and less common events that at least 15% of residents had never experienced.

 

We also examined residents’ self-reported confidence for each PGY class in aggregate and for each clinical acute scenario. Confidence was investigated in a dichotomized manner with a “definitely can” rating indicating “Confident” and with “probably can,” “neutral,” “probably cannot,” or “definitely cannot” ratings indicating “Not Confident” to manage the condition independently. Dichotomization thus allowed us to set a high bar for confidence, reflecting the self-perceived ability of the residents to manage the conditions as future independent physicians.

We used logistic regression models with the generalized estimating equations (GEE) approach to take into account the repeated measures of 50 clinical acute clinical events assessed for each resident. We compared the distribution of self-reported exposure and confidence among different PGY classes and examined the relationship between confidence and self-reported exposure stratified by level of training. We also assessed the independent effect of exposure on confidence controlling for level of training in a multivariable logistic regression model.

RESULTS

A total of 140 of 170 IM residents completed the survey (82% overall response rate: 72% of all PGY-1 residents, 86% of PGY-2 residents, and 89% of PGY-3/4 residents). In total, 41 PGY-1 residents (29% of respondents), 50 PGY-2 residents (36%), and 49 PGY-3 or PGY-4 residents (35%) participated. The majority of residents were in the Categorical IM training track (106 residents, 76% of respondents), whereas the remainder of respondents were in various subspecialty training tracks within our IM residency program, including Primary Care (14 residents, 10%), and four-year tracks, including Global Health (six residents, 4%), and Medicine-Pediatrics (14 residents, 10%).

Assessment of Exposure

Residents reported increasingly independent exposures as they progressed through residency training. PGY-1 residents on average had never seen 16.3% of the 50 acute events, whereas PGY-3/4 residents had never seen only 4.0% of the events (P < .0001). PGY-1 residents had managed 31.3% of events independently (or both independently and in simulation) as opposed to 71.7% of events for PGY-3/4 residents (P < .0001). Simulation alone accounted for a substantial proportion of exposures (16.4%) for PGY-1 residents, but this was significantly lower for PGY-2 or PGY-3/4 residents (P < .0001), who reported a greater percentage of exposures in nonsimulation clinical scenarios either independently or as a part of an inpatient team. There were no outlier residents who reported lower exposure compared with their PGY peers.

There was a wide spectrum of resident-reported exposures when individual acute events were examined (Table, full data in Supplementary Appendix Table 1). Events with the highest levels of exposure, which >85% of PGY-1 residents had managed independently, included alcohol withdrawal, chronic obstructive pulmonary disease exacerbation, rapid atrial fibrillation, agitated delirium, hypertensive urgency, and hyperkalemia. Events with the lowest levels of exposure, which at least 15% of graduating residents had never encountered in the hospital, included the following eight of 50 events (16%): torsades de pointes (51% of PGY-3/4 residents), acute mechanical valve failure (49%), tension pneumothorax (38.8%), use of emergency transcutaneous pacing (38.8%), elevated intracranial pressure (ICP)/herniation (24.5%), aortic dissection (22.4%), cord compression (16.3%), and use of emergency cardioversion (16.3%). Several PGY-3/4 residents had managed several of these events only in mannequin simulations, including torsades de pointes (41%), transcutaneous pacing (33%), and tension pneumothorax (24%).

 

Assessment of Confidence

Both levels of training and exposure to acute events were associated with increased confidence in managing such events. PGY-1 residents felt confident in managing 24.9% of acute events independently, compared to 48.4% of events for PGY-2 residents and 72.5% of events for PGY-3/4 residents (P < .0001). There was considerable variation in confidence among the individual acute events (Supplementary Appendix Table 2). A majority of graduating PGY-3/4 residents did not feel confident in managing the following 10 of the 50 events (20%): use of emergency cardioversion, aortic dissection, thrombotic thrombocytopenic purpura/hemolytic uremic syndrome (TTP/HUS), torsades de pointes, posterior reversible encephalopathy syndrome (PRES), intracranial hemorrhage, use of emergency transcutaneous pacing, tension pneumothorax, elevated ICP/herniation, and acute mechanical valve failure.

Residents’ self-reported confidence also correlated with level of exposure. There was a significant increase in resident confidence with increasingly independent exposure stratified by level of training (Figure; all with P < .0001). In the multivariable logistic regression model, increasing exposure correlated with increased resident confidence (P < .0001) while controlling for PGY year (P = .001).

DISCUSSION

We developed an instrument to assess resident exposure to and confidence in managing 50 inpatient acute clinical events. Both exposure and level of training were associated with increasing resident confidence. We identified specific events with low levels of exposure and confidence that could be targeted for educational interventions.

To our knowledge, this is the first study to examine IM residents’ exposure to and confidence in managing a wide range of inpatient acute clinical events. A primary goal of residency is to provide physicians-in-training graduated responsibility to prepare them for eventual independent practice. Although our survey confirmed that IM residents’ exposure and confidence significantly increased as they advanced through training (a not unexpected finding), our data also show that even after controlling for year in training, independent exposures significantly correlated with increased confidence. This speaks to the importance of preserving opportunities for residents to manage critical events in a supported manner, an admittedly challenging prospect given the oft-competing calls for supervision of and mentored feedback for trainees.¹¹

Despite identifying independent exposure as an important factor that impacts resident confidence, we found that there was still a substantial proportion of events (28.3%) that senior medical residents near the end of their training had not managed independently in a primary leadership role. Although our study was not designed to determine the reasons for this varied resident exposure, possible explanations may include the relative rarity of certain acute clinical events compared with others, or less likely the effect of duty hour limitations, attending supervision of trainees, or programmatic changes in resident leadership responsibilities. Whatever the cause, this finding uniquely identifies an area for improvement to prevent new attending physicians from feeling unprepared to manage potentially critical emergencies.

An important goal of our study was to develop an instrument that would enable training programs to identify their learning needs. Both program-wide and individual assessments of resident case exposure and confidence are essential for identifying such learning needs and areas for curricular development. Program-wide assessments can spur an important debate about program goals and requirements with respect to what scenarios residents must be able to manage competently by graduation.¹² In addition, such assessments can help individualize learning exposures based on a specific learner’s needs and career goals. The administration of our survey instrument required minimal resources, and the high response rate in our study suggests that other programs can implement our instrument to accomplish these goals.

Alternative methods, such as electronic learning portfolios (efolios), can be utilized to assess resident case exposure. In comparison to our survey instrument, efolios limit recall bias by utilizing case logs and have additional capabilities such as compiling evaluations and enabling trainees to set learning goals. However, there are considerable barriers to the effective use of efolios, including software cost, learner attitudes, and time constraints.¹³ Tools such as our end-of-year assessment offer an alternative method that limits these barriers.

Once educational growth opportunities have been identified through survey-based or other methods, residency programs must determine how to optimize curricula for the needs and career goals of their trainees. We found considerable overlap among conditions that graduating residents had both limited exposure to and low confidence in managing (eg, torsades de pointes, tension pneumothorax, and emergency cardioversion), which are logical topics for future curriculum development. We also identified a few conditions (including PRES, TTP/HUS, and intracranial hemorrhage) that graduating residents did not feel confident in managing despite a relatively higher reported level of exposure. Whether to focus specific educational interventions on the most rare or most commonly encountered acute clinical events is likely to be a topic of debate among individual training programs, but the results of our survey indicate that there is likely to be educational benefit to both strategies.

Residency programs can employ a variety of modalities to enhance learner exposure and confidence in managing clinical scenarios that are deemed important by the program, including didactics, simulation, and changes in program structure. There is a substantial literature on the use of dedicated curricula for crisis management and the use of simulation as a training tool for responding to acute clinical events in multiple specialties^14-24 and in nonmedical domains such as aviation.^25-27 Simulation has been shown to improve residents’ clinical skills and comfort level with some acute events^28-30 and may even be superior to traditional clinical medical education.³¹ In addition, programs can utilize targeted clinical experiences such as intensive care unit and subspecialty rotations^32,33 in an effort to customize educational interventions to fill identified gaps in learner exposure or confidence.

Our study has several limitations. First, we investigated a single large IM residency program at a quaternary academic medical center, and therefore, our findings may not be externally generalizable to all IM residencies or other medical specialties. Our unique peer-led simulation curriculum, including 16 PGY-1 and 8 PGY-2 cases chosen based on clinical rotations at Massachusetts General Hospital,⁷ likely impacted residents’ exposure to simulation that is specific to our institution. However, although specific inpatient acute events may vary among other institutions, our finding that graduating residents still reported gaps in their clinical experience is likely generalizable to other programs given the varied and unpredictable nature of ward medicine training. In addition, our survey tool was simple to administer and could be tailored to reflect the acute events and training needs relevant to other residency programs, specialties, and institutions. Second, the retrospective nature of our study may be subject to participants’ recall bias. We did not restrict our survey questions to urgent conditions managed only on IM hospital wards and some may have been experienced in the emergency room or intensive care units; however, these exposures are still relevant as key components of IM training. Third, our list of 50 acute clinical events was intentionally broad and included several conditions that require multidisciplinary subspecialist consultation, which could have impacted residents’ self-report of “independent” exposures. However, these scenarios are ones that hospitalists may independently recognize and stabilize, engaging appropriate specialists. Fourth, we were not able to validate residents’ self-reported exposures against other measures of the frequency of housestaff management of acute events (such as billing data or patient logs) as this information is not routinely collected. We also did not attempt to identify the reasons underlying the variation seen in resident exposure and confidence for individual acute events, but as a needs assessment, this was beyond the scope of our study. Finally, our assessment of resident confidence was subjective and we were not able to assess competence, with prior studies demonstrating conflicting results regarding the relationship between self-reported proficiency and observed competence.^34-36 Future studies are needed to investigate whether case exposure assessment leads to changes in residency curricula and whether such curricula increase resident confidence and competence in managing hospital acute clinical events.

 

CONCLUSION

We developed an easy-to-administer tool to assess IM residents’ exposure to and confidence in managing inpatient acute events. We found that both significantly increased as residents advanced through training, and self-reported confidence additionally correlated with level of exposure independent of PGY class. We identified several specific inpatient acute clinical events with low levels of resident exposure and confidence that can serve as targets for future IM residency curriculum development. Future studies assessing the impact of such curricula on resident confidence and competence are needed.

Disclosures

The authors declare no conflict of interest.

Internal Medicine (IM) residency graduates are expected to manage a wide range of acute clinical events.¹ Urgent and emergent inpatient situations require a broad knowledge base for rapid bedside diagnosis, yet the essential clinical skills required to manage acute clinical events pose a unique training challenge given the rarity and high-stakes nature of several such emergencies. For example, in three years of residency, a trainee may never have the opportunity to manage anaphylaxis, yet IM graduates must be able to recognize and quickly initiate proper lifesaving treatment for this relatively rare event² when it does occur.

In an era of work-hour limitations and heightened trainee supervision, residents perceive diminished familiarity with several clinical situations^3-5 and may feel unprepared to handle crisis events such as cardiac arrest.⁶ Given the sporadic nature of clinical medicine, many residents may not be exposed to certain acute inpatient clinical scenarios by the end of their training, a potentially critical education gap. To our knowledge, IM residents’ level of exposure to acute clinical events has not previously been studied. The aims of this study were to develop an instrument aimed at assessing IM residents’ exposure to hospital acute clinical events at a large academic medical center and to investigate the relationship between exposure and confidence in managing these events.

 

METHODS

Survey Development

We reviewed the Massachusetts General Hospital (MGH) IM residency program curriculum (including simulation, conferences, and other didactics), the American Board of Internal Medicine certification requirements (primarily related to Advanced Cardiac Life Support [ACLS]), and the MGH inpatient rapid response events and gained input from the IM program leadership to develop a list of 50 acute clinical events that a graduating resident may be expected to manage independently (Box 1, Supplementary Appendix).^7-9 We then developed a survey assessing residents’ exposure to and confidence in managing such events. To classify the level of exposure, residents were asked to distinguish whether they had managed these events during a simulation session, inpatient as a part of a team, or inpatient independently. At our institution, IM postgraduate year 1 (PGY-1) interns manage a floor of patients overnight under a senior resident’s supervision, PGY-2 residents manage a team of several interns often without attending presence on ward rounds,¹⁰ and senior PGY-3 or -4 residents are expected to lead the hospital’s rapid response and code team and triage decompensating patients to the intensive care unit. Therefore, there are ample opportunities for IM residents to manage conditions independently (ie, in a direct leadership role) with attending supervision. House officers’ role in medical management, including calling appropriate subspecialty consultation, depends on the clinical condition; for example, a graduating senior resident would be expected to evaluate comprehensively a hypotensive patient and diagnose tension pneumothorax (while calling interventional pulmonary support for needle decompression and chest tube placement) and independently run an ACLS algorithm in the case of an unstable arrhythmia or cardiac arrest.

Residents were also asked to rate their perceived confidence in managing each condition independently on a five-point scale (ranging from “definitely cannot manage this condition independently” to “definitely can manage this condition independently”). We refined the survey instrument through a collaborative, iterative review process, including cognitive interviews and piloting with IM subspecialty fellows.

Participants and Data Collection

All IM residents at the Massachusetts General Hospital were invited to participate in the study. The study was conducted in May 2015 to reflect training throughout the prior academic year(s) and allow us to evaluate graduating residents’ exposures across all prior years of training. The instrument was administered anonymously via a web-based survey tool, Qualtrics (Provo, Utah). The study was approved as exempt by the Partners Institutional Review Board.

Data Analysis

Residents’ self-reported exposure to hospital acute events was classified into the following six ordinal categories: (1) never seen (have never seen the condition under any circumstances); (2) simulation alone (have managed the condition only during a mannequin-simulated patient case); (3) team alone (have managed the condition inpatient as a part of a team of providers, not in a primary leadership role); (4) team plus simulation; (5) independently (have managed the condition inpatient alone or in a primary leadership role); and (6) independently plus simulation. Residents’ self-reported exposure was examined for each postgraduate year (PGY) class both in aggregate and for each individual acute event. We sought to identify events that the majority of residents had managed independently (85% of residents or greater) and less common events that at least 15% of residents had never experienced.

 

We also examined residents’ self-reported confidence for each PGY class in aggregate and for each clinical acute scenario. Confidence was investigated in a dichotomized manner with a “definitely can” rating indicating “Confident” and with “probably can,” “neutral,” “probably cannot,” or “definitely cannot” ratings indicating “Not Confident” to manage the condition independently. Dichotomization thus allowed us to set a high bar for confidence, reflecting the self-perceived ability of the residents to manage the conditions as future independent physicians.

We used logistic regression models with the generalized estimating equations (GEE) approach to take into account the repeated measures of 50 clinical acute clinical events assessed for each resident. We compared the distribution of self-reported exposure and confidence among different PGY classes and examined the relationship between confidence and self-reported exposure stratified by level of training. We also assessed the independent effect of exposure on confidence controlling for level of training in a multivariable logistic regression model.

RESULTS

A total of 140 of 170 IM residents completed the survey (82% overall response rate: 72% of all PGY-1 residents, 86% of PGY-2 residents, and 89% of PGY-3/4 residents). In total, 41 PGY-1 residents (29% of respondents), 50 PGY-2 residents (36%), and 49 PGY-3 or PGY-4 residents (35%) participated. The majority of residents were in the Categorical IM training track (106 residents, 76% of respondents), whereas the remainder of respondents were in various subspecialty training tracks within our IM residency program, including Primary Care (14 residents, 10%), and four-year tracks, including Global Health (six residents, 4%), and Medicine-Pediatrics (14 residents, 10%).

Assessment of Exposure

Residents reported increasingly independent exposures as they progressed through residency training. PGY-1 residents on average had never seen 16.3% of the 50 acute events, whereas PGY-3/4 residents had never seen only 4.0% of the events (P < .0001). PGY-1 residents had managed 31.3% of events independently (or both independently and in simulation) as opposed to 71.7% of events for PGY-3/4 residents (P < .0001). Simulation alone accounted for a substantial proportion of exposures (16.4%) for PGY-1 residents, but this was significantly lower for PGY-2 or PGY-3/4 residents (P < .0001), who reported a greater percentage of exposures in nonsimulation clinical scenarios either independently or as a part of an inpatient team. There were no outlier residents who reported lower exposure compared with their PGY peers.

There was a wide spectrum of resident-reported exposures when individual acute events were examined (Table, full data in Supplementary Appendix Table 1). Events with the highest levels of exposure, which >85% of PGY-1 residents had managed independently, included alcohol withdrawal, chronic obstructive pulmonary disease exacerbation, rapid atrial fibrillation, agitated delirium, hypertensive urgency, and hyperkalemia. Events with the lowest levels of exposure, which at least 15% of graduating residents had never encountered in the hospital, included the following eight of 50 events (16%): torsades de pointes (51% of PGY-3/4 residents), acute mechanical valve failure (49%), tension pneumothorax (38.8%), use of emergency transcutaneous pacing (38.8%), elevated intracranial pressure (ICP)/herniation (24.5%), aortic dissection (22.4%), cord compression (16.3%), and use of emergency cardioversion (16.3%). Several PGY-3/4 residents had managed several of these events only in mannequin simulations, including torsades de pointes (41%), transcutaneous pacing (33%), and tension pneumothorax (24%).

 

Assessment of Confidence

Both levels of training and exposure to acute events were associated with increased confidence in managing such events. PGY-1 residents felt confident in managing 24.9% of acute events independently, compared to 48.4% of events for PGY-2 residents and 72.5% of events for PGY-3/4 residents (P < .0001). There was considerable variation in confidence among the individual acute events (Supplementary Appendix Table 2). A majority of graduating PGY-3/4 residents did not feel confident in managing the following 10 of the 50 events (20%): use of emergency cardioversion, aortic dissection, thrombotic thrombocytopenic purpura/hemolytic uremic syndrome (TTP/HUS), torsades de pointes, posterior reversible encephalopathy syndrome (PRES), intracranial hemorrhage, use of emergency transcutaneous pacing, tension pneumothorax, elevated ICP/herniation, and acute mechanical valve failure.

Residents’ self-reported confidence also correlated with level of exposure. There was a significant increase in resident confidence with increasingly independent exposure stratified by level of training (Figure; all with P < .0001). In the multivariable logistic regression model, increasing exposure correlated with increased resident confidence (P < .0001) while controlling for PGY year (P = .001).

DISCUSSION

We developed an instrument to assess resident exposure to and confidence in managing 50 inpatient acute clinical events. Both exposure and level of training were associated with increasing resident confidence. We identified specific events with low levels of exposure and confidence that could be targeted for educational interventions.

To our knowledge, this is the first study to examine IM residents’ exposure to and confidence in managing a wide range of inpatient acute clinical events. A primary goal of residency is to provide physicians-in-training graduated responsibility to prepare them for eventual independent practice. Although our survey confirmed that IM residents’ exposure and confidence significantly increased as they advanced through training (a not unexpected finding), our data also show that even after controlling for year in training, independent exposures significantly correlated with increased confidence. This speaks to the importance of preserving opportunities for residents to manage critical events in a supported manner, an admittedly challenging prospect given the oft-competing calls for supervision of and mentored feedback for trainees.¹¹

Despite identifying independent exposure as an important factor that impacts resident confidence, we found that there was still a substantial proportion of events (28.3%) that senior medical residents near the end of their training had not managed independently in a primary leadership role. Although our study was not designed to determine the reasons for this varied resident exposure, possible explanations may include the relative rarity of certain acute clinical events compared with others, or less likely the effect of duty hour limitations, attending supervision of trainees, or programmatic changes in resident leadership responsibilities. Whatever the cause, this finding uniquely identifies an area for improvement to prevent new attending physicians from feeling unprepared to manage potentially critical emergencies.

An important goal of our study was to develop an instrument that would enable training programs to identify their learning needs. Both program-wide and individual assessments of resident case exposure and confidence are essential for identifying such learning needs and areas for curricular development. Program-wide assessments can spur an important debate about program goals and requirements with respect to what scenarios residents must be able to manage competently by graduation.¹² In addition, such assessments can help individualize learning exposures based on a specific learner’s needs and career goals. The administration of our survey instrument required minimal resources, and the high response rate in our study suggests that other programs can implement our instrument to accomplish these goals.

Alternative methods, such as electronic learning portfolios (efolios), can be utilized to assess resident case exposure. In comparison to our survey instrument, efolios limit recall bias by utilizing case logs and have additional capabilities such as compiling evaluations and enabling trainees to set learning goals. However, there are considerable barriers to the effective use of efolios, including software cost, learner attitudes, and time constraints.¹³ Tools such as our end-of-year assessment offer an alternative method that limits these barriers.

Once educational growth opportunities have been identified through survey-based or other methods, residency programs must determine how to optimize curricula for the needs and career goals of their trainees. We found considerable overlap among conditions that graduating residents had both limited exposure to and low confidence in managing (eg, torsades de pointes, tension pneumothorax, and emergency cardioversion), which are logical topics for future curriculum development. We also identified a few conditions (including PRES, TTP/HUS, and intracranial hemorrhage) that graduating residents did not feel confident in managing despite a relatively higher reported level of exposure. Whether to focus specific educational interventions on the most rare or most commonly encountered acute clinical events is likely to be a topic of debate among individual training programs, but the results of our survey indicate that there is likely to be educational benefit to both strategies.

Residency programs can employ a variety of modalities to enhance learner exposure and confidence in managing clinical scenarios that are deemed important by the program, including didactics, simulation, and changes in program structure. There is a substantial literature on the use of dedicated curricula for crisis management and the use of simulation as a training tool for responding to acute clinical events in multiple specialties^14-24 and in nonmedical domains such as aviation.^25-27 Simulation has been shown to improve residents’ clinical skills and comfort level with some acute events^28-30 and may even be superior to traditional clinical medical education.³¹ In addition, programs can utilize targeted clinical experiences such as intensive care unit and subspecialty rotations^32,33 in an effort to customize educational interventions to fill identified gaps in learner exposure or confidence.

Our study has several limitations. First, we investigated a single large IM residency program at a quaternary academic medical center, and therefore, our findings may not be externally generalizable to all IM residencies or other medical specialties. Our unique peer-led simulation curriculum, including 16 PGY-1 and 8 PGY-2 cases chosen based on clinical rotations at Massachusetts General Hospital,⁷ likely impacted residents’ exposure to simulation that is specific to our institution. However, although specific inpatient acute events may vary among other institutions, our finding that graduating residents still reported gaps in their clinical experience is likely generalizable to other programs given the varied and unpredictable nature of ward medicine training. In addition, our survey tool was simple to administer and could be tailored to reflect the acute events and training needs relevant to other residency programs, specialties, and institutions. Second, the retrospective nature of our study may be subject to participants’ recall bias. We did not restrict our survey questions to urgent conditions managed only on IM hospital wards and some may have been experienced in the emergency room or intensive care units; however, these exposures are still relevant as key components of IM training. Third, our list of 50 acute clinical events was intentionally broad and included several conditions that require multidisciplinary subspecialist consultation, which could have impacted residents’ self-report of “independent” exposures. However, these scenarios are ones that hospitalists may independently recognize and stabilize, engaging appropriate specialists. Fourth, we were not able to validate residents’ self-reported exposures against other measures of the frequency of housestaff management of acute events (such as billing data or patient logs) as this information is not routinely collected. We also did not attempt to identify the reasons underlying the variation seen in resident exposure and confidence for individual acute events, but as a needs assessment, this was beyond the scope of our study. Finally, our assessment of resident confidence was subjective and we were not able to assess competence, with prior studies demonstrating conflicting results regarding the relationship between self-reported proficiency and observed competence.^34-36 Future studies are needed to investigate whether case exposure assessment leads to changes in residency curricula and whether such curricula increase resident confidence and competence in managing hospital acute clinical events.

 

CONCLUSION

We developed an easy-to-administer tool to assess IM residents’ exposure to and confidence in managing inpatient acute events. We found that both significantly increased as residents advanced through training, and self-reported confidence additionally correlated with level of exposure independent of PGY class. We identified several specific inpatient acute clinical events with low levels of resident exposure and confidence that can serve as targets for future IM residency curriculum development. Future studies assessing the impact of such curricula on resident confidence and competence are needed.

Disclosures

The authors declare no conflict of interest.

From the Maine Medical Center, Portland, ME (Dr. Kramer, Ms. Palmeri, Dr. Robich, Mr. Groom, Dr. Hayes, Ms. Janoushek, Dr. Rappold, Dr. Swarz, and Dr. Quinn), and the Center for Outcomes Research and Evaluation, Maine Medical Center Research Institute, Portland, ME (Dr. Lucas).

Abstract

• Objective. To determine the accuracy of the glucometer currently used for point-of-care testing (POCT) of blood glucose in our cardiothoracic surgery intensive care unit (CTICU).
• Design. Prospective cohort study.
• Setting. Tertiary care community hospital affiliated with a school of medicine.
• Participants. Coronary artery bypass graft (CABG) surgery patients.
• Measurements. Blood glucose levels obtained via POCT with a glucometer using fingerstick and radial artery blood samples were compared with values obtained via central laboratory testing of radial artery blood samples (gold standard) in 106 CABG patients on continuous insulin infusions (CII) upon arrival to the CTICU from the operating room and 102 CABG patients on CII in the CTICU 6 hours later.
• Results. Fingerstick POCT and central lab blood glucose values correlated well (r = 0.83 for admission and 0.86 for 6-hour values), but the mean values were significantly different as determined by paired t-tests. Upon arrival, the fingerstick POCT mean value was 120.9 mg/dL, while the central laboratory value was 127.9 mg/dL (P value = 0.03). At the 6-hour time point, the mean value for fingerstick POCT was 129.7 mg/dL compared to a central laboratory value of 137.3 (P value = 0.02).
• Conclusion. The blood glucose POCT values correlated well with central laboratory values, but the values were statistically significantly different. Nevertheless, accurate clinical decisions were made despite the inaccuracies of POCT glucose testing, as experienced bedside nurses were able to use the glucometer successfully and safely. The device’s results informed them when the blood glucose was out of a prescibed range and the direction of the change, and they were able to adjust the CII accordingly.

Keywords: quality improvement; glucose management; point-of-care testing; critical care.

Achieving glycemic control in patients with and without diabetes during coronary artery bypass graft (CABG) surgery is associated with reduced perioperative morbidity and mortality and improved long-term survival.¹ Hyperglycemia has detrimental effects on the cardiovascular system and insulin has beneficial effects on the ischemic myocardium.² The current recommendations of the Society of Thoracic Surgery regarding blood glucose management include the use of continuous insulin infusions (CII) during and after surgery in the critical care unit,³ keeping blood glucose in a moderate range. Glucometers are commonly used in the critical care perioperative setting for point-of-care testing (POCT) for timely determinations of blood glucose levels for patients on CII.

POCT for glucose monitoring is a valuable tool for managing patients with diabetes in the outpatient setting. Evolving from urinary test strips that depended on a colorimetric model, glucometers now incoroporate digital technology that allows patients to determine their blood glucose using a drop of blood from a fingerstick. The US Food and Drug Administration’s approval for most glucose POCT technology includes home use by diabetic patients and use in the hospital setting, with the exception of critically ill patients, who may be affected by hypoxemia, poor capillary perfusion, tissue edema, severe anemia⁴ or other pathophysiologic states that could impact the accuracy of the devices. For example, poor peripheral perfusion related to shock or vasoconstrictors and interstitial edema are variables that could contribute to an erroneous reading. Therefore, many glucometers used in the critical care setting are being used off-label. Because much of the current POCT technology for glucose monitoring may provide erroneous results in certain ranges and in some clinical settings, the safety of most glucometers has been called into question.^5,6

Given the concern regarding the potential inaccuracies of commonly used glucometers in the critical care setting, we undertook a quality improvement project to analyze the clinical performance of the glucometer currently used in our critically ill postoperative cardiac surgery population. The cardiac surgery division policy at our institution is to place all patients, both diabetic and nondiabetic, on a CII intraoperatively and to continue the infusion for at least 24 to 48 hours postoperatively. The CII start rate is determined utilizing the division’s Insulin Start Chart, and then the CII is adjusted according to the nomogram through the postoperative course. Both the Insulin Start Chart and nomogram have been previously described by Kramer et al.⁷

Currently, POCT of glucose in all post cardiac surgery patients is done hourly or more frequently in the first 24 to 48 hours after surgery in order to adjust the CII. In patients undergoing the stress of cardiac surgery, the action of insulin is counter-regulated by glucagon, epinephrine, norepinephrine, cortisol, and growth hormone. The resulting varying degrees of insulin resistance in this population of patients requires close monitoring of blood glucose, keeping it in a prescribed range, which in our center is 110 to 150 mg/dL, both in diabetic and nondiabetic patients. Frequent laboratory and POCT determinations of glucose are made. Providers and bedside nurses adjust the CII according to central laboratory values, POCT values, and trends, as previously described.⁷

 

Methods

Setting

Maine Medical Center is a 600-bed tertiary care teaching hospital. It is a level 1 trauma center where 1000 cardiac surgical operations are performed annually. POCT glucose monitoring is relied upon to monitor blood glucose and adjust the CII accordingly. This project, which did not require any additional procedures outside of the standard of care for this population of patients, was reviewed by the Institutional Review Board, who determined that this activity does not meet either the definition of research as specified under 45 CFR 46.102 (d) or the definition of clinical investigation as specified in 21 CFR 56.102 (c).

Patients

Using central laboratory glucose values drawn from the radial artery as the gold standard, we created a registry of consecutive postoperative cardiac surgery patients who had undergone CABG surgery and had blood glucose determinations from both POCT (fingerstick and radial artery samples) and central laboratory testing (radial artery sample) during a 7-month period (May 2016 through February 2017). To be included in the registry, patients had to (1) be postoperative following isolated CABG or CABG plus Maze procedure; (2) have been on cardiopulmonary bypass (CPB); (3) have radial arterial lines; and (4) be on a CII. A total of 116 patients qualified according to the inclusion criteria. Patients missing glucose results in 1 or more of the variables were excluded from data analysis.

Measurements and Variables

Using a POCT glucometer (FreeStyle Precision Pro, Abbott Laboratories, Abbott Park, IL), blood glucose conentrations were measured on samples obtained from both fingerstick and radial artery. Concurrently, radial arterial blood was sent to the central laboratory for glucose measurement. Blood glucose values were compared in CABG patients on CII upon arrival to the cardiothoracic surgery intensive care unit (CTICU) from the operating room and CABG patients on CII 6 hours after arrival in the CTICU. During the 6-hour interval, blood glucose levels were tested hourly or more frequently, allowing nurses to identify trends in blood glucose changes in order to keep blood glucose in the prescribed goal range of 110 to 150 mg/dL. At each of these 2 time points, on arrival to CTICU and 6 hours later, blood glucose values obtained with radial artery POCT and fingerstick POCT were compared with values obtained with central laboratory testing of radial artery samples. The amount of blood required was 1 drop each for POCT fingerstick and POCT radial artery and 2 mL for central lab testing.

Patient characteristics were identified from the electronic medical record. The variables recorded were type of operation, time on CPB, time of CTICU arrival, temperature, vasoconstrictor infusions (norepinephrine, vasopressin, phenylephrine), preoperative diagnosis of diabetes mellitus, preoperative HbA1c, and hemoglobin/hematocrit. Hemoglobin/hematocrit was only available at the time of the patient’s arrival to CTICU. The study was completed within the confines of our center’s standard of care protocol for postoperative cardiac surgical patients.

Analysis

We used standard statistical techniques to describe the study population, including proportions for categorical variables and means (standard deviations) for continuous variables. Correlation and regression techniques were used to describe the relationship between POCT and laboratory (gold standard) tests, both measured as continuous variables, and paired t-tests with Bonferroni correction were used to compare the central tendency and range of these comparisons. We calculated the differences between the gold standard measure and the POCT measure as an indication of outliers (ie, cases in which the 2 tests gave markedly different results). We examined plots to ascertain at which levels of the gold standard test these outliers occurred. An interim analysis was done at the halfway point and submitted to the Institutional Review Board, but no correction to the P value was done based on this analysis, which was largely qualitative. We used Bonferroni correction to declare a P value of 0.025 statistically significant with the 2-way comparisons of both fingerstick and radial artery values to central laboratory values. When the data was stratified by a clinical characteristic creating a 4-way comparison, we used Bonferroni correction to declare a P value of 0.0125 to be statistically significant when comparing both fingerstick and radial artery values to central laboratory values.

 

Results

Glucose POCT evaluations were carried out on 116 consecutive patients who underwent CABG surgery with or without a Maze procedure on CPB with a CII and an arterial line. Due to missing glucose results in 1 or more of the variables, 10 patients were excluded from data analysis for the time point of arrival in the CTICU and 14 patients were excluded from data analysis for the time point of 6 hours post CTICU arrival. This gave a final count of 106 CABG patients for CTICU arrival data analysis and 102 CABG patients for the 6 hours after CTICU arrival data analysis.

Patients ranged in age from 43 to 85 years, with a mean of age of 66 years, 22% were were women, 41% were diabetic, and 18% had peripheral vascular disease (Table 1). The average preoperative HbA1c was 6.4% ± 1.3% (range, 4.6% to 11.1%). Mean time on CBP for the group was 101 ± 31 minutes (range, 43 to 233 minutes). Postoperative mean hematocrit and hemoglobin were 32.5% and 11.4 g/dL, respectively. The average core temperature of patients on arrival was 36.0°C, which rose to an average of 36.6°C 6 hours later. A vasoconstrictor drip was infusing on 52% of patients upon CTICU arrival; 65% had a vasoconstrictor drip infusing 6 hours after arrival to the CTICU. Hemoglobin results were available only upon CTICU arrival as they are not routinely checked at 6 hours; 74 (64%) patients had a hemoglobin < 12 g/dL.

Compared to central laboratory testing, which we are defining as the gold standard, fingerstick POCT performed better on arrival, while radial artery POCT performed better at 6 hours (Table 2). At CTICU arrival, the mean blood glucose value for fingerstick POCT was 121 ± 24.1 mg/dL, 116 ± 27.2 mg/dL for radial artery POCT, and 128 ± 23.5 mg/dL for central lab testing. The difference in mean blood glucose between the fingerstick POCT and central lab testing was not statistically significant (P = 0.032), while the difference in mean blood glucose between radial artery POCT and central lab testing was statistically significant (P = 0.001). At 6 hours post arrival to the CTICU, the mean fingerstick POCT blood glucose value was 130 ± 23.9 mg/dL, compared to the mean central lab testing value of 137 ± 22.4 mg/dL; this difference was statistically significant (P = 0.019), while the radial artery POCT blood glucose value (133 ± 24.6 mg/dL) was not significantly different from the central lab testing value.

Blood glucose values from fingerstick POCT and central laboratory testing correlated well (r = 0.83 for admission and 0.86 for 6-hour values), as did radial artery POCT and central lab values (r = 0.87 for admission and 0.90 for 6-hour values) (Figures 1, 2, 3, and 4). Comparing individual values for fingerstick POCT and central lab testing, within-person differences between the 2 values ranged from –45 to 25 mg/dL, with 21% of pairs discrepant by 20 mg/dL or more (Figure 1); results were similar at 6 hours (Figure 2), with slightly less discrepancy.

The differences between radial artery POCT and central lab testing values at CTICU arrival ranged from –43 to 80 mg/dL, with 24% of pairs discrepant by 20 mg/dL or more (Figure 3). At 6 hours post CTICU arrival, the difference between radial artery POCT and central lab testing values ranged from –130 to 27 mg/dL, with 11% of pairs discrepant by 20 mg/dL or more (Figure 4). Ninety-two percent of central laboratory values were either close to (± 20) or within the moderate glycemic control target range (110–150 mg/dL).

When the patient cohort was stratified by anemia, diabetes, body temperature, and receipt of vasoconstrictor, there were no significant differences between mean fingerstick POCT and central lab testing values for any strata on CTICU arrival, while there were significant differences between radial artery POCT and central lab testing means for both vasoconstrictor strata as well as for patients with core temperature > 36.1°C (Table 2). At 6 hours, there were no statistically significant differences when stratified for receipt of vasoconstrictor or presence of diabetes. Stratification for anemia or core body temperature was not done for patients at the 6-hour post CTICU arrival time because no hemoglobin value was available and all patients except 1 reached a core temperature of 36.1°C.

Although we measured POCT values obtained using 2 different blood sample sources, fingerstick POCT performed better than radial artery POCT testing with regard to the mean values when compared with the central lab. However, radial artery POCT performed better with regard to correlation with the central lab value. In other words, fingerstick POCT values were less significantly different than radial artery POCT values when compared with the central lab, while radial artery POCT values correlated better with values from the central lab. In spite of this unexplained variability in differences and correlation, the blood glucose values stayed in the target goal range (Figures 1-4).

 

Discussion

The accuracy of glucose POCT in the critical care setting has been called into question.^4,5 The clinical demands of glucose management using CII include timely and accurate guidance in postoperaptive cardiac surgery, in this case, CABG. A previous study compared POCT and central laboratory blood glucose values in medical intensive care unit patients,⁸ but not in patients who have had CABG surgery. Another study has reviewed the difference in glucose values from POCT and central lab analysis in the critically ill population, but not in the post cardiac surgical population.⁹ We have shown that the POCT blood glucose values correlate well with the clinical lab values, but the values are statistically different. Our study adds an additional observation in that, although the POCT inconsistencies were statistically significant, they were not clinically significant. That is, POCT of blood glucose was inaccurate, but it still helped guide care by providing enough information to keep the blood glucose in range (most of the time) and allowing the bedside nurse to detect trends and make appropriate adjustments to the infusion. However, given these inconsistencies, we recommend a low threshold for sending additional samples to the central lab to double-check the glucose values, especially when they are outside the prescribed range. Our analysis provides some measure of reassurance with regard to current postoperative CABG glucose management by showing that the limitations of the blood glucose meter do not jeopardize the safety of patients. Nonetheless, we look forward to advances in the accuracy of POCT blood glucose technology so that critical care patients can be better managed when blood glucose is outside the prescribed range.

This analysis of 116 CABG patients points out both the inaccuracy and the utility of a representative POCT glucometer (in this case, the FreeStyle Precision Pro) used at the bedside to manage CIIs in postoperative CABG patients, keeping the blood glucose level in the moderate control range (110-150 mg/dL). The correlation plot shows that in this population the bedside nurses were able to keep blood glucose in range most of the time, in spite of the inaccuracy of POCT of blood glucose, given that the error of the test fits in the wide margin of 40 mg/dL. The fact that the 6-hour values were slightly less variable than the admission values indicates that sequential determinations of blood glucose over the 6-hour period to detect trends allowed good clinical management even in the face of such inaccuracy. The correlation allows the inaccurate number (blood glucose value) to indicate direction, and frequent determinations allow the bedside nurse to keep that number in the prescribed range most of the time in this population of patients.

Conclusion

We have found that glucometer blood glucose determinations in our center used on a homogenous population (CABG surgery) utilizing a single type of glucometer correlated well with those of the central lab, but were not always accurate. In spite of the inaccuracies, experienced bedside nurses were able to use the instrument successfully and safely, as it informed them if the blood glucose was in or out of a predetermined range and in which direction it was going.

Acknowledgment: The authors are indebted to the nurses of the Cardiothoracic Surgery Intensive Care Unit at Maine Medical Center for their support and assistance, without which this analysis would not have been possible.

Corresponding author: Robert S. Kramer, MD, Division of Cardiothoracic Surgery, Maine Medical Center Cardiovascular Institute, 22 Bramhall St., Portland ME 04102; [email protected].

Financial disclosures: None.

From the Maine Medical Center, Portland, ME (Dr. Kramer, Ms. Palmeri, Dr. Robich, Mr. Groom, Dr. Hayes, Ms. Janoushek, Dr. Rappold, Dr. Swarz, and Dr. Quinn), and the Center for Outcomes Research and Evaluation, Maine Medical Center Research Institute, Portland, ME (Dr. Lucas).

Abstract

• Objective. To determine the accuracy of the glucometer currently used for point-of-care testing (POCT) of blood glucose in our cardiothoracic surgery intensive care unit (CTICU).
• Design. Prospective cohort study.
• Setting. Tertiary care community hospital affiliated with a school of medicine.
• Participants. Coronary artery bypass graft (CABG) surgery patients.
• Measurements. Blood glucose levels obtained via POCT with a glucometer using fingerstick and radial artery blood samples were compared with values obtained via central laboratory testing of radial artery blood samples (gold standard) in 106 CABG patients on continuous insulin infusions (CII) upon arrival to the CTICU from the operating room and 102 CABG patients on CII in the CTICU 6 hours later.
• Results. Fingerstick POCT and central lab blood glucose values correlated well (r = 0.83 for admission and 0.86 for 6-hour values), but the mean values were significantly different as determined by paired t-tests. Upon arrival, the fingerstick POCT mean value was 120.9 mg/dL, while the central laboratory value was 127.9 mg/dL (P value = 0.03). At the 6-hour time point, the mean value for fingerstick POCT was 129.7 mg/dL compared to a central laboratory value of 137.3 (P value = 0.02).
• Conclusion. The blood glucose POCT values correlated well with central laboratory values, but the values were statistically significantly different. Nevertheless, accurate clinical decisions were made despite the inaccuracies of POCT glucose testing, as experienced bedside nurses were able to use the glucometer successfully and safely. The device’s results informed them when the blood glucose was out of a prescibed range and the direction of the change, and they were able to adjust the CII accordingly.

Keywords: quality improvement; glucose management; point-of-care testing; critical care.

Achieving glycemic control in patients with and without diabetes during coronary artery bypass graft (CABG) surgery is associated with reduced perioperative morbidity and mortality and improved long-term survival.¹ Hyperglycemia has detrimental effects on the cardiovascular system and insulin has beneficial effects on the ischemic myocardium.² The current recommendations of the Society of Thoracic Surgery regarding blood glucose management include the use of continuous insulin infusions (CII) during and after surgery in the critical care unit,³ keeping blood glucose in a moderate range. Glucometers are commonly used in the critical care perioperative setting for point-of-care testing (POCT) for timely determinations of blood glucose levels for patients on CII.

POCT for glucose monitoring is a valuable tool for managing patients with diabetes in the outpatient setting. Evolving from urinary test strips that depended on a colorimetric model, glucometers now incoroporate digital technology that allows patients to determine their blood glucose using a drop of blood from a fingerstick. The US Food and Drug Administration’s approval for most glucose POCT technology includes home use by diabetic patients and use in the hospital setting, with the exception of critically ill patients, who may be affected by hypoxemia, poor capillary perfusion, tissue edema, severe anemia⁴ or other pathophysiologic states that could impact the accuracy of the devices. For example, poor peripheral perfusion related to shock or vasoconstrictors and interstitial edema are variables that could contribute to an erroneous reading. Therefore, many glucometers used in the critical care setting are being used off-label. Because much of the current POCT technology for glucose monitoring may provide erroneous results in certain ranges and in some clinical settings, the safety of most glucometers has been called into question.^5,6

Given the concern regarding the potential inaccuracies of commonly used glucometers in the critical care setting, we undertook a quality improvement project to analyze the clinical performance of the glucometer currently used in our critically ill postoperative cardiac surgery population. The cardiac surgery division policy at our institution is to place all patients, both diabetic and nondiabetic, on a CII intraoperatively and to continue the infusion for at least 24 to 48 hours postoperatively. The CII start rate is determined utilizing the division’s Insulin Start Chart, and then the CII is adjusted according to the nomogram through the postoperative course. Both the Insulin Start Chart and nomogram have been previously described by Kramer et al.⁷

Currently, POCT of glucose in all post cardiac surgery patients is done hourly or more frequently in the first 24 to 48 hours after surgery in order to adjust the CII. In patients undergoing the stress of cardiac surgery, the action of insulin is counter-regulated by glucagon, epinephrine, norepinephrine, cortisol, and growth hormone. The resulting varying degrees of insulin resistance in this population of patients requires close monitoring of blood glucose, keeping it in a prescribed range, which in our center is 110 to 150 mg/dL, both in diabetic and nondiabetic patients. Frequent laboratory and POCT determinations of glucose are made. Providers and bedside nurses adjust the CII according to central laboratory values, POCT values, and trends, as previously described.⁷

 

Methods

Setting

Maine Medical Center is a 600-bed tertiary care teaching hospital. It is a level 1 trauma center where 1000 cardiac surgical operations are performed annually. POCT glucose monitoring is relied upon to monitor blood glucose and adjust the CII accordingly. This project, which did not require any additional procedures outside of the standard of care for this population of patients, was reviewed by the Institutional Review Board, who determined that this activity does not meet either the definition of research as specified under 45 CFR 46.102 (d) or the definition of clinical investigation as specified in 21 CFR 56.102 (c).

Patients

Using central laboratory glucose values drawn from the radial artery as the gold standard, we created a registry of consecutive postoperative cardiac surgery patients who had undergone CABG surgery and had blood glucose determinations from both POCT (fingerstick and radial artery samples) and central laboratory testing (radial artery sample) during a 7-month period (May 2016 through February 2017). To be included in the registry, patients had to (1) be postoperative following isolated CABG or CABG plus Maze procedure; (2) have been on cardiopulmonary bypass (CPB); (3) have radial arterial lines; and (4) be on a CII. A total of 116 patients qualified according to the inclusion criteria. Patients missing glucose results in 1 or more of the variables were excluded from data analysis.

Measurements and Variables

Using a POCT glucometer (FreeStyle Precision Pro, Abbott Laboratories, Abbott Park, IL), blood glucose conentrations were measured on samples obtained from both fingerstick and radial artery. Concurrently, radial arterial blood was sent to the central laboratory for glucose measurement. Blood glucose values were compared in CABG patients on CII upon arrival to the cardiothoracic surgery intensive care unit (CTICU) from the operating room and CABG patients on CII 6 hours after arrival in the CTICU. During the 6-hour interval, blood glucose levels were tested hourly or more frequently, allowing nurses to identify trends in blood glucose changes in order to keep blood glucose in the prescribed goal range of 110 to 150 mg/dL. At each of these 2 time points, on arrival to CTICU and 6 hours later, blood glucose values obtained with radial artery POCT and fingerstick POCT were compared with values obtained with central laboratory testing of radial artery samples. The amount of blood required was 1 drop each for POCT fingerstick and POCT radial artery and 2 mL for central lab testing.

Patient characteristics were identified from the electronic medical record. The variables recorded were type of operation, time on CPB, time of CTICU arrival, temperature, vasoconstrictor infusions (norepinephrine, vasopressin, phenylephrine), preoperative diagnosis of diabetes mellitus, preoperative HbA1c, and hemoglobin/hematocrit. Hemoglobin/hematocrit was only available at the time of the patient’s arrival to CTICU. The study was completed within the confines of our center’s standard of care protocol for postoperative cardiac surgical patients.

Analysis

We used standard statistical techniques to describe the study population, including proportions for categorical variables and means (standard deviations) for continuous variables. Correlation and regression techniques were used to describe the relationship between POCT and laboratory (gold standard) tests, both measured as continuous variables, and paired t-tests with Bonferroni correction were used to compare the central tendency and range of these comparisons. We calculated the differences between the gold standard measure and the POCT measure as an indication of outliers (ie, cases in which the 2 tests gave markedly different results). We examined plots to ascertain at which levels of the gold standard test these outliers occurred. An interim analysis was done at the halfway point and submitted to the Institutional Review Board, but no correction to the P value was done based on this analysis, which was largely qualitative. We used Bonferroni correction to declare a P value of 0.025 statistically significant with the 2-way comparisons of both fingerstick and radial artery values to central laboratory values. When the data was stratified by a clinical characteristic creating a 4-way comparison, we used Bonferroni correction to declare a P value of 0.0125 to be statistically significant when comparing both fingerstick and radial artery values to central laboratory values.

 

Results

Glucose POCT evaluations were carried out on 116 consecutive patients who underwent CABG surgery with or without a Maze procedure on CPB with a CII and an arterial line. Due to missing glucose results in 1 or more of the variables, 10 patients were excluded from data analysis for the time point of arrival in the CTICU and 14 patients were excluded from data analysis for the time point of 6 hours post CTICU arrival. This gave a final count of 106 CABG patients for CTICU arrival data analysis and 102 CABG patients for the 6 hours after CTICU arrival data analysis.

Patients ranged in age from 43 to 85 years, with a mean of age of 66 years, 22% were were women, 41% were diabetic, and 18% had peripheral vascular disease (Table 1). The average preoperative HbA1c was 6.4% ± 1.3% (range, 4.6% to 11.1%). Mean time on CBP for the group was 101 ± 31 minutes (range, 43 to 233 minutes). Postoperative mean hematocrit and hemoglobin were 32.5% and 11.4 g/dL, respectively. The average core temperature of patients on arrival was 36.0°C, which rose to an average of 36.6°C 6 hours later. A vasoconstrictor drip was infusing on 52% of patients upon CTICU arrival; 65% had a vasoconstrictor drip infusing 6 hours after arrival to the CTICU. Hemoglobin results were available only upon CTICU arrival as they are not routinely checked at 6 hours; 74 (64%) patients had a hemoglobin < 12 g/dL.

Compared to central laboratory testing, which we are defining as the gold standard, fingerstick POCT performed better on arrival, while radial artery POCT performed better at 6 hours (Table 2). At CTICU arrival, the mean blood glucose value for fingerstick POCT was 121 ± 24.1 mg/dL, 116 ± 27.2 mg/dL for radial artery POCT, and 128 ± 23.5 mg/dL for central lab testing. The difference in mean blood glucose between the fingerstick POCT and central lab testing was not statistically significant (P = 0.032), while the difference in mean blood glucose between radial artery POCT and central lab testing was statistically significant (P = 0.001). At 6 hours post arrival to the CTICU, the mean fingerstick POCT blood glucose value was 130 ± 23.9 mg/dL, compared to the mean central lab testing value of 137 ± 22.4 mg/dL; this difference was statistically significant (P = 0.019), while the radial artery POCT blood glucose value (133 ± 24.6 mg/dL) was not significantly different from the central lab testing value.

Blood glucose values from fingerstick POCT and central laboratory testing correlated well (r = 0.83 for admission and 0.86 for 6-hour values), as did radial artery POCT and central lab values (r = 0.87 for admission and 0.90 for 6-hour values) (Figures 1, 2, 3, and 4). Comparing individual values for fingerstick POCT and central lab testing, within-person differences between the 2 values ranged from –45 to 25 mg/dL, with 21% of pairs discrepant by 20 mg/dL or more (Figure 1); results were similar at 6 hours (Figure 2), with slightly less discrepancy.

The differences between radial artery POCT and central lab testing values at CTICU arrival ranged from –43 to 80 mg/dL, with 24% of pairs discrepant by 20 mg/dL or more (Figure 3). At 6 hours post CTICU arrival, the difference between radial artery POCT and central lab testing values ranged from –130 to 27 mg/dL, with 11% of pairs discrepant by 20 mg/dL or more (Figure 4). Ninety-two percent of central laboratory values were either close to (± 20) or within the moderate glycemic control target range (110–150 mg/dL).

When the patient cohort was stratified by anemia, diabetes, body temperature, and receipt of vasoconstrictor, there were no significant differences between mean fingerstick POCT and central lab testing values for any strata on CTICU arrival, while there were significant differences between radial artery POCT and central lab testing means for both vasoconstrictor strata as well as for patients with core temperature > 36.1°C (Table 2). At 6 hours, there were no statistically significant differences when stratified for receipt of vasoconstrictor or presence of diabetes. Stratification for anemia or core body temperature was not done for patients at the 6-hour post CTICU arrival time because no hemoglobin value was available and all patients except 1 reached a core temperature of 36.1°C.

Although we measured POCT values obtained using 2 different blood sample sources, fingerstick POCT performed better than radial artery POCT testing with regard to the mean values when compared with the central lab. However, radial artery POCT performed better with regard to correlation with the central lab value. In other words, fingerstick POCT values were less significantly different than radial artery POCT values when compared with the central lab, while radial artery POCT values correlated better with values from the central lab. In spite of this unexplained variability in differences and correlation, the blood glucose values stayed in the target goal range (Figures 1-4).

 

Discussion

The accuracy of glucose POCT in the critical care setting has been called into question.^4,5 The clinical demands of glucose management using CII include timely and accurate guidance in postoperaptive cardiac surgery, in this case, CABG. A previous study compared POCT and central laboratory blood glucose values in medical intensive care unit patients,⁸ but not in patients who have had CABG surgery. Another study has reviewed the difference in glucose values from POCT and central lab analysis in the critically ill population, but not in the post cardiac surgical population.⁹ We have shown that the POCT blood glucose values correlate well with the clinical lab values, but the values are statistically different. Our study adds an additional observation in that, although the POCT inconsistencies were statistically significant, they were not clinically significant. That is, POCT of blood glucose was inaccurate, but it still helped guide care by providing enough information to keep the blood glucose in range (most of the time) and allowing the bedside nurse to detect trends and make appropriate adjustments to the infusion. However, given these inconsistencies, we recommend a low threshold for sending additional samples to the central lab to double-check the glucose values, especially when they are outside the prescribed range. Our analysis provides some measure of reassurance with regard to current postoperative CABG glucose management by showing that the limitations of the blood glucose meter do not jeopardize the safety of patients. Nonetheless, we look forward to advances in the accuracy of POCT blood glucose technology so that critical care patients can be better managed when blood glucose is outside the prescribed range.

This analysis of 116 CABG patients points out both the inaccuracy and the utility of a representative POCT glucometer (in this case, the FreeStyle Precision Pro) used at the bedside to manage CIIs in postoperative CABG patients, keeping the blood glucose level in the moderate control range (110-150 mg/dL). The correlation plot shows that in this population the bedside nurses were able to keep blood glucose in range most of the time, in spite of the inaccuracy of POCT of blood glucose, given that the error of the test fits in the wide margin of 40 mg/dL. The fact that the 6-hour values were slightly less variable than the admission values indicates that sequential determinations of blood glucose over the 6-hour period to detect trends allowed good clinical management even in the face of such inaccuracy. The correlation allows the inaccurate number (blood glucose value) to indicate direction, and frequent determinations allow the bedside nurse to keep that number in the prescribed range most of the time in this population of patients.

Conclusion

We have found that glucometer blood glucose determinations in our center used on a homogenous population (CABG surgery) utilizing a single type of glucometer correlated well with those of the central lab, but were not always accurate. In spite of the inaccuracies, experienced bedside nurses were able to use the instrument successfully and safely, as it informed them if the blood glucose was in or out of a predetermined range and in which direction it was going.

Acknowledgment: The authors are indebted to the nurses of the Cardiothoracic Surgery Intensive Care Unit at Maine Medical Center for their support and assistance, without which this analysis would not have been possible.

Corresponding author: Robert S. Kramer, MD, Division of Cardiothoracic Surgery, Maine Medical Center Cardiovascular Institute, 22 Bramhall St., Portland ME 04102; [email protected].

Financial disclosures: None.

From the Maine Medical Center, Portland, ME (Dr. Kramer, Ms. Palmeri, Dr. Robich, Mr. Groom, Dr. Hayes, Ms. Janoushek, Dr. Rappold, Dr. Swarz, and Dr. Quinn), and the Center for Outcomes Research and Evaluation, Maine Medical Center Research Institute, Portland, ME (Dr. Lucas).

Abstract

• Objective. To determine the accuracy of the glucometer currently used for point-of-care testing (POCT) of blood glucose in our cardiothoracic surgery intensive care unit (CTICU).
• Design. Prospective cohort study.
• Setting. Tertiary care community hospital affiliated with a school of medicine.
• Participants. Coronary artery bypass graft (CABG) surgery patients.
• Measurements. Blood glucose levels obtained via POCT with a glucometer using fingerstick and radial artery blood samples were compared with values obtained via central laboratory testing of radial artery blood samples (gold standard) in 106 CABG patients on continuous insulin infusions (CII) upon arrival to the CTICU from the operating room and 102 CABG patients on CII in the CTICU 6 hours later.
• Results. Fingerstick POCT and central lab blood glucose values correlated well (r = 0.83 for admission and 0.86 for 6-hour values), but the mean values were significantly different as determined by paired t-tests. Upon arrival, the fingerstick POCT mean value was 120.9 mg/dL, while the central laboratory value was 127.9 mg/dL (P value = 0.03). At the 6-hour time point, the mean value for fingerstick POCT was 129.7 mg/dL compared to a central laboratory value of 137.3 (P value = 0.02).
• Conclusion. The blood glucose POCT values correlated well with central laboratory values, but the values were statistically significantly different. Nevertheless, accurate clinical decisions were made despite the inaccuracies of POCT glucose testing, as experienced bedside nurses were able to use the glucometer successfully and safely. The device’s results informed them when the blood glucose was out of a prescibed range and the direction of the change, and they were able to adjust the CII accordingly.

Keywords: quality improvement; glucose management; point-of-care testing; critical care.

Achieving glycemic control in patients with and without diabetes during coronary artery bypass graft (CABG) surgery is associated with reduced perioperative morbidity and mortality and improved long-term survival.¹ Hyperglycemia has detrimental effects on the cardiovascular system and insulin has beneficial effects on the ischemic myocardium.² The current recommendations of the Society of Thoracic Surgery regarding blood glucose management include the use of continuous insulin infusions (CII) during and after surgery in the critical care unit,³ keeping blood glucose in a moderate range. Glucometers are commonly used in the critical care perioperative setting for point-of-care testing (POCT) for timely determinations of blood glucose levels for patients on CII.

POCT for glucose monitoring is a valuable tool for managing patients with diabetes in the outpatient setting. Evolving from urinary test strips that depended on a colorimetric model, glucometers now incoroporate digital technology that allows patients to determine their blood glucose using a drop of blood from a fingerstick. The US Food and Drug Administration’s approval for most glucose POCT technology includes home use by diabetic patients and use in the hospital setting, with the exception of critically ill patients, who may be affected by hypoxemia, poor capillary perfusion, tissue edema, severe anemia⁴ or other pathophysiologic states that could impact the accuracy of the devices. For example, poor peripheral perfusion related to shock or vasoconstrictors and interstitial edema are variables that could contribute to an erroneous reading. Therefore, many glucometers used in the critical care setting are being used off-label. Because much of the current POCT technology for glucose monitoring may provide erroneous results in certain ranges and in some clinical settings, the safety of most glucometers has been called into question.^5,6

Given the concern regarding the potential inaccuracies of commonly used glucometers in the critical care setting, we undertook a quality improvement project to analyze the clinical performance of the glucometer currently used in our critically ill postoperative cardiac surgery population. The cardiac surgery division policy at our institution is to place all patients, both diabetic and nondiabetic, on a CII intraoperatively and to continue the infusion for at least 24 to 48 hours postoperatively. The CII start rate is determined utilizing the division’s Insulin Start Chart, and then the CII is adjusted according to the nomogram through the postoperative course. Both the Insulin Start Chart and nomogram have been previously described by Kramer et al.⁷

Currently, POCT of glucose in all post cardiac surgery patients is done hourly or more frequently in the first 24 to 48 hours after surgery in order to adjust the CII. In patients undergoing the stress of cardiac surgery, the action of insulin is counter-regulated by glucagon, epinephrine, norepinephrine, cortisol, and growth hormone. The resulting varying degrees of insulin resistance in this population of patients requires close monitoring of blood glucose, keeping it in a prescribed range, which in our center is 110 to 150 mg/dL, both in diabetic and nondiabetic patients. Frequent laboratory and POCT determinations of glucose are made. Providers and bedside nurses adjust the CII according to central laboratory values, POCT values, and trends, as previously described.⁷

 

Methods

Setting

Maine Medical Center is a 600-bed tertiary care teaching hospital. It is a level 1 trauma center where 1000 cardiac surgical operations are performed annually. POCT glucose monitoring is relied upon to monitor blood glucose and adjust the CII accordingly. This project, which did not require any additional procedures outside of the standard of care for this population of patients, was reviewed by the Institutional Review Board, who determined that this activity does not meet either the definition of research as specified under 45 CFR 46.102 (d) or the definition of clinical investigation as specified in 21 CFR 56.102 (c).

Patients

Using central laboratory glucose values drawn from the radial artery as the gold standard, we created a registry of consecutive postoperative cardiac surgery patients who had undergone CABG surgery and had blood glucose determinations from both POCT (fingerstick and radial artery samples) and central laboratory testing (radial artery sample) during a 7-month period (May 2016 through February 2017). To be included in the registry, patients had to (1) be postoperative following isolated CABG or CABG plus Maze procedure; (2) have been on cardiopulmonary bypass (CPB); (3) have radial arterial lines; and (4) be on a CII. A total of 116 patients qualified according to the inclusion criteria. Patients missing glucose results in 1 or more of the variables were excluded from data analysis.

Measurements and Variables

Using a POCT glucometer (FreeStyle Precision Pro, Abbott Laboratories, Abbott Park, IL), blood glucose conentrations were measured on samples obtained from both fingerstick and radial artery. Concurrently, radial arterial blood was sent to the central laboratory for glucose measurement. Blood glucose values were compared in CABG patients on CII upon arrival to the cardiothoracic surgery intensive care unit (CTICU) from the operating room and CABG patients on CII 6 hours after arrival in the CTICU. During the 6-hour interval, blood glucose levels were tested hourly or more frequently, allowing nurses to identify trends in blood glucose changes in order to keep blood glucose in the prescribed goal range of 110 to 150 mg/dL. At each of these 2 time points, on arrival to CTICU and 6 hours later, blood glucose values obtained with radial artery POCT and fingerstick POCT were compared with values obtained with central laboratory testing of radial artery samples. The amount of blood required was 1 drop each for POCT fingerstick and POCT radial artery and 2 mL for central lab testing.

Patient characteristics were identified from the electronic medical record. The variables recorded were type of operation, time on CPB, time of CTICU arrival, temperature, vasoconstrictor infusions (norepinephrine, vasopressin, phenylephrine), preoperative diagnosis of diabetes mellitus, preoperative HbA1c, and hemoglobin/hematocrit. Hemoglobin/hematocrit was only available at the time of the patient’s arrival to CTICU. The study was completed within the confines of our center’s standard of care protocol for postoperative cardiac surgical patients.

Analysis

We used standard statistical techniques to describe the study population, including proportions for categorical variables and means (standard deviations) for continuous variables. Correlation and regression techniques were used to describe the relationship between POCT and laboratory (gold standard) tests, both measured as continuous variables, and paired t-tests with Bonferroni correction were used to compare the central tendency and range of these comparisons. We calculated the differences between the gold standard measure and the POCT measure as an indication of outliers (ie, cases in which the 2 tests gave markedly different results). We examined plots to ascertain at which levels of the gold standard test these outliers occurred. An interim analysis was done at the halfway point and submitted to the Institutional Review Board, but no correction to the P value was done based on this analysis, which was largely qualitative. We used Bonferroni correction to declare a P value of 0.025 statistically significant with the 2-way comparisons of both fingerstick and radial artery values to central laboratory values. When the data was stratified by a clinical characteristic creating a 4-way comparison, we used Bonferroni correction to declare a P value of 0.0125 to be statistically significant when comparing both fingerstick and radial artery values to central laboratory values.

 

Results

Glucose POCT evaluations were carried out on 116 consecutive patients who underwent CABG surgery with or without a Maze procedure on CPB with a CII and an arterial line. Due to missing glucose results in 1 or more of the variables, 10 patients were excluded from data analysis for the time point of arrival in the CTICU and 14 patients were excluded from data analysis for the time point of 6 hours post CTICU arrival. This gave a final count of 106 CABG patients for CTICU arrival data analysis and 102 CABG patients for the 6 hours after CTICU arrival data analysis.

Patients ranged in age from 43 to 85 years, with a mean of age of 66 years, 22% were were women, 41% were diabetic, and 18% had peripheral vascular disease (Table 1). The average preoperative HbA1c was 6.4% ± 1.3% (range, 4.6% to 11.1%). Mean time on CBP for the group was 101 ± 31 minutes (range, 43 to 233 minutes). Postoperative mean hematocrit and hemoglobin were 32.5% and 11.4 g/dL, respectively. The average core temperature of patients on arrival was 36.0°C, which rose to an average of 36.6°C 6 hours later. A vasoconstrictor drip was infusing on 52% of patients upon CTICU arrival; 65% had a vasoconstrictor drip infusing 6 hours after arrival to the CTICU. Hemoglobin results were available only upon CTICU arrival as they are not routinely checked at 6 hours; 74 (64%) patients had a hemoglobin < 12 g/dL.

Compared to central laboratory testing, which we are defining as the gold standard, fingerstick POCT performed better on arrival, while radial artery POCT performed better at 6 hours (Table 2). At CTICU arrival, the mean blood glucose value for fingerstick POCT was 121 ± 24.1 mg/dL, 116 ± 27.2 mg/dL for radial artery POCT, and 128 ± 23.5 mg/dL for central lab testing. The difference in mean blood glucose between the fingerstick POCT and central lab testing was not statistically significant (P = 0.032), while the difference in mean blood glucose between radial artery POCT and central lab testing was statistically significant (P = 0.001). At 6 hours post arrival to the CTICU, the mean fingerstick POCT blood glucose value was 130 ± 23.9 mg/dL, compared to the mean central lab testing value of 137 ± 22.4 mg/dL; this difference was statistically significant (P = 0.019), while the radial artery POCT blood glucose value (133 ± 24.6 mg/dL) was not significantly different from the central lab testing value.

Blood glucose values from fingerstick POCT and central laboratory testing correlated well (r = 0.83 for admission and 0.86 for 6-hour values), as did radial artery POCT and central lab values (r = 0.87 for admission and 0.90 for 6-hour values) (Figures 1, 2, 3, and 4). Comparing individual values for fingerstick POCT and central lab testing, within-person differences between the 2 values ranged from –45 to 25 mg/dL, with 21% of pairs discrepant by 20 mg/dL or more (Figure 1); results were similar at 6 hours (Figure 2), with slightly less discrepancy.

The differences between radial artery POCT and central lab testing values at CTICU arrival ranged from –43 to 80 mg/dL, with 24% of pairs discrepant by 20 mg/dL or more (Figure 3). At 6 hours post CTICU arrival, the difference between radial artery POCT and central lab testing values ranged from –130 to 27 mg/dL, with 11% of pairs discrepant by 20 mg/dL or more (Figure 4). Ninety-two percent of central laboratory values were either close to (± 20) or within the moderate glycemic control target range (110–150 mg/dL).

When the patient cohort was stratified by anemia, diabetes, body temperature, and receipt of vasoconstrictor, there were no significant differences between mean fingerstick POCT and central lab testing values for any strata on CTICU arrival, while there were significant differences between radial artery POCT and central lab testing means for both vasoconstrictor strata as well as for patients with core temperature > 36.1°C (Table 2). At 6 hours, there were no statistically significant differences when stratified for receipt of vasoconstrictor or presence of diabetes. Stratification for anemia or core body temperature was not done for patients at the 6-hour post CTICU arrival time because no hemoglobin value was available and all patients except 1 reached a core temperature of 36.1°C.

Although we measured POCT values obtained using 2 different blood sample sources, fingerstick POCT performed better than radial artery POCT testing with regard to the mean values when compared with the central lab. However, radial artery POCT performed better with regard to correlation with the central lab value. In other words, fingerstick POCT values were less significantly different than radial artery POCT values when compared with the central lab, while radial artery POCT values correlated better with values from the central lab. In spite of this unexplained variability in differences and correlation, the blood glucose values stayed in the target goal range (Figures 1-4).

 

Discussion

The accuracy of glucose POCT in the critical care setting has been called into question.^4,5 The clinical demands of glucose management using CII include timely and accurate guidance in postoperaptive cardiac surgery, in this case, CABG. A previous study compared POCT and central laboratory blood glucose values in medical intensive care unit patients,⁸ but not in patients who have had CABG surgery. Another study has reviewed the difference in glucose values from POCT and central lab analysis in the critically ill population, but not in the post cardiac surgical population.⁹ We have shown that the POCT blood glucose values correlate well with the clinical lab values, but the values are statistically different. Our study adds an additional observation in that, although the POCT inconsistencies were statistically significant, they were not clinically significant. That is, POCT of blood glucose was inaccurate, but it still helped guide care by providing enough information to keep the blood glucose in range (most of the time) and allowing the bedside nurse to detect trends and make appropriate adjustments to the infusion. However, given these inconsistencies, we recommend a low threshold for sending additional samples to the central lab to double-check the glucose values, especially when they are outside the prescribed range. Our analysis provides some measure of reassurance with regard to current postoperative CABG glucose management by showing that the limitations of the blood glucose meter do not jeopardize the safety of patients. Nonetheless, we look forward to advances in the accuracy of POCT blood glucose technology so that critical care patients can be better managed when blood glucose is outside the prescribed range.

This analysis of 116 CABG patients points out both the inaccuracy and the utility of a representative POCT glucometer (in this case, the FreeStyle Precision Pro) used at the bedside to manage CIIs in postoperative CABG patients, keeping the blood glucose level in the moderate control range (110-150 mg/dL). The correlation plot shows that in this population the bedside nurses were able to keep blood glucose in range most of the time, in spite of the inaccuracy of POCT of blood glucose, given that the error of the test fits in the wide margin of 40 mg/dL. The fact that the 6-hour values were slightly less variable than the admission values indicates that sequential determinations of blood glucose over the 6-hour period to detect trends allowed good clinical management even in the face of such inaccuracy. The correlation allows the inaccurate number (blood glucose value) to indicate direction, and frequent determinations allow the bedside nurse to keep that number in the prescribed range most of the time in this population of patients.

Conclusion

We have found that glucometer blood glucose determinations in our center used on a homogenous population (CABG surgery) utilizing a single type of glucometer correlated well with those of the central lab, but were not always accurate. In spite of the inaccuracies, experienced bedside nurses were able to use the instrument successfully and safely, as it informed them if the blood glucose was in or out of a predetermined range and in which direction it was going.

Acknowledgment: The authors are indebted to the nurses of the Cardiothoracic Surgery Intensive Care Unit at Maine Medical Center for their support and assistance, without which this analysis would not have been possible.

Corresponding author: Robert S. Kramer, MD, Division of Cardiothoracic Surgery, Maine Medical Center Cardiovascular Institute, 22 Bramhall St., Portland ME 04102; [email protected].

Financial disclosures: None.

From the Department of Surgery and Perioperative Care, Dell Medical School, The University of Texas at Austin, Austin, TX (Dr. Kortlever, Ms. Haidar, Dr. Reichel, Dr. Driscoll, Dr. Ring, and Dr. Vagner) and University Medical Center Utrecht, Utrecht, The Netherlands (Dr. Teunis).

Abstract

• Objective: Patient satisfaction is considered a quality measure. Satisfaction is typically measured directly after an in-person visit in research and 2 weeks later in practice surveys. We assessed if there was a difference in immediate and delayed measurement of satisfaction.
• Questions: (1) There is no difference in patient satisfaction (measured by Numerical Rating Scale [NRS]) and (2) perceived empathy (measured by the Jefferson Scale of Patient Perceptions of Physician Empathy [JSPPPE]) immediately after the initial visit compared to 2 weeks later. (3) Change in disability (measured by the Patient-Reported Outcome Measurement Information System Physical Function-Upper Extremity [PROMIS PF-UE]) is not independently associated with change in satisfaction and (4) empathy after the initial visit compared to 2 weeks later.
• Methods: 150 new patients completed a survey of demographics, satisfaction with the surgeon, rating of the surgeon’s empathy, and upper extremity specific limitations. The satisfaction, empathy, and limitation questionnaires were repeated 2 weeks later.
• Results: We found a slight but significant decrease in satisfaction 2 weeks after the in-person visit (–0.41, P = 0.001). There was no significant change in perceived empathy (–0.71, P = 0.19). Change in limitations did not account for a change in satisfaction (P = 0.79) or perceived empathy (P = 0.93).
• Conclusion: Satisfaction and perceived empathy are relatively stable constructs that can be measured immediately after the visit.

Keywords: satisfaction, empathy, change, upper extremity, disability.

Patient satisfaction is increasingly being used as a performance measure to evaluate quality of care.^1-8 Patient satisfaction correlates with adherence with recommended treatment.^1,6,8-10 Satisfaction measured on an 11-point ordinal scale immediately after the visit correlates strongly with the perception of clinician empathy.^2,3 Indeed, some satisfaction questionnaires such as the Medical Interview Satisfaction Scale (MISS)^11,12 have questions very similar to empathy questionnaires. It may be that satisfaction is a construct similar to feeling that your doctor listened and cared about you as an individual (perceived physician empathy).

Higher ratings of satisfaction also seem to be related to a physician’s communication style.^1,4,7-10 One study of 13 fertility doctors found that training in effective communication strategies led to improved patient satisfaction.⁷ A qualitative study of 36 patients, health professionals, and clinical support staff in an orthopaedic outpatient setting held interviews and focus group sessions to identify themes influencing patient satisfaction.⁴ Communication and expectation were among the 7 themes identified. We have noticed a high ceiling effect (maximum scores) with measures of patient satisfaction and perceived empathy.^2,3 Another study also noted a high ceiling effect when using an ordinal scale.⁵ It may be that people with a positive feeling shortly after a health care encounter give top ratings out of politeness or gratefulness. It is also possible they will feel differently a few weeks after they leave the office. Furthermore, ratings of satisfaction gathered by a practice or health care system for practice assessment/improvement are often obtained several days to weeks after the visit, while research often obtains satisfaction ratings immediately after the visit for practical reasons. There may be differences between immediate and delayed measurement of satisfaction beyond the mentioned social norms.

Therefore, this study tested the primary null hypothesis that there is no difference in patient satisfaction (measured by Numerical Rating Scale [NRS]) immediately after the initial visit compared to 2 weeks later. Additionally, we assessed the difference in perceived empathy immediately after the initial visit compared to 2 weeks later, and whether change in disability was independently associated with change in satisfaction and empathy after the initial visit compared to 2 weeks later.

 

Methods

Study Design

After Institutional Review Board approval of this prospective, longitudinal, observational cohort study, we prospectively enrolled 150 adult patients between November 29, 2017 and January 10, 2018. Patients were seen at 5 orthopaedic clinics in a large urban area. We included all new English-speaking patients aged 18 to 89 years who were visiting 1 of 6 participating orthopaedic surgeons for any upper extremity problem and who were able to provide informed consent. We excluded follow-up visits and patients who were unable to speak and understand English. Four research assistants who were not involved with patient treatment described the study to patients before or after the visit with the surgeon. We were granted a waiver of written informed consent; patients indicated their consent by completing the surveys.

Patients could choose either phone or email as their preferred mode of contact for follow-up in this study. For patients who selected email as the preferred mode of contact, the follow-up survey was sent automatically 2 weeks after completion date, and a maximum of 3 reminder emails with 2-day time intervals between them were sent to those who did not respond to the initial invitation. For patients who selected phone as the preferred mode of contact, the follow-up survey was done by an English-speaking research assistant who was not involved with patient treatment. When a response was not obtained on the initial phone call, 3 additional phone calls were made (1 later that same day and 2 the next day). One patient declined participation because he was not interested in the study and had no time after his visit.

Measurements

Patients were asked to complete a set of questionnaires at the end of their visit:

1. A demographic questionnaire consisting of preferred mode of contact for follow-up (phone or email), age, sex, race/ethnicity, marital status, education status, work status, insurance status, and type of visit (first visit or second opinion);

2. An 11-point ordinal measure of satisfaction with the surgeon, with scores ranging from 0 (Worst Surgeon Possible) to 10 (Best Surgeon Possible);

3. The patient’s rating of the surgeon’s empathy, measured by the Jefferson Scale of Patient Perceptions of Physician Empathy (JSPPPE).¹³ The JSPPPE is a 5-item questionnaire, measured on a 7-point Likert scale, with scores ranging from 1 (Strongly Disagree) to 7 (Strongly Agree), that assesses agreement with statements about the physician. The total score is the sum of all item scores (5-35), with higher scores representing a higher degree of perceived physician empathy.

4. Upper extremity disability, measured by the Patient-Reported Outcomes Measurement Information System Physical Function-Upper Extremity (PROMIS PF-UE) Computer Adaptive Test (CAT).^14-16 This is a measure of physical limitations in the upper extremity. It can be completed with as few as 4 questions while still achieving high precision in scoring and thereby decreasing survey burden. PROMIS presents a continuous T-score with a mean of 50 and standard deviation (SD) of 10, with higher scores reflecting better physical function compared to the average of the US general population.¹⁵

After completing the initial questionnaire, the research assistant filled out the office and surgeon name and asked the surgeon to complete the diagnosis. All questionnaires were administered on an encrypted tablet via the secure, HIPAA-compliant electronic platform REDCap (Research Electronic Data Capture), a web-based application for building and managing online surveys and databases.¹⁷ The follow-up survey was sent automatically or was done by phone call as previously described. The follow-up survey consisted of (1) the 11-point ordinal measure of satisfaction with the surgeon, (2) the JSPPPE for perceived empathy, and (3) the PROMIS PF-UE for physical limitations in the upper extremity.

 

Analysis

Continuous variables are presented as mean ± SD and discrete data as proportions. We used Student’s t-tests to assess baseline differences between continuous variables and Fisher’s exact tests for discrete variables. To assess differences in satisfaction and perceived empathy after 2 weeks, we used Student’s paired t-tests. We created 2 multilevel multivariable linear regression models to assess factors associated with (1) change in satisfaction with the surgeon and (2) change in perceived physician empathy. These models account for correlation of patients treated by the same surgeon. We selected variables to be included in the final models by running multilevel models with only 1 independent variable of interest (Appendix 1). Variables with P < 0.10 were included in our final models. We also included change in PROMIS PF-UE in both models because this was our variable of interest. We considered P < 0.05 significant.

We performed a power analysis for the difference in patient satisfaction immediately after the first visit compared to 2 weeks later. Based on our pilot data where we found an initial mean satisfaction score of 9.4 and mean satisfaction score after 2 weeks of 9.1 (SD of difference 1.0), a priori power analysis showed that we needed a minimum sample size of 90 patients to detect a difference with power set at 0.80 and alpha set at 0.05. In order to account for loss to follow-up as previously noted,¹⁸ we enrolled 67% more patients (total of 150).

 

Results

Respondent Characteristics

None of the 150 patients were excluded from the analysis. The study patients’ mean age was 51 ± 16 years (range, 18-87 years), and 73 (49%) were men (Table 1). Mean scores directly after the visit were 9.4 ± 1.2 (range, 2-10) for satisfaction with the surgeon, 31 ± 5.2 (range, 9-35) for perceived physician empathy, and 40 ± 10 (range 15-56) for upper extremity disability. Most patients (n = 130, 87%) were seen in 2 of 5 offices, and 106 (71%) were seen by 2 out of 6 participating surgeons.

Ninety-seven (65%) patients completed their follow-up assessment 2 weeks after their initial visit, 49 (51%) by phone and 48 (49%) by email. This is a slightly better rate than the 36% rate reported in previous research.¹⁸ After 2 weeks, the mean score for satisfaction with the surgeon was 9.1 ± 1.5 (range, 0-10), the mean perceived empathy score was 31 ± 5.1 (range, 6-35), and the mean upper extremity disability score was 40 ± 8.7 (range, 23-56). Responders did not differ from nonresponders based on demographic data (Table 2). However, nonresponders had lower perceived empathy scores directly after their visit (P = 0.03) and none had initially chosen phone as their preferred mode of contact for follow-up (P < 0.001). A list of all diagnoses with frequencies the surgeons stated is listed in Appendix 2.

Difference in Satisfaction with the Surgeon

Satisfaction with the surgeon 2 weeks after the in-person visit was slightly, but significantly, lower on bivariate analysis compared to satisfaction with the surgeon immediately after the initial visit (–0.41 ± 1.2, P = 0.001; Table 3).

Difference in Perceived Physician Empathy

Perceived physician empathy 2 weeks after the in-person visit was not significantly lower on bivariate analysis compared to perceived physician empathy immediately after the initial visit (–0.71 ± 5.3, P = 0.19; Table 3).

Factors Associated with Change in Satisfaction with the Surgeon

Accounting for potential interaction of variables using multilevel multivariable analysis, change in disability of the upper extremity was not associated with change in satisfaction with the surgeon (regression coefficient [beta], 0.00 [95% confidence interval {CI}, –0.02 to 0.03]; standard error [SE], 0.01; P = 0.79 [Table 4]). Being Latino was independently associated with less change in satisfaction with the surgeon (beta coefficient, –0.57 [95% CI, –1.1 to 0.00]; SE, 0.29; P = 0.049).

Factors Associated with Change in Perceived Physician Empathy

Accounting for potential interaction of variables using multilevel multivariable analysis, change in disability of the upper extremity was not associated with change in perceived physician empathy (beta coefficient = 0.00 [95% CI, –0.10 to 0.11]; SE, 0.06; P = 0.93 [Table 4]). Race/ethnicity other than white or Latino was independently associated with more change in perceived physician empathy (beta coefficient, 3.5 [95% CI, 0.34 to 6.6]; SE, 1.6; P = 0.030), and preferring email as mode of contact for follow-up was independently associated with less change in perceived physician empathy (beta coefficient, –3.2 [95% CI, –5.2 to –1.3]; SE, 1.0; P = 0.001).

 

Discussion

Patient satisfaction is considered a quality measure^1-8 and is typically measured directly after an in-person visit. This study tested differences in patient satisfaction and perceived empathy immediately after the initial visit compared to 2 weeks later. In addition, we assessed whether change in disability was independently associated with change in satisfaction and empathy after the initial visit compared to 2 weeks later.

We acknowledge some study limitations. First, we only measured satisfaction based on 1 visit rather than multiple visits over time. It might be that satisfaction ratings differ when the physician-patient relationship is more established. However, we found overall high satisfaction ratings and a well-established relationship might not add to this finding. Second, surgeons were aware of the study and its purpose, which might have resulted in subconsciously altering the behavior to improve satisfaction. The effect of people acting differently as a result of being observed is called the Hawthorne effect.¹⁹ Third, we only used 1 simple ordinal measure to assess patient satisfaction with the surgeon. There is a wide variety of satisfaction measures,²⁰ though the focus of this study was not to test the best possible satisfaction measure but to assess changes in satisfaction over time and its predictors. The simple 11-point ordinal satisfaction measure has proved reliable.⁶ Fourth, 35% of patients did not make a second rating. This is not unusual for phone or email studies. Our response rate was relatively high compared to other studies in our field,¹⁸ perhaps because the time to the second assessment was only 2 weeks and all people were available for follow-up by phone. Fifth, we analyzed 4 surgeons as 1 group and 3 offices as 1 group since we did not enroll enough patients per surgeon and office for individual analysis. However, multilevel linear analysis takes surgeon specific factors into account within that group.

The finding that satisfaction with the surgeon after 2 weeks was significantly lower on bivariate analysis compared to immediately after the initial visit is different from a study that found small increases in satisfaction after 2 weeks and 3 months,¹ but comparable to another study in our field.²¹ Although significant, we believe the decrease in satisfaction is probably not clinically relevant. It might also be that satisfaction at follow-up is lower than measured, but that the least satisfied people did not respond on the follow-up survey.

We found no significant change in perceived empathy after 2 weeks. Since empathy is a strong driver of satisfaction,^2,4-7 we did not expect to find differing results for empathy and for satisfaction over time. Both satisfaction and empathy seem to be relatively durable measures with current measurement tools.

The finding that change in disability was neither independently associated with change in satisfaction nor change in empathy is consistent with prior research.^2,3,21 We cannot adequately study the impact of changes since we did not find an important change in either satisfaction or empathy over time. Jackson et al found higher satisfaction ratings over time in patients who had an increase in physical function and a decrease in symptoms.¹ They also found that met expectations was associated with higher satisfaction immediately after the visit, after 2 weeks, and after 3 months.¹ We feel that met expectations and fewer symptoms and limitations are likely highly co-linear with satisfaction. We therefore may not be able to learn much about one from the others.

The slight change we found in satisfaction with the surgeon among Latino patients was significantly less than the change among white patients. This suggests Latino patients might have a more stable opinion over time (a cultural phenomenon), or it might be spurious given the small number of Latino patients included in the study. The same can be said for the finding that race/ethnicity other than white or Latino was independently associated with greater change in empathy. Providing email as the preferred mode of contact was found to be independently associated with less change in perceived empathy compared to follow-up by phone. We had a 100% success rate for our follow-ups by phone. Our findings suggest that patients might more easily switch ratings on an 11-point ordinal scale than on a 5-item Likert scale. However, both measures are often rated at the ceiling of the scale.^2,21

 

Conclusion

Satisfaction and perceived empathy are relatively stable constructs, are not clearly associated with other factors, and are strongly correlated with one another. This study supports the research practice of measuring satisfaction immediately after the visit, which is more convenient for both participant and researcher and avoids the loss of more than one third of the patients, and those with a worse experience in particular. To improve the utility and interpretation of patient-reported experience measures such as these, we might direct our efforts to developing scales with less ceiling effect.

Corresponding author: David Ring, MD, PhD, Dell Medical School, The University of Texas at Austin, Health Discovery Building HDB 6.706, 1701 Trinity St., Austin, TX 78705; [email protected].

Financial disclosures: Dr. Ring has or may receive payment or benefits from Skeletal Dynamics, Wright Medical for elbow implants, Deputy Editor for Clinical Orthopaedics and Related Research, Universities and Hospitals, Lawyers outside the submitted work.

Dr. Teunis has or may receive payment or benefits from VCC, PATIENT+, and AO Trauma TK network unrelated to this work and consultant fees from Synthes.

From the Department of Surgery and Perioperative Care, Dell Medical School, The University of Texas at Austin, Austin, TX (Dr. Kortlever, Ms. Haidar, Dr. Reichel, Dr. Driscoll, Dr. Ring, and Dr. Vagner) and University Medical Center Utrecht, Utrecht, The Netherlands (Dr. Teunis).

Abstract

• Objective: Patient satisfaction is considered a quality measure. Satisfaction is typically measured directly after an in-person visit in research and 2 weeks later in practice surveys. We assessed if there was a difference in immediate and delayed measurement of satisfaction.
• Questions: (1) There is no difference in patient satisfaction (measured by Numerical Rating Scale [NRS]) and (2) perceived empathy (measured by the Jefferson Scale of Patient Perceptions of Physician Empathy [JSPPPE]) immediately after the initial visit compared to 2 weeks later. (3) Change in disability (measured by the Patient-Reported Outcome Measurement Information System Physical Function-Upper Extremity [PROMIS PF-UE]) is not independently associated with change in satisfaction and (4) empathy after the initial visit compared to 2 weeks later.
• Methods: 150 new patients completed a survey of demographics, satisfaction with the surgeon, rating of the surgeon’s empathy, and upper extremity specific limitations. The satisfaction, empathy, and limitation questionnaires were repeated 2 weeks later.
• Results: We found a slight but significant decrease in satisfaction 2 weeks after the in-person visit (–0.41, P = 0.001). There was no significant change in perceived empathy (–0.71, P = 0.19). Change in limitations did not account for a change in satisfaction (P = 0.79) or perceived empathy (P = 0.93).
• Conclusion: Satisfaction and perceived empathy are relatively stable constructs that can be measured immediately after the visit.

Keywords: satisfaction, empathy, change, upper extremity, disability.

Patient satisfaction is increasingly being used as a performance measure to evaluate quality of care.^1-8 Patient satisfaction correlates with adherence with recommended treatment.^1,6,8-10 Satisfaction measured on an 11-point ordinal scale immediately after the visit correlates strongly with the perception of clinician empathy.^2,3 Indeed, some satisfaction questionnaires such as the Medical Interview Satisfaction Scale (MISS)^11,12 have questions very similar to empathy questionnaires. It may be that satisfaction is a construct similar to feeling that your doctor listened and cared about you as an individual (perceived physician empathy).

Higher ratings of satisfaction also seem to be related to a physician’s communication style.^1,4,7-10 One study of 13 fertility doctors found that training in effective communication strategies led to improved patient satisfaction.⁷ A qualitative study of 36 patients, health professionals, and clinical support staff in an orthopaedic outpatient setting held interviews and focus group sessions to identify themes influencing patient satisfaction.⁴ Communication and expectation were among the 7 themes identified. We have noticed a high ceiling effect (maximum scores) with measures of patient satisfaction and perceived empathy.^2,3 Another study also noted a high ceiling effect when using an ordinal scale.⁵ It may be that people with a positive feeling shortly after a health care encounter give top ratings out of politeness or gratefulness. It is also possible they will feel differently a few weeks after they leave the office. Furthermore, ratings of satisfaction gathered by a practice or health care system for practice assessment/improvement are often obtained several days to weeks after the visit, while research often obtains satisfaction ratings immediately after the visit for practical reasons. There may be differences between immediate and delayed measurement of satisfaction beyond the mentioned social norms.

Therefore, this study tested the primary null hypothesis that there is no difference in patient satisfaction (measured by Numerical Rating Scale [NRS]) immediately after the initial visit compared to 2 weeks later. Additionally, we assessed the difference in perceived empathy immediately after the initial visit compared to 2 weeks later, and whether change in disability was independently associated with change in satisfaction and empathy after the initial visit compared to 2 weeks later.

 

Methods

Study Design

After Institutional Review Board approval of this prospective, longitudinal, observational cohort study, we prospectively enrolled 150 adult patients between November 29, 2017 and January 10, 2018. Patients were seen at 5 orthopaedic clinics in a large urban area. We included all new English-speaking patients aged 18 to 89 years who were visiting 1 of 6 participating orthopaedic surgeons for any upper extremity problem and who were able to provide informed consent. We excluded follow-up visits and patients who were unable to speak and understand English. Four research assistants who were not involved with patient treatment described the study to patients before or after the visit with the surgeon. We were granted a waiver of written informed consent; patients indicated their consent by completing the surveys.

Patients could choose either phone or email as their preferred mode of contact for follow-up in this study. For patients who selected email as the preferred mode of contact, the follow-up survey was sent automatically 2 weeks after completion date, and a maximum of 3 reminder emails with 2-day time intervals between them were sent to those who did not respond to the initial invitation. For patients who selected phone as the preferred mode of contact, the follow-up survey was done by an English-speaking research assistant who was not involved with patient treatment. When a response was not obtained on the initial phone call, 3 additional phone calls were made (1 later that same day and 2 the next day). One patient declined participation because he was not interested in the study and had no time after his visit.

Measurements

Patients were asked to complete a set of questionnaires at the end of their visit:

1. A demographic questionnaire consisting of preferred mode of contact for follow-up (phone or email), age, sex, race/ethnicity, marital status, education status, work status, insurance status, and type of visit (first visit or second opinion);

2. An 11-point ordinal measure of satisfaction with the surgeon, with scores ranging from 0 (Worst Surgeon Possible) to 10 (Best Surgeon Possible);

3. The patient’s rating of the surgeon’s empathy, measured by the Jefferson Scale of Patient Perceptions of Physician Empathy (JSPPPE).¹³ The JSPPPE is a 5-item questionnaire, measured on a 7-point Likert scale, with scores ranging from 1 (Strongly Disagree) to 7 (Strongly Agree), that assesses agreement with statements about the physician. The total score is the sum of all item scores (5-35), with higher scores representing a higher degree of perceived physician empathy.

4. Upper extremity disability, measured by the Patient-Reported Outcomes Measurement Information System Physical Function-Upper Extremity (PROMIS PF-UE) Computer Adaptive Test (CAT).^14-16 This is a measure of physical limitations in the upper extremity. It can be completed with as few as 4 questions while still achieving high precision in scoring and thereby decreasing survey burden. PROMIS presents a continuous T-score with a mean of 50 and standard deviation (SD) of 10, with higher scores reflecting better physical function compared to the average of the US general population.¹⁵

After completing the initial questionnaire, the research assistant filled out the office and surgeon name and asked the surgeon to complete the diagnosis. All questionnaires were administered on an encrypted tablet via the secure, HIPAA-compliant electronic platform REDCap (Research Electronic Data Capture), a web-based application for building and managing online surveys and databases.¹⁷ The follow-up survey was sent automatically or was done by phone call as previously described. The follow-up survey consisted of (1) the 11-point ordinal measure of satisfaction with the surgeon, (2) the JSPPPE for perceived empathy, and (3) the PROMIS PF-UE for physical limitations in the upper extremity.

 

Analysis

Continuous variables are presented as mean ± SD and discrete data as proportions. We used Student’s t-tests to assess baseline differences between continuous variables and Fisher’s exact tests for discrete variables. To assess differences in satisfaction and perceived empathy after 2 weeks, we used Student’s paired t-tests. We created 2 multilevel multivariable linear regression models to assess factors associated with (1) change in satisfaction with the surgeon and (2) change in perceived physician empathy. These models account for correlation of patients treated by the same surgeon. We selected variables to be included in the final models by running multilevel models with only 1 independent variable of interest (Appendix 1). Variables with P < 0.10 were included in our final models. We also included change in PROMIS PF-UE in both models because this was our variable of interest. We considered P < 0.05 significant.

We performed a power analysis for the difference in patient satisfaction immediately after the first visit compared to 2 weeks later. Based on our pilot data where we found an initial mean satisfaction score of 9.4 and mean satisfaction score after 2 weeks of 9.1 (SD of difference 1.0), a priori power analysis showed that we needed a minimum sample size of 90 patients to detect a difference with power set at 0.80 and alpha set at 0.05. In order to account for loss to follow-up as previously noted,¹⁸ we enrolled 67% more patients (total of 150).

 

Results

Respondent Characteristics

None of the 150 patients were excluded from the analysis. The study patients’ mean age was 51 ± 16 years (range, 18-87 years), and 73 (49%) were men (Table 1). Mean scores directly after the visit were 9.4 ± 1.2 (range, 2-10) for satisfaction with the surgeon, 31 ± 5.2 (range, 9-35) for perceived physician empathy, and 40 ± 10 (range 15-56) for upper extremity disability. Most patients (n = 130, 87%) were seen in 2 of 5 offices, and 106 (71%) were seen by 2 out of 6 participating surgeons.

Ninety-seven (65%) patients completed their follow-up assessment 2 weeks after their initial visit, 49 (51%) by phone and 48 (49%) by email. This is a slightly better rate than the 36% rate reported in previous research.¹⁸ After 2 weeks, the mean score for satisfaction with the surgeon was 9.1 ± 1.5 (range, 0-10), the mean perceived empathy score was 31 ± 5.1 (range, 6-35), and the mean upper extremity disability score was 40 ± 8.7 (range, 23-56). Responders did not differ from nonresponders based on demographic data (Table 2). However, nonresponders had lower perceived empathy scores directly after their visit (P = 0.03) and none had initially chosen phone as their preferred mode of contact for follow-up (P < 0.001). A list of all diagnoses with frequencies the surgeons stated is listed in Appendix 2.

Difference in Satisfaction with the Surgeon

Satisfaction with the surgeon 2 weeks after the in-person visit was slightly, but significantly, lower on bivariate analysis compared to satisfaction with the surgeon immediately after the initial visit (–0.41 ± 1.2, P = 0.001; Table 3).

Difference in Perceived Physician Empathy

Perceived physician empathy 2 weeks after the in-person visit was not significantly lower on bivariate analysis compared to perceived physician empathy immediately after the initial visit (–0.71 ± 5.3, P = 0.19; Table 3).

Factors Associated with Change in Satisfaction with the Surgeon

Accounting for potential interaction of variables using multilevel multivariable analysis, change in disability of the upper extremity was not associated with change in satisfaction with the surgeon (regression coefficient [beta], 0.00 [95% confidence interval {CI}, –0.02 to 0.03]; standard error [SE], 0.01; P = 0.79 [Table 4]). Being Latino was independently associated with less change in satisfaction with the surgeon (beta coefficient, –0.57 [95% CI, –1.1 to 0.00]; SE, 0.29; P = 0.049).

Factors Associated with Change in Perceived Physician Empathy

Accounting for potential interaction of variables using multilevel multivariable analysis, change in disability of the upper extremity was not associated with change in perceived physician empathy (beta coefficient = 0.00 [95% CI, –0.10 to 0.11]; SE, 0.06; P = 0.93 [Table 4]). Race/ethnicity other than white or Latino was independently associated with more change in perceived physician empathy (beta coefficient, 3.5 [95% CI, 0.34 to 6.6]; SE, 1.6; P = 0.030), and preferring email as mode of contact for follow-up was independently associated with less change in perceived physician empathy (beta coefficient, –3.2 [95% CI, –5.2 to –1.3]; SE, 1.0; P = 0.001).

 

Discussion

Patient satisfaction is considered a quality measure^1-8 and is typically measured directly after an in-person visit. This study tested differences in patient satisfaction and perceived empathy immediately after the initial visit compared to 2 weeks later. In addition, we assessed whether change in disability was independently associated with change in satisfaction and empathy after the initial visit compared to 2 weeks later.

We acknowledge some study limitations. First, we only measured satisfaction based on 1 visit rather than multiple visits over time. It might be that satisfaction ratings differ when the physician-patient relationship is more established. However, we found overall high satisfaction ratings and a well-established relationship might not add to this finding. Second, surgeons were aware of the study and its purpose, which might have resulted in subconsciously altering the behavior to improve satisfaction. The effect of people acting differently as a result of being observed is called the Hawthorne effect.¹⁹ Third, we only used 1 simple ordinal measure to assess patient satisfaction with the surgeon. There is a wide variety of satisfaction measures,²⁰ though the focus of this study was not to test the best possible satisfaction measure but to assess changes in satisfaction over time and its predictors. The simple 11-point ordinal satisfaction measure has proved reliable.⁶ Fourth, 35% of patients did not make a second rating. This is not unusual for phone or email studies. Our response rate was relatively high compared to other studies in our field,¹⁸ perhaps because the time to the second assessment was only 2 weeks and all people were available for follow-up by phone. Fifth, we analyzed 4 surgeons as 1 group and 3 offices as 1 group since we did not enroll enough patients per surgeon and office for individual analysis. However, multilevel linear analysis takes surgeon specific factors into account within that group.

The finding that satisfaction with the surgeon after 2 weeks was significantly lower on bivariate analysis compared to immediately after the initial visit is different from a study that found small increases in satisfaction after 2 weeks and 3 months,¹ but comparable to another study in our field.²¹ Although significant, we believe the decrease in satisfaction is probably not clinically relevant. It might also be that satisfaction at follow-up is lower than measured, but that the least satisfied people did not respond on the follow-up survey.

We found no significant change in perceived empathy after 2 weeks. Since empathy is a strong driver of satisfaction,^2,4-7 we did not expect to find differing results for empathy and for satisfaction over time. Both satisfaction and empathy seem to be relatively durable measures with current measurement tools.

The finding that change in disability was neither independently associated with change in satisfaction nor change in empathy is consistent with prior research.^2,3,21 We cannot adequately study the impact of changes since we did not find an important change in either satisfaction or empathy over time. Jackson et al found higher satisfaction ratings over time in patients who had an increase in physical function and a decrease in symptoms.¹ They also found that met expectations was associated with higher satisfaction immediately after the visit, after 2 weeks, and after 3 months.¹ We feel that met expectations and fewer symptoms and limitations are likely highly co-linear with satisfaction. We therefore may not be able to learn much about one from the others.

The slight change we found in satisfaction with the surgeon among Latino patients was significantly less than the change among white patients. This suggests Latino patients might have a more stable opinion over time (a cultural phenomenon), or it might be spurious given the small number of Latino patients included in the study. The same can be said for the finding that race/ethnicity other than white or Latino was independently associated with greater change in empathy. Providing email as the preferred mode of contact was found to be independently associated with less change in perceived empathy compared to follow-up by phone. We had a 100% success rate for our follow-ups by phone. Our findings suggest that patients might more easily switch ratings on an 11-point ordinal scale than on a 5-item Likert scale. However, both measures are often rated at the ceiling of the scale.^2,21

 

Conclusion

Satisfaction and perceived empathy are relatively stable constructs, are not clearly associated with other factors, and are strongly correlated with one another. This study supports the research practice of measuring satisfaction immediately after the visit, which is more convenient for both participant and researcher and avoids the loss of more than one third of the patients, and those with a worse experience in particular. To improve the utility and interpretation of patient-reported experience measures such as these, we might direct our efforts to developing scales with less ceiling effect.

Corresponding author: David Ring, MD, PhD, Dell Medical School, The University of Texas at Austin, Health Discovery Building HDB 6.706, 1701 Trinity St., Austin, TX 78705; [email protected].

Financial disclosures: Dr. Ring has or may receive payment or benefits from Skeletal Dynamics, Wright Medical for elbow implants, Deputy Editor for Clinical Orthopaedics and Related Research, Universities and Hospitals, Lawyers outside the submitted work.

Dr. Teunis has or may receive payment or benefits from VCC, PATIENT+, and AO Trauma TK network unrelated to this work and consultant fees from Synthes.

From the Department of Surgery and Perioperative Care, Dell Medical School, The University of Texas at Austin, Austin, TX (Dr. Kortlever, Ms. Haidar, Dr. Reichel, Dr. Driscoll, Dr. Ring, and Dr. Vagner) and University Medical Center Utrecht, Utrecht, The Netherlands (Dr. Teunis).

Abstract

• Objective: Patient satisfaction is considered a quality measure. Satisfaction is typically measured directly after an in-person visit in research and 2 weeks later in practice surveys. We assessed if there was a difference in immediate and delayed measurement of satisfaction.
• Questions: (1) There is no difference in patient satisfaction (measured by Numerical Rating Scale [NRS]) and (2) perceived empathy (measured by the Jefferson Scale of Patient Perceptions of Physician Empathy [JSPPPE]) immediately after the initial visit compared to 2 weeks later. (3) Change in disability (measured by the Patient-Reported Outcome Measurement Information System Physical Function-Upper Extremity [PROMIS PF-UE]) is not independently associated with change in satisfaction and (4) empathy after the initial visit compared to 2 weeks later.
• Methods: 150 new patients completed a survey of demographics, satisfaction with the surgeon, rating of the surgeon’s empathy, and upper extremity specific limitations. The satisfaction, empathy, and limitation questionnaires were repeated 2 weeks later.
• Results: We found a slight but significant decrease in satisfaction 2 weeks after the in-person visit (–0.41, P = 0.001). There was no significant change in perceived empathy (–0.71, P = 0.19). Change in limitations did not account for a change in satisfaction (P = 0.79) or perceived empathy (P = 0.93).
• Conclusion: Satisfaction and perceived empathy are relatively stable constructs that can be measured immediately after the visit.

Keywords: satisfaction, empathy, change, upper extremity, disability.

Patient satisfaction is increasingly being used as a performance measure to evaluate quality of care.^1-8 Patient satisfaction correlates with adherence with recommended treatment.^1,6,8-10 Satisfaction measured on an 11-point ordinal scale immediately after the visit correlates strongly with the perception of clinician empathy.^2,3 Indeed, some satisfaction questionnaires such as the Medical Interview Satisfaction Scale (MISS)^11,12 have questions very similar to empathy questionnaires. It may be that satisfaction is a construct similar to feeling that your doctor listened and cared about you as an individual (perceived physician empathy).

Higher ratings of satisfaction also seem to be related to a physician’s communication style.^1,4,7-10 One study of 13 fertility doctors found that training in effective communication strategies led to improved patient satisfaction.⁷ A qualitative study of 36 patients, health professionals, and clinical support staff in an orthopaedic outpatient setting held interviews and focus group sessions to identify themes influencing patient satisfaction.⁴ Communication and expectation were among the 7 themes identified. We have noticed a high ceiling effect (maximum scores) with measures of patient satisfaction and perceived empathy.^2,3 Another study also noted a high ceiling effect when using an ordinal scale.⁵ It may be that people with a positive feeling shortly after a health care encounter give top ratings out of politeness or gratefulness. It is also possible they will feel differently a few weeks after they leave the office. Furthermore, ratings of satisfaction gathered by a practice or health care system for practice assessment/improvement are often obtained several days to weeks after the visit, while research often obtains satisfaction ratings immediately after the visit for practical reasons. There may be differences between immediate and delayed measurement of satisfaction beyond the mentioned social norms.

Therefore, this study tested the primary null hypothesis that there is no difference in patient satisfaction (measured by Numerical Rating Scale [NRS]) immediately after the initial visit compared to 2 weeks later. Additionally, we assessed the difference in perceived empathy immediately after the initial visit compared to 2 weeks later, and whether change in disability was independently associated with change in satisfaction and empathy after the initial visit compared to 2 weeks later.

 

Methods

Study Design

After Institutional Review Board approval of this prospective, longitudinal, observational cohort study, we prospectively enrolled 150 adult patients between November 29, 2017 and January 10, 2018. Patients were seen at 5 orthopaedic clinics in a large urban area. We included all new English-speaking patients aged 18 to 89 years who were visiting 1 of 6 participating orthopaedic surgeons for any upper extremity problem and who were able to provide informed consent. We excluded follow-up visits and patients who were unable to speak and understand English. Four research assistants who were not involved with patient treatment described the study to patients before or after the visit with the surgeon. We were granted a waiver of written informed consent; patients indicated their consent by completing the surveys.

Patients could choose either phone or email as their preferred mode of contact for follow-up in this study. For patients who selected email as the preferred mode of contact, the follow-up survey was sent automatically 2 weeks after completion date, and a maximum of 3 reminder emails with 2-day time intervals between them were sent to those who did not respond to the initial invitation. For patients who selected phone as the preferred mode of contact, the follow-up survey was done by an English-speaking research assistant who was not involved with patient treatment. When a response was not obtained on the initial phone call, 3 additional phone calls were made (1 later that same day and 2 the next day). One patient declined participation because he was not interested in the study and had no time after his visit.

Measurements

Patients were asked to complete a set of questionnaires at the end of their visit:

1. A demographic questionnaire consisting of preferred mode of contact for follow-up (phone or email), age, sex, race/ethnicity, marital status, education status, work status, insurance status, and type of visit (first visit or second opinion);

2. An 11-point ordinal measure of satisfaction with the surgeon, with scores ranging from 0 (Worst Surgeon Possible) to 10 (Best Surgeon Possible);

3. The patient’s rating of the surgeon’s empathy, measured by the Jefferson Scale of Patient Perceptions of Physician Empathy (JSPPPE).¹³ The JSPPPE is a 5-item questionnaire, measured on a 7-point Likert scale, with scores ranging from 1 (Strongly Disagree) to 7 (Strongly Agree), that assesses agreement with statements about the physician. The total score is the sum of all item scores (5-35), with higher scores representing a higher degree of perceived physician empathy.

4. Upper extremity disability, measured by the Patient-Reported Outcomes Measurement Information System Physical Function-Upper Extremity (PROMIS PF-UE) Computer Adaptive Test (CAT).^14-16 This is a measure of physical limitations in the upper extremity. It can be completed with as few as 4 questions while still achieving high precision in scoring and thereby decreasing survey burden. PROMIS presents a continuous T-score with a mean of 50 and standard deviation (SD) of 10, with higher scores reflecting better physical function compared to the average of the US general population.¹⁵

After completing the initial questionnaire, the research assistant filled out the office and surgeon name and asked the surgeon to complete the diagnosis. All questionnaires were administered on an encrypted tablet via the secure, HIPAA-compliant electronic platform REDCap (Research Electronic Data Capture), a web-based application for building and managing online surveys and databases.¹⁷ The follow-up survey was sent automatically or was done by phone call as previously described. The follow-up survey consisted of (1) the 11-point ordinal measure of satisfaction with the surgeon, (2) the JSPPPE for perceived empathy, and (3) the PROMIS PF-UE for physical limitations in the upper extremity.

 

Analysis

Continuous variables are presented as mean ± SD and discrete data as proportions. We used Student’s t-tests to assess baseline differences between continuous variables and Fisher’s exact tests for discrete variables. To assess differences in satisfaction and perceived empathy after 2 weeks, we used Student’s paired t-tests. We created 2 multilevel multivariable linear regression models to assess factors associated with (1) change in satisfaction with the surgeon and (2) change in perceived physician empathy. These models account for correlation of patients treated by the same surgeon. We selected variables to be included in the final models by running multilevel models with only 1 independent variable of interest (Appendix 1). Variables with P < 0.10 were included in our final models. We also included change in PROMIS PF-UE in both models because this was our variable of interest. We considered P < 0.05 significant.

We performed a power analysis for the difference in patient satisfaction immediately after the first visit compared to 2 weeks later. Based on our pilot data where we found an initial mean satisfaction score of 9.4 and mean satisfaction score after 2 weeks of 9.1 (SD of difference 1.0), a priori power analysis showed that we needed a minimum sample size of 90 patients to detect a difference with power set at 0.80 and alpha set at 0.05. In order to account for loss to follow-up as previously noted,¹⁸ we enrolled 67% more patients (total of 150).

 

Results

Respondent Characteristics

None of the 150 patients were excluded from the analysis. The study patients’ mean age was 51 ± 16 years (range, 18-87 years), and 73 (49%) were men (Table 1). Mean scores directly after the visit were 9.4 ± 1.2 (range, 2-10) for satisfaction with the surgeon, 31 ± 5.2 (range, 9-35) for perceived physician empathy, and 40 ± 10 (range 15-56) for upper extremity disability. Most patients (n = 130, 87%) were seen in 2 of 5 offices, and 106 (71%) were seen by 2 out of 6 participating surgeons.

Ninety-seven (65%) patients completed their follow-up assessment 2 weeks after their initial visit, 49 (51%) by phone and 48 (49%) by email. This is a slightly better rate than the 36% rate reported in previous research.¹⁸ After 2 weeks, the mean score for satisfaction with the surgeon was 9.1 ± 1.5 (range, 0-10), the mean perceived empathy score was 31 ± 5.1 (range, 6-35), and the mean upper extremity disability score was 40 ± 8.7 (range, 23-56). Responders did not differ from nonresponders based on demographic data (Table 2). However, nonresponders had lower perceived empathy scores directly after their visit (P = 0.03) and none had initially chosen phone as their preferred mode of contact for follow-up (P < 0.001). A list of all diagnoses with frequencies the surgeons stated is listed in Appendix 2.

Difference in Satisfaction with the Surgeon

Satisfaction with the surgeon 2 weeks after the in-person visit was slightly, but significantly, lower on bivariate analysis compared to satisfaction with the surgeon immediately after the initial visit (–0.41 ± 1.2, P = 0.001; Table 3).

Difference in Perceived Physician Empathy

Perceived physician empathy 2 weeks after the in-person visit was not significantly lower on bivariate analysis compared to perceived physician empathy immediately after the initial visit (–0.71 ± 5.3, P = 0.19; Table 3).

Factors Associated with Change in Satisfaction with the Surgeon

Accounting for potential interaction of variables using multilevel multivariable analysis, change in disability of the upper extremity was not associated with change in satisfaction with the surgeon (regression coefficient [beta], 0.00 [95% confidence interval {CI}, –0.02 to 0.03]; standard error [SE], 0.01; P = 0.79 [Table 4]). Being Latino was independently associated with less change in satisfaction with the surgeon (beta coefficient, –0.57 [95% CI, –1.1 to 0.00]; SE, 0.29; P = 0.049).

Factors Associated with Change in Perceived Physician Empathy

Accounting for potential interaction of variables using multilevel multivariable analysis, change in disability of the upper extremity was not associated with change in perceived physician empathy (beta coefficient = 0.00 [95% CI, –0.10 to 0.11]; SE, 0.06; P = 0.93 [Table 4]). Race/ethnicity other than white or Latino was independently associated with more change in perceived physician empathy (beta coefficient, 3.5 [95% CI, 0.34 to 6.6]; SE, 1.6; P = 0.030), and preferring email as mode of contact for follow-up was independently associated with less change in perceived physician empathy (beta coefficient, –3.2 [95% CI, –5.2 to –1.3]; SE, 1.0; P = 0.001).

 

Discussion

Patient satisfaction is considered a quality measure^1-8 and is typically measured directly after an in-person visit. This study tested differences in patient satisfaction and perceived empathy immediately after the initial visit compared to 2 weeks later. In addition, we assessed whether change in disability was independently associated with change in satisfaction and empathy after the initial visit compared to 2 weeks later.

We acknowledge some study limitations. First, we only measured satisfaction based on 1 visit rather than multiple visits over time. It might be that satisfaction ratings differ when the physician-patient relationship is more established. However, we found overall high satisfaction ratings and a well-established relationship might not add to this finding. Second, surgeons were aware of the study and its purpose, which might have resulted in subconsciously altering the behavior to improve satisfaction. The effect of people acting differently as a result of being observed is called the Hawthorne effect.¹⁹ Third, we only used 1 simple ordinal measure to assess patient satisfaction with the surgeon. There is a wide variety of satisfaction measures,²⁰ though the focus of this study was not to test the best possible satisfaction measure but to assess changes in satisfaction over time and its predictors. The simple 11-point ordinal satisfaction measure has proved reliable.⁶ Fourth, 35% of patients did not make a second rating. This is not unusual for phone or email studies. Our response rate was relatively high compared to other studies in our field,¹⁸ perhaps because the time to the second assessment was only 2 weeks and all people were available for follow-up by phone. Fifth, we analyzed 4 surgeons as 1 group and 3 offices as 1 group since we did not enroll enough patients per surgeon and office for individual analysis. However, multilevel linear analysis takes surgeon specific factors into account within that group.

The finding that satisfaction with the surgeon after 2 weeks was significantly lower on bivariate analysis compared to immediately after the initial visit is different from a study that found small increases in satisfaction after 2 weeks and 3 months,¹ but comparable to another study in our field.²¹ Although significant, we believe the decrease in satisfaction is probably not clinically relevant. It might also be that satisfaction at follow-up is lower than measured, but that the least satisfied people did not respond on the follow-up survey.

We found no significant change in perceived empathy after 2 weeks. Since empathy is a strong driver of satisfaction,^2,4-7 we did not expect to find differing results for empathy and for satisfaction over time. Both satisfaction and empathy seem to be relatively durable measures with current measurement tools.

The finding that change in disability was neither independently associated with change in satisfaction nor change in empathy is consistent with prior research.^2,3,21 We cannot adequately study the impact of changes since we did not find an important change in either satisfaction or empathy over time. Jackson et al found higher satisfaction ratings over time in patients who had an increase in physical function and a decrease in symptoms.¹ They also found that met expectations was associated with higher satisfaction immediately after the visit, after 2 weeks, and after 3 months.¹ We feel that met expectations and fewer symptoms and limitations are likely highly co-linear with satisfaction. We therefore may not be able to learn much about one from the others.

The slight change we found in satisfaction with the surgeon among Latino patients was significantly less than the change among white patients. This suggests Latino patients might have a more stable opinion over time (a cultural phenomenon), or it might be spurious given the small number of Latino patients included in the study. The same can be said for the finding that race/ethnicity other than white or Latino was independently associated with greater change in empathy. Providing email as the preferred mode of contact was found to be independently associated with less change in perceived empathy compared to follow-up by phone. We had a 100% success rate for our follow-ups by phone. Our findings suggest that patients might more easily switch ratings on an 11-point ordinal scale than on a 5-item Likert scale. However, both measures are often rated at the ceiling of the scale.^2,21

 

Conclusion

Satisfaction and perceived empathy are relatively stable constructs, are not clearly associated with other factors, and are strongly correlated with one another. This study supports the research practice of measuring satisfaction immediately after the visit, which is more convenient for both participant and researcher and avoids the loss of more than one third of the patients, and those with a worse experience in particular. To improve the utility and interpretation of patient-reported experience measures such as these, we might direct our efforts to developing scales with less ceiling effect.

Corresponding author: David Ring, MD, PhD, Dell Medical School, The University of Texas at Austin, Health Discovery Building HDB 6.706, 1701 Trinity St., Austin, TX 78705; [email protected].

Financial disclosures: Dr. Ring has or may receive payment or benefits from Skeletal Dynamics, Wright Medical for elbow implants, Deputy Editor for Clinical Orthopaedics and Related Research, Universities and Hospitals, Lawyers outside the submitted work.

Dr. Teunis has or may receive payment or benefits from VCC, PATIENT+, and AO Trauma TK network unrelated to this work and consultant fees from Synthes.