Original Research

From the Maine Medical Center, Portland, ME (Dr. Kramer, Ms. Palmeri, Dr. Robich, Mr. Groom, Dr. Hayes, Ms. Janoushek, Dr. Rappold, Dr. Swarz, and Dr. Quinn), and the Center for Outcomes Research and Evaluation, Maine Medical Center Research Institute, Portland, ME (Dr. Lucas).

Abstract

• Objective. To determine the accuracy of the glucometer currently used for point-of-care testing (POCT) of blood glucose in our cardiothoracic surgery intensive care unit (CTICU).
• Design. Prospective cohort study.
• Setting. Tertiary care community hospital affiliated with a school of medicine.
• Participants. Coronary artery bypass graft (CABG) surgery patients.
• Measurements. Blood glucose levels obtained via POCT with a glucometer using fingerstick and radial artery blood samples were compared with values obtained via central laboratory testing of radial artery blood samples (gold standard) in 106 CABG patients on continuous insulin infusions (CII) upon arrival to the CTICU from the operating room and 102 CABG patients on CII in the CTICU 6 hours later.
• Results. Fingerstick POCT and central lab blood glucose values correlated well (r = 0.83 for admission and 0.86 for 6-hour values), but the mean values were significantly different as determined by paired t-tests. Upon arrival, the fingerstick POCT mean value was 120.9 mg/dL, while the central laboratory value was 127.9 mg/dL (P value = 0.03). At the 6-hour time point, the mean value for fingerstick POCT was 129.7 mg/dL compared to a central laboratory value of 137.3 (P value = 0.02).
• Conclusion. The blood glucose POCT values correlated well with central laboratory values, but the values were statistically significantly different. Nevertheless, accurate clinical decisions were made despite the inaccuracies of POCT glucose testing, as experienced bedside nurses were able to use the glucometer successfully and safely. The device’s results informed them when the blood glucose was out of a prescibed range and the direction of the change, and they were able to adjust the CII accordingly.

Keywords: quality improvement; glucose management; point-of-care testing; critical care.

Achieving glycemic control in patients with and without diabetes during coronary artery bypass graft (CABG) surgery is associated with reduced perioperative morbidity and mortality and improved long-term survival.¹ Hyperglycemia has detrimental effects on the cardiovascular system and insulin has beneficial effects on the ischemic myocardium.² The current recommendations of the Society of Thoracic Surgery regarding blood glucose management include the use of continuous insulin infusions (CII) during and after surgery in the critical care unit,³ keeping blood glucose in a moderate range. Glucometers are commonly used in the critical care perioperative setting for point-of-care testing (POCT) for timely determinations of blood glucose levels for patients on CII.

POCT for glucose monitoring is a valuable tool for managing patients with diabetes in the outpatient setting. Evolving from urinary test strips that depended on a colorimetric model, glucometers now incoroporate digital technology that allows patients to determine their blood glucose using a drop of blood from a fingerstick. The US Food and Drug Administration’s approval for most glucose POCT technology includes home use by diabetic patients and use in the hospital setting, with the exception of critically ill patients, who may be affected by hypoxemia, poor capillary perfusion, tissue edema, severe anemia⁴ or other pathophysiologic states that could impact the accuracy of the devices. For example, poor peripheral perfusion related to shock or vasoconstrictors and interstitial edema are variables that could contribute to an erroneous reading. Therefore, many glucometers used in the critical care setting are being used off-label. Because much of the current POCT technology for glucose monitoring may provide erroneous results in certain ranges and in some clinical settings, the safety of most glucometers has been called into question.^5,6

Given the concern regarding the potential inaccuracies of commonly used glucometers in the critical care setting, we undertook a quality improvement project to analyze the clinical performance of the glucometer currently used in our critically ill postoperative cardiac surgery population. The cardiac surgery division policy at our institution is to place all patients, both diabetic and nondiabetic, on a CII intraoperatively and to continue the infusion for at least 24 to 48 hours postoperatively. The CII start rate is determined utilizing the division’s Insulin Start Chart, and then the CII is adjusted according to the nomogram through the postoperative course. Both the Insulin Start Chart and nomogram have been previously described by Kramer et al.⁷

Currently, POCT of glucose in all post cardiac surgery patients is done hourly or more frequently in the first 24 to 48 hours after surgery in order to adjust the CII. In patients undergoing the stress of cardiac surgery, the action of insulin is counter-regulated by glucagon, epinephrine, norepinephrine, cortisol, and growth hormone. The resulting varying degrees of insulin resistance in this population of patients requires close monitoring of blood glucose, keeping it in a prescribed range, which in our center is 110 to 150 mg/dL, both in diabetic and nondiabetic patients. Frequent laboratory and POCT determinations of glucose are made. Providers and bedside nurses adjust the CII according to central laboratory values, POCT values, and trends, as previously described.⁷

Methods

Setting

Maine Medical Center is a 600-bed tertiary care teaching hospital. It is a level 1 trauma center where 1000 cardiac surgical operations are performed annually. POCT glucose monitoring is relied upon to monitor blood glucose and adjust the CII accordingly. This project, which did not require any additional procedures outside of the standard of care for this population of patients, was reviewed by the Institutional Review Board, who determined that this activity does not meet either the definition of research as specified under 45 CFR 46.102 (d) or the definition of clinical investigation as specified in 21 CFR 56.102 (c).

Patients

Using central laboratory glucose values drawn from the radial artery as the gold standard, we created a registry of consecutive postoperative cardiac surgery patients who had undergone CABG surgery and had blood glucose determinations from both POCT (fingerstick and radial artery samples) and central laboratory testing (radial artery sample) during a 7-month period (May 2016 through February 2017). To be included in the registry, patients had to (1) be postoperative following isolated CABG or CABG plus Maze procedure; (2) have been on cardiopulmonary bypass (CPB); (3) have radial arterial lines; and (4) be on a CII. A total of 116 patients qualified according to the inclusion criteria. Patients missing glucose results in 1 or more of the variables were excluded from data analysis.

Measurements and Variables

Using a POCT glucometer (FreeStyle Precision Pro, Abbott Laboratories, Abbott Park, IL), blood glucose conentrations were measured on samples obtained from both fingerstick and radial artery. Concurrently, radial arterial blood was sent to the central laboratory for glucose measurement. Blood glucose values were compared in CABG patients on CII upon arrival to the cardiothoracic surgery intensive care unit (CTICU) from the operating room and CABG patients on CII 6 hours after arrival in the CTICU. During the 6-hour interval, blood glucose levels were tested hourly or more frequently, allowing nurses to identify trends in blood glucose changes in order to keep blood glucose in the prescribed goal range of 110 to 150 mg/dL. At each of these 2 time points, on arrival to CTICU and 6 hours later, blood glucose values obtained with radial artery POCT and fingerstick POCT were compared with values obtained with central laboratory testing of radial artery samples. The amount of blood required was 1 drop each for POCT fingerstick and POCT radial artery and 2 mL for central lab testing.

Patient characteristics were identified from the electronic medical record. The variables recorded were type of operation, time on CPB, time of CTICU arrival, temperature, vasoconstrictor infusions (norepinephrine, vasopressin, phenylephrine), preoperative diagnosis of diabetes mellitus, preoperative HbA1c, and hemoglobin/hematocrit. Hemoglobin/hematocrit was only available at the time of the patient’s arrival to CTICU. The study was completed within the confines of our center’s standard of care protocol for postoperative cardiac surgical patients.

Analysis

We used standard statistical techniques to describe the study population, including proportions for categorical variables and means (standard deviations) for continuous variables. Correlation and regression techniques were used to describe the relationship between POCT and laboratory (gold standard) tests, both measured as continuous variables, and paired t-tests with Bonferroni correction were used to compare the central tendency and range of these comparisons. We calculated the differences between the gold standard measure and the POCT measure as an indication of outliers (ie, cases in which the 2 tests gave markedly different results). We examined plots to ascertain at which levels of the gold standard test these outliers occurred. An interim analysis was done at the halfway point and submitted to the Institutional Review Board, but no correction to the P value was done based on this analysis, which was largely qualitative. We used Bonferroni correction to declare a P value of 0.025 statistically significant with the 2-way comparisons of both fingerstick and radial artery values to central laboratory values. When the data was stratified by a clinical characteristic creating a 4-way comparison, we used Bonferroni correction to declare a P value of 0.0125 to be statistically significant when comparing both fingerstick and radial artery values to central laboratory values.

Results

Glucose POCT evaluations were carried out on 116 consecutive patients who underwent CABG surgery with or without a Maze procedure on CPB with a CII and an arterial line. Due to missing glucose results in 1 or more of the variables, 10 patients were excluded from data analysis for the time point of arrival in the CTICU and 14 patients were excluded from data analysis for the time point of 6 hours post CTICU arrival. This gave a final count of 106 CABG patients for CTICU arrival data analysis and 102 CABG patients for the 6 hours after CTICU arrival data analysis.

Patients ranged in age from 43 to 85 years, with a mean of age of 66 years, 22% were were women, 41% were diabetic, and 18% had peripheral vascular disease (Table 1). The average preoperative HbA1c was 6.4% ± 1.3% (range, 4.6% to 11.1%). Mean time on CBP for the group was 101 ± 31 minutes (range, 43 to 233 minutes). Postoperative mean hematocrit and hemoglobin were 32.5% and 11.4 g/dL, respectively. The average core temperature of patients on arrival was 36.0°C, which rose to an average of 36.6°C 6 hours later. A vasoconstrictor drip was infusing on 52% of patients upon CTICU arrival; 65% had a vasoconstrictor drip infusing 6 hours after arrival to the CTICU. Hemoglobin results were available only upon CTICU arrival as they are not routinely checked at 6 hours; 74 (64%) patients had a hemoglobin < 12 g/dL.

Compared to central laboratory testing, which we are defining as the gold standard, fingerstick POCT performed better on arrival, while radial artery POCT performed better at 6 hours (Table 2). At CTICU arrival, the mean blood glucose value for fingerstick POCT was 121 ± 24.1 mg/dL, 116 ± 27.2 mg/dL for radial artery POCT, and 128 ± 23.5 mg/dL for central lab testing. The difference in mean blood glucose between the fingerstick POCT and central lab testing was not statistically significant (P = 0.032), while the difference in mean blood glucose between radial artery POCT and central lab testing was statistically significant (P = 0.001). At 6 hours post arrival to the CTICU, the mean fingerstick POCT blood glucose value was 130 ± 23.9 mg/dL, compared to the mean central lab testing value of 137 ± 22.4 mg/dL; this difference was statistically significant (P = 0.019), while the radial artery POCT blood glucose value (133 ± 24.6 mg/dL) was not significantly different from the central lab testing value.

Blood glucose values from fingerstick POCT and central laboratory testing correlated well (r = 0.83 for admission and 0.86 for 6-hour values), as did radial artery POCT and central lab values (r = 0.87 for admission and 0.90 for 6-hour values) (Figures 1, 2, 3, and 4). Comparing individual values for fingerstick POCT and central lab testing, within-person differences between the 2 values ranged from –45 to 25 mg/dL, with 21% of pairs discrepant by 20 mg/dL or more (Figure 1); results were similar at 6 hours (Figure 2), with slightly less discrepancy.

The differences between radial artery POCT and central lab testing values at CTICU arrival ranged from –43 to 80 mg/dL, with 24% of pairs discrepant by 20 mg/dL or more (Figure 3). At 6 hours post CTICU arrival, the difference between radial artery POCT and central lab testing values ranged from –130 to 27 mg/dL, with 11% of pairs discrepant by 20 mg/dL or more (Figure 4). Ninety-two percent of central laboratory values were either close to (± 20) or within the moderate glycemic control target range (110–150 mg/dL).

When the patient cohort was stratified by anemia, diabetes, body temperature, and receipt of vasoconstrictor, there were no significant differences between mean fingerstick POCT and central lab testing values for any strata on CTICU arrival, while there were significant differences between radial artery POCT and central lab testing means for both vasoconstrictor strata as well as for patients with core temperature > 36.1°C (Table 2). At 6 hours, there were no statistically significant differences when stratified for receipt of vasoconstrictor or presence of diabetes. Stratification for anemia or core body temperature was not done for patients at the 6-hour post CTICU arrival time because no hemoglobin value was available and all patients except 1 reached a core temperature of 36.1°C.

Although we measured POCT values obtained using 2 different blood sample sources, fingerstick POCT performed better than radial artery POCT testing with regard to the mean values when compared with the central lab. However, radial artery POCT performed better with regard to correlation with the central lab value. In other words, fingerstick POCT values were less significantly different than radial artery POCT values when compared with the central lab, while radial artery POCT values correlated better with values from the central lab. In spite of this unexplained variability in differences and correlation, the blood glucose values stayed in the target goal range (Figures 1-4).

Discussion

The accuracy of glucose POCT in the critical care setting has been called into question.^4,5 The clinical demands of glucose management using CII include timely and accurate guidance in postoperaptive cardiac surgery, in this case, CABG. A previous study compared POCT and central laboratory blood glucose values in medical intensive care unit patients,⁸ but not in patients who have had CABG surgery. Another study has reviewed the difference in glucose values from POCT and central lab analysis in the critically ill population, but not in the post cardiac surgical population.⁹ We have shown that the POCT blood glucose values correlate well with the clinical lab values, but the values are statistically different. Our study adds an additional observation in that, although the POCT inconsistencies were statistically significant, they were not clinically significant. That is, POCT of blood glucose was inaccurate, but it still helped guide care by providing enough information to keep the blood glucose in range (most of the time) and allowing the bedside nurse to detect trends and make appropriate adjustments to the infusion. However, given these inconsistencies, we recommend a low threshold for sending additional samples to the central lab to double-check the glucose values, especially when they are outside the prescribed range. Our analysis provides some measure of reassurance with regard to current postoperative CABG glucose management by showing that the limitations of the blood glucose meter do not jeopardize the safety of patients. Nonetheless, we look forward to advances in the accuracy of POCT blood glucose technology so that critical care patients can be better managed when blood glucose is outside the prescribed range.

This analysis of 116 CABG patients points out both the inaccuracy and the utility of a representative POCT glucometer (in this case, the FreeStyle Precision Pro) used at the bedside to manage CIIs in postoperative CABG patients, keeping the blood glucose level in the moderate control range (110-150 mg/dL). The correlation plot shows that in this population the bedside nurses were able to keep blood glucose in range most of the time, in spite of the inaccuracy of POCT of blood glucose, given that the error of the test fits in the wide margin of 40 mg/dL. The fact that the 6-hour values were slightly less variable than the admission values indicates that sequential determinations of blood glucose over the 6-hour period to detect trends allowed good clinical management even in the face of such inaccuracy. The correlation allows the inaccurate number (blood glucose value) to indicate direction, and frequent determinations allow the bedside nurse to keep that number in the prescribed range most of the time in this population of patients.

Conclusion

We have found that glucometer blood glucose determinations in our center used on a homogenous population (CABG surgery) utilizing a single type of glucometer correlated well with those of the central lab, but were not always accurate. In spite of the inaccuracies, experienced bedside nurses were able to use the instrument successfully and safely, as it informed them if the blood glucose was in or out of a predetermined range and in which direction it was going.

Acknowledgment: The authors are indebted to the nurses of the Cardiothoracic Surgery Intensive Care Unit at Maine Medical Center for their support and assistance, without which this analysis would not have been possible.

Corresponding author: Robert S. Kramer, MD, Division of Cardiothoracic Surgery, Maine Medical Center Cardiovascular Institute, 22 Bramhall St., Portland ME 04102; [email protected].

Financial disclosures: None.

From the Maine Medical Center, Portland, ME (Dr. Kramer, Ms. Palmeri, Dr. Robich, Mr. Groom, Dr. Hayes, Ms. Janoushek, Dr. Rappold, Dr. Swarz, and Dr. Quinn), and the Center for Outcomes Research and Evaluation, Maine Medical Center Research Institute, Portland, ME (Dr. Lucas).

Abstract

• Objective. To determine the accuracy of the glucometer currently used for point-of-care testing (POCT) of blood glucose in our cardiothoracic surgery intensive care unit (CTICU).
• Design. Prospective cohort study.
• Setting. Tertiary care community hospital affiliated with a school of medicine.
• Participants. Coronary artery bypass graft (CABG) surgery patients.
• Measurements. Blood glucose levels obtained via POCT with a glucometer using fingerstick and radial artery blood samples were compared with values obtained via central laboratory testing of radial artery blood samples (gold standard) in 106 CABG patients on continuous insulin infusions (CII) upon arrival to the CTICU from the operating room and 102 CABG patients on CII in the CTICU 6 hours later.
• Results. Fingerstick POCT and central lab blood glucose values correlated well (r = 0.83 for admission and 0.86 for 6-hour values), but the mean values were significantly different as determined by paired t-tests. Upon arrival, the fingerstick POCT mean value was 120.9 mg/dL, while the central laboratory value was 127.9 mg/dL (P value = 0.03). At the 6-hour time point, the mean value for fingerstick POCT was 129.7 mg/dL compared to a central laboratory value of 137.3 (P value = 0.02).
• Conclusion. The blood glucose POCT values correlated well with central laboratory values, but the values were statistically significantly different. Nevertheless, accurate clinical decisions were made despite the inaccuracies of POCT glucose testing, as experienced bedside nurses were able to use the glucometer successfully and safely. The device’s results informed them when the blood glucose was out of a prescibed range and the direction of the change, and they were able to adjust the CII accordingly.

Keywords: quality improvement; glucose management; point-of-care testing; critical care.

Achieving glycemic control in patients with and without diabetes during coronary artery bypass graft (CABG) surgery is associated with reduced perioperative morbidity and mortality and improved long-term survival.¹ Hyperglycemia has detrimental effects on the cardiovascular system and insulin has beneficial effects on the ischemic myocardium.² The current recommendations of the Society of Thoracic Surgery regarding blood glucose management include the use of continuous insulin infusions (CII) during and after surgery in the critical care unit,³ keeping blood glucose in a moderate range. Glucometers are commonly used in the critical care perioperative setting for point-of-care testing (POCT) for timely determinations of blood glucose levels for patients on CII.

POCT for glucose monitoring is a valuable tool for managing patients with diabetes in the outpatient setting. Evolving from urinary test strips that depended on a colorimetric model, glucometers now incoroporate digital technology that allows patients to determine their blood glucose using a drop of blood from a fingerstick. The US Food and Drug Administration’s approval for most glucose POCT technology includes home use by diabetic patients and use in the hospital setting, with the exception of critically ill patients, who may be affected by hypoxemia, poor capillary perfusion, tissue edema, severe anemia⁴ or other pathophysiologic states that could impact the accuracy of the devices. For example, poor peripheral perfusion related to shock or vasoconstrictors and interstitial edema are variables that could contribute to an erroneous reading. Therefore, many glucometers used in the critical care setting are being used off-label. Because much of the current POCT technology for glucose monitoring may provide erroneous results in certain ranges and in some clinical settings, the safety of most glucometers has been called into question.^5,6

Given the concern regarding the potential inaccuracies of commonly used glucometers in the critical care setting, we undertook a quality improvement project to analyze the clinical performance of the glucometer currently used in our critically ill postoperative cardiac surgery population. The cardiac surgery division policy at our institution is to place all patients, both diabetic and nondiabetic, on a CII intraoperatively and to continue the infusion for at least 24 to 48 hours postoperatively. The CII start rate is determined utilizing the division’s Insulin Start Chart, and then the CII is adjusted according to the nomogram through the postoperative course. Both the Insulin Start Chart and nomogram have been previously described by Kramer et al.⁷

Currently, POCT of glucose in all post cardiac surgery patients is done hourly or more frequently in the first 24 to 48 hours after surgery in order to adjust the CII. In patients undergoing the stress of cardiac surgery, the action of insulin is counter-regulated by glucagon, epinephrine, norepinephrine, cortisol, and growth hormone. The resulting varying degrees of insulin resistance in this population of patients requires close monitoring of blood glucose, keeping it in a prescribed range, which in our center is 110 to 150 mg/dL, both in diabetic and nondiabetic patients. Frequent laboratory and POCT determinations of glucose are made. Providers and bedside nurses adjust the CII according to central laboratory values, POCT values, and trends, as previously described.⁷

Methods

Setting

Maine Medical Center is a 600-bed tertiary care teaching hospital. It is a level 1 trauma center where 1000 cardiac surgical operations are performed annually. POCT glucose monitoring is relied upon to monitor blood glucose and adjust the CII accordingly. This project, which did not require any additional procedures outside of the standard of care for this population of patients, was reviewed by the Institutional Review Board, who determined that this activity does not meet either the definition of research as specified under 45 CFR 46.102 (d) or the definition of clinical investigation as specified in 21 CFR 56.102 (c).

Patients

Using central laboratory glucose values drawn from the radial artery as the gold standard, we created a registry of consecutive postoperative cardiac surgery patients who had undergone CABG surgery and had blood glucose determinations from both POCT (fingerstick and radial artery samples) and central laboratory testing (radial artery sample) during a 7-month period (May 2016 through February 2017). To be included in the registry, patients had to (1) be postoperative following isolated CABG or CABG plus Maze procedure; (2) have been on cardiopulmonary bypass (CPB); (3) have radial arterial lines; and (4) be on a CII. A total of 116 patients qualified according to the inclusion criteria. Patients missing glucose results in 1 or more of the variables were excluded from data analysis.

Measurements and Variables

Using a POCT glucometer (FreeStyle Precision Pro, Abbott Laboratories, Abbott Park, IL), blood glucose conentrations were measured on samples obtained from both fingerstick and radial artery. Concurrently, radial arterial blood was sent to the central laboratory for glucose measurement. Blood glucose values were compared in CABG patients on CII upon arrival to the cardiothoracic surgery intensive care unit (CTICU) from the operating room and CABG patients on CII 6 hours after arrival in the CTICU. During the 6-hour interval, blood glucose levels were tested hourly or more frequently, allowing nurses to identify trends in blood glucose changes in order to keep blood glucose in the prescribed goal range of 110 to 150 mg/dL. At each of these 2 time points, on arrival to CTICU and 6 hours later, blood glucose values obtained with radial artery POCT and fingerstick POCT were compared with values obtained with central laboratory testing of radial artery samples. The amount of blood required was 1 drop each for POCT fingerstick and POCT radial artery and 2 mL for central lab testing.

Patient characteristics were identified from the electronic medical record. The variables recorded were type of operation, time on CPB, time of CTICU arrival, temperature, vasoconstrictor infusions (norepinephrine, vasopressin, phenylephrine), preoperative diagnosis of diabetes mellitus, preoperative HbA1c, and hemoglobin/hematocrit. Hemoglobin/hematocrit was only available at the time of the patient’s arrival to CTICU. The study was completed within the confines of our center’s standard of care protocol for postoperative cardiac surgical patients.

Analysis

We used standard statistical techniques to describe the study population, including proportions for categorical variables and means (standard deviations) for continuous variables. Correlation and regression techniques were used to describe the relationship between POCT and laboratory (gold standard) tests, both measured as continuous variables, and paired t-tests with Bonferroni correction were used to compare the central tendency and range of these comparisons. We calculated the differences between the gold standard measure and the POCT measure as an indication of outliers (ie, cases in which the 2 tests gave markedly different results). We examined plots to ascertain at which levels of the gold standard test these outliers occurred. An interim analysis was done at the halfway point and submitted to the Institutional Review Board, but no correction to the P value was done based on this analysis, which was largely qualitative. We used Bonferroni correction to declare a P value of 0.025 statistically significant with the 2-way comparisons of both fingerstick and radial artery values to central laboratory values. When the data was stratified by a clinical characteristic creating a 4-way comparison, we used Bonferroni correction to declare a P value of 0.0125 to be statistically significant when comparing both fingerstick and radial artery values to central laboratory values.

Results

Glucose POCT evaluations were carried out on 116 consecutive patients who underwent CABG surgery with or without a Maze procedure on CPB with a CII and an arterial line. Due to missing glucose results in 1 or more of the variables, 10 patients were excluded from data analysis for the time point of arrival in the CTICU and 14 patients were excluded from data analysis for the time point of 6 hours post CTICU arrival. This gave a final count of 106 CABG patients for CTICU arrival data analysis and 102 CABG patients for the 6 hours after CTICU arrival data analysis.

Patients ranged in age from 43 to 85 years, with a mean of age of 66 years, 22% were were women, 41% were diabetic, and 18% had peripheral vascular disease (Table 1). The average preoperative HbA1c was 6.4% ± 1.3% (range, 4.6% to 11.1%). Mean time on CBP for the group was 101 ± 31 minutes (range, 43 to 233 minutes). Postoperative mean hematocrit and hemoglobin were 32.5% and 11.4 g/dL, respectively. The average core temperature of patients on arrival was 36.0°C, which rose to an average of 36.6°C 6 hours later. A vasoconstrictor drip was infusing on 52% of patients upon CTICU arrival; 65% had a vasoconstrictor drip infusing 6 hours after arrival to the CTICU. Hemoglobin results were available only upon CTICU arrival as they are not routinely checked at 6 hours; 74 (64%) patients had a hemoglobin < 12 g/dL.

Compared to central laboratory testing, which we are defining as the gold standard, fingerstick POCT performed better on arrival, while radial artery POCT performed better at 6 hours (Table 2). At CTICU arrival, the mean blood glucose value for fingerstick POCT was 121 ± 24.1 mg/dL, 116 ± 27.2 mg/dL for radial artery POCT, and 128 ± 23.5 mg/dL for central lab testing. The difference in mean blood glucose between the fingerstick POCT and central lab testing was not statistically significant (P = 0.032), while the difference in mean blood glucose between radial artery POCT and central lab testing was statistically significant (P = 0.001). At 6 hours post arrival to the CTICU, the mean fingerstick POCT blood glucose value was 130 ± 23.9 mg/dL, compared to the mean central lab testing value of 137 ± 22.4 mg/dL; this difference was statistically significant (P = 0.019), while the radial artery POCT blood glucose value (133 ± 24.6 mg/dL) was not significantly different from the central lab testing value.

Blood glucose values from fingerstick POCT and central laboratory testing correlated well (r = 0.83 for admission and 0.86 for 6-hour values), as did radial artery POCT and central lab values (r = 0.87 for admission and 0.90 for 6-hour values) (Figures 1, 2, 3, and 4). Comparing individual values for fingerstick POCT and central lab testing, within-person differences between the 2 values ranged from –45 to 25 mg/dL, with 21% of pairs discrepant by 20 mg/dL or more (Figure 1); results were similar at 6 hours (Figure 2), with slightly less discrepancy.

The differences between radial artery POCT and central lab testing values at CTICU arrival ranged from –43 to 80 mg/dL, with 24% of pairs discrepant by 20 mg/dL or more (Figure 3). At 6 hours post CTICU arrival, the difference between radial artery POCT and central lab testing values ranged from –130 to 27 mg/dL, with 11% of pairs discrepant by 20 mg/dL or more (Figure 4). Ninety-two percent of central laboratory values were either close to (± 20) or within the moderate glycemic control target range (110–150 mg/dL).

When the patient cohort was stratified by anemia, diabetes, body temperature, and receipt of vasoconstrictor, there were no significant differences between mean fingerstick POCT and central lab testing values for any strata on CTICU arrival, while there were significant differences between radial artery POCT and central lab testing means for both vasoconstrictor strata as well as for patients with core temperature > 36.1°C (Table 2). At 6 hours, there were no statistically significant differences when stratified for receipt of vasoconstrictor or presence of diabetes. Stratification for anemia or core body temperature was not done for patients at the 6-hour post CTICU arrival time because no hemoglobin value was available and all patients except 1 reached a core temperature of 36.1°C.

Although we measured POCT values obtained using 2 different blood sample sources, fingerstick POCT performed better than radial artery POCT testing with regard to the mean values when compared with the central lab. However, radial artery POCT performed better with regard to correlation with the central lab value. In other words, fingerstick POCT values were less significantly different than radial artery POCT values when compared with the central lab, while radial artery POCT values correlated better with values from the central lab. In spite of this unexplained variability in differences and correlation, the blood glucose values stayed in the target goal range (Figures 1-4).

Discussion

The accuracy of glucose POCT in the critical care setting has been called into question.^4,5 The clinical demands of glucose management using CII include timely and accurate guidance in postoperaptive cardiac surgery, in this case, CABG. A previous study compared POCT and central laboratory blood glucose values in medical intensive care unit patients,⁸ but not in patients who have had CABG surgery. Another study has reviewed the difference in glucose values from POCT and central lab analysis in the critically ill population, but not in the post cardiac surgical population.⁹ We have shown that the POCT blood glucose values correlate well with the clinical lab values, but the values are statistically different. Our study adds an additional observation in that, although the POCT inconsistencies were statistically significant, they were not clinically significant. That is, POCT of blood glucose was inaccurate, but it still helped guide care by providing enough information to keep the blood glucose in range (most of the time) and allowing the bedside nurse to detect trends and make appropriate adjustments to the infusion. However, given these inconsistencies, we recommend a low threshold for sending additional samples to the central lab to double-check the glucose values, especially when they are outside the prescribed range. Our analysis provides some measure of reassurance with regard to current postoperative CABG glucose management by showing that the limitations of the blood glucose meter do not jeopardize the safety of patients. Nonetheless, we look forward to advances in the accuracy of POCT blood glucose technology so that critical care patients can be better managed when blood glucose is outside the prescribed range.

This analysis of 116 CABG patients points out both the inaccuracy and the utility of a representative POCT glucometer (in this case, the FreeStyle Precision Pro) used at the bedside to manage CIIs in postoperative CABG patients, keeping the blood glucose level in the moderate control range (110-150 mg/dL). The correlation plot shows that in this population the bedside nurses were able to keep blood glucose in range most of the time, in spite of the inaccuracy of POCT of blood glucose, given that the error of the test fits in the wide margin of 40 mg/dL. The fact that the 6-hour values were slightly less variable than the admission values indicates that sequential determinations of blood glucose over the 6-hour period to detect trends allowed good clinical management even in the face of such inaccuracy. The correlation allows the inaccurate number (blood glucose value) to indicate direction, and frequent determinations allow the bedside nurse to keep that number in the prescribed range most of the time in this population of patients.

Conclusion

We have found that glucometer blood glucose determinations in our center used on a homogenous population (CABG surgery) utilizing a single type of glucometer correlated well with those of the central lab, but were not always accurate. In spite of the inaccuracies, experienced bedside nurses were able to use the instrument successfully and safely, as it informed them if the blood glucose was in or out of a predetermined range and in which direction it was going.

Acknowledgment: The authors are indebted to the nurses of the Cardiothoracic Surgery Intensive Care Unit at Maine Medical Center for their support and assistance, without which this analysis would not have been possible.

Corresponding author: Robert S. Kramer, MD, Division of Cardiothoracic Surgery, Maine Medical Center Cardiovascular Institute, 22 Bramhall St., Portland ME 04102; [email protected].

Financial disclosures: None.

From the Maine Medical Center, Portland, ME (Dr. Kramer, Ms. Palmeri, Dr. Robich, Mr. Groom, Dr. Hayes, Ms. Janoushek, Dr. Rappold, Dr. Swarz, and Dr. Quinn), and the Center for Outcomes Research and Evaluation, Maine Medical Center Research Institute, Portland, ME (Dr. Lucas).

Abstract

• Objective. To determine the accuracy of the glucometer currently used for point-of-care testing (POCT) of blood glucose in our cardiothoracic surgery intensive care unit (CTICU).
• Design. Prospective cohort study.
• Setting. Tertiary care community hospital affiliated with a school of medicine.
• Participants. Coronary artery bypass graft (CABG) surgery patients.
• Measurements. Blood glucose levels obtained via POCT with a glucometer using fingerstick and radial artery blood samples were compared with values obtained via central laboratory testing of radial artery blood samples (gold standard) in 106 CABG patients on continuous insulin infusions (CII) upon arrival to the CTICU from the operating room and 102 CABG patients on CII in the CTICU 6 hours later.
• Results. Fingerstick POCT and central lab blood glucose values correlated well (r = 0.83 for admission and 0.86 for 6-hour values), but the mean values were significantly different as determined by paired t-tests. Upon arrival, the fingerstick POCT mean value was 120.9 mg/dL, while the central laboratory value was 127.9 mg/dL (P value = 0.03). At the 6-hour time point, the mean value for fingerstick POCT was 129.7 mg/dL compared to a central laboratory value of 137.3 (P value = 0.02).
• Conclusion. The blood glucose POCT values correlated well with central laboratory values, but the values were statistically significantly different. Nevertheless, accurate clinical decisions were made despite the inaccuracies of POCT glucose testing, as experienced bedside nurses were able to use the glucometer successfully and safely. The device’s results informed them when the blood glucose was out of a prescibed range and the direction of the change, and they were able to adjust the CII accordingly.

Keywords: quality improvement; glucose management; point-of-care testing; critical care.

Achieving glycemic control in patients with and without diabetes during coronary artery bypass graft (CABG) surgery is associated with reduced perioperative morbidity and mortality and improved long-term survival.¹ Hyperglycemia has detrimental effects on the cardiovascular system and insulin has beneficial effects on the ischemic myocardium.² The current recommendations of the Society of Thoracic Surgery regarding blood glucose management include the use of continuous insulin infusions (CII) during and after surgery in the critical care unit,³ keeping blood glucose in a moderate range. Glucometers are commonly used in the critical care perioperative setting for point-of-care testing (POCT) for timely determinations of blood glucose levels for patients on CII.

POCT for glucose monitoring is a valuable tool for managing patients with diabetes in the outpatient setting. Evolving from urinary test strips that depended on a colorimetric model, glucometers now incoroporate digital technology that allows patients to determine their blood glucose using a drop of blood from a fingerstick. The US Food and Drug Administration’s approval for most glucose POCT technology includes home use by diabetic patients and use in the hospital setting, with the exception of critically ill patients, who may be affected by hypoxemia, poor capillary perfusion, tissue edema, severe anemia⁴ or other pathophysiologic states that could impact the accuracy of the devices. For example, poor peripheral perfusion related to shock or vasoconstrictors and interstitial edema are variables that could contribute to an erroneous reading. Therefore, many glucometers used in the critical care setting are being used off-label. Because much of the current POCT technology for glucose monitoring may provide erroneous results in certain ranges and in some clinical settings, the safety of most glucometers has been called into question.^5,6

Given the concern regarding the potential inaccuracies of commonly used glucometers in the critical care setting, we undertook a quality improvement project to analyze the clinical performance of the glucometer currently used in our critically ill postoperative cardiac surgery population. The cardiac surgery division policy at our institution is to place all patients, both diabetic and nondiabetic, on a CII intraoperatively and to continue the infusion for at least 24 to 48 hours postoperatively. The CII start rate is determined utilizing the division’s Insulin Start Chart, and then the CII is adjusted according to the nomogram through the postoperative course. Both the Insulin Start Chart and nomogram have been previously described by Kramer et al.⁷

Currently, POCT of glucose in all post cardiac surgery patients is done hourly or more frequently in the first 24 to 48 hours after surgery in order to adjust the CII. In patients undergoing the stress of cardiac surgery, the action of insulin is counter-regulated by glucagon, epinephrine, norepinephrine, cortisol, and growth hormone. The resulting varying degrees of insulin resistance in this population of patients requires close monitoring of blood glucose, keeping it in a prescribed range, which in our center is 110 to 150 mg/dL, both in diabetic and nondiabetic patients. Frequent laboratory and POCT determinations of glucose are made. Providers and bedside nurses adjust the CII according to central laboratory values, POCT values, and trends, as previously described.⁷

Methods

Setting

Maine Medical Center is a 600-bed tertiary care teaching hospital. It is a level 1 trauma center where 1000 cardiac surgical operations are performed annually. POCT glucose monitoring is relied upon to monitor blood glucose and adjust the CII accordingly. This project, which did not require any additional procedures outside of the standard of care for this population of patients, was reviewed by the Institutional Review Board, who determined that this activity does not meet either the definition of research as specified under 45 CFR 46.102 (d) or the definition of clinical investigation as specified in 21 CFR 56.102 (c).

Patients

Using central laboratory glucose values drawn from the radial artery as the gold standard, we created a registry of consecutive postoperative cardiac surgery patients who had undergone CABG surgery and had blood glucose determinations from both POCT (fingerstick and radial artery samples) and central laboratory testing (radial artery sample) during a 7-month period (May 2016 through February 2017). To be included in the registry, patients had to (1) be postoperative following isolated CABG or CABG plus Maze procedure; (2) have been on cardiopulmonary bypass (CPB); (3) have radial arterial lines; and (4) be on a CII. A total of 116 patients qualified according to the inclusion criteria. Patients missing glucose results in 1 or more of the variables were excluded from data analysis.

Measurements and Variables

Using a POCT glucometer (FreeStyle Precision Pro, Abbott Laboratories, Abbott Park, IL), blood glucose conentrations were measured on samples obtained from both fingerstick and radial artery. Concurrently, radial arterial blood was sent to the central laboratory for glucose measurement. Blood glucose values were compared in CABG patients on CII upon arrival to the cardiothoracic surgery intensive care unit (CTICU) from the operating room and CABG patients on CII 6 hours after arrival in the CTICU. During the 6-hour interval, blood glucose levels were tested hourly or more frequently, allowing nurses to identify trends in blood glucose changes in order to keep blood glucose in the prescribed goal range of 110 to 150 mg/dL. At each of these 2 time points, on arrival to CTICU and 6 hours later, blood glucose values obtained with radial artery POCT and fingerstick POCT were compared with values obtained with central laboratory testing of radial artery samples. The amount of blood required was 1 drop each for POCT fingerstick and POCT radial artery and 2 mL for central lab testing.

Patient characteristics were identified from the electronic medical record. The variables recorded were type of operation, time on CPB, time of CTICU arrival, temperature, vasoconstrictor infusions (norepinephrine, vasopressin, phenylephrine), preoperative diagnosis of diabetes mellitus, preoperative HbA1c, and hemoglobin/hematocrit. Hemoglobin/hematocrit was only available at the time of the patient’s arrival to CTICU. The study was completed within the confines of our center’s standard of care protocol for postoperative cardiac surgical patients.

Analysis

We used standard statistical techniques to describe the study population, including proportions for categorical variables and means (standard deviations) for continuous variables. Correlation and regression techniques were used to describe the relationship between POCT and laboratory (gold standard) tests, both measured as continuous variables, and paired t-tests with Bonferroni correction were used to compare the central tendency and range of these comparisons. We calculated the differences between the gold standard measure and the POCT measure as an indication of outliers (ie, cases in which the 2 tests gave markedly different results). We examined plots to ascertain at which levels of the gold standard test these outliers occurred. An interim analysis was done at the halfway point and submitted to the Institutional Review Board, but no correction to the P value was done based on this analysis, which was largely qualitative. We used Bonferroni correction to declare a P value of 0.025 statistically significant with the 2-way comparisons of both fingerstick and radial artery values to central laboratory values. When the data was stratified by a clinical characteristic creating a 4-way comparison, we used Bonferroni correction to declare a P value of 0.0125 to be statistically significant when comparing both fingerstick and radial artery values to central laboratory values.

Results

Glucose POCT evaluations were carried out on 116 consecutive patients who underwent CABG surgery with or without a Maze procedure on CPB with a CII and an arterial line. Due to missing glucose results in 1 or more of the variables, 10 patients were excluded from data analysis for the time point of arrival in the CTICU and 14 patients were excluded from data analysis for the time point of 6 hours post CTICU arrival. This gave a final count of 106 CABG patients for CTICU arrival data analysis and 102 CABG patients for the 6 hours after CTICU arrival data analysis.

Patients ranged in age from 43 to 85 years, with a mean of age of 66 years, 22% were were women, 41% were diabetic, and 18% had peripheral vascular disease (Table 1). The average preoperative HbA1c was 6.4% ± 1.3% (range, 4.6% to 11.1%). Mean time on CBP for the group was 101 ± 31 minutes (range, 43 to 233 minutes). Postoperative mean hematocrit and hemoglobin were 32.5% and 11.4 g/dL, respectively. The average core temperature of patients on arrival was 36.0°C, which rose to an average of 36.6°C 6 hours later. A vasoconstrictor drip was infusing on 52% of patients upon CTICU arrival; 65% had a vasoconstrictor drip infusing 6 hours after arrival to the CTICU. Hemoglobin results were available only upon CTICU arrival as they are not routinely checked at 6 hours; 74 (64%) patients had a hemoglobin < 12 g/dL.

Compared to central laboratory testing, which we are defining as the gold standard, fingerstick POCT performed better on arrival, while radial artery POCT performed better at 6 hours (Table 2). At CTICU arrival, the mean blood glucose value for fingerstick POCT was 121 ± 24.1 mg/dL, 116 ± 27.2 mg/dL for radial artery POCT, and 128 ± 23.5 mg/dL for central lab testing. The difference in mean blood glucose between the fingerstick POCT and central lab testing was not statistically significant (P = 0.032), while the difference in mean blood glucose between radial artery POCT and central lab testing was statistically significant (P = 0.001). At 6 hours post arrival to the CTICU, the mean fingerstick POCT blood glucose value was 130 ± 23.9 mg/dL, compared to the mean central lab testing value of 137 ± 22.4 mg/dL; this difference was statistically significant (P = 0.019), while the radial artery POCT blood glucose value (133 ± 24.6 mg/dL) was not significantly different from the central lab testing value.

Blood glucose values from fingerstick POCT and central laboratory testing correlated well (r = 0.83 for admission and 0.86 for 6-hour values), as did radial artery POCT and central lab values (r = 0.87 for admission and 0.90 for 6-hour values) (Figures 1, 2, 3, and 4). Comparing individual values for fingerstick POCT and central lab testing, within-person differences between the 2 values ranged from –45 to 25 mg/dL, with 21% of pairs discrepant by 20 mg/dL or more (Figure 1); results were similar at 6 hours (Figure 2), with slightly less discrepancy.

The differences between radial artery POCT and central lab testing values at CTICU arrival ranged from –43 to 80 mg/dL, with 24% of pairs discrepant by 20 mg/dL or more (Figure 3). At 6 hours post CTICU arrival, the difference between radial artery POCT and central lab testing values ranged from –130 to 27 mg/dL, with 11% of pairs discrepant by 20 mg/dL or more (Figure 4). Ninety-two percent of central laboratory values were either close to (± 20) or within the moderate glycemic control target range (110–150 mg/dL).

When the patient cohort was stratified by anemia, diabetes, body temperature, and receipt of vasoconstrictor, there were no significant differences between mean fingerstick POCT and central lab testing values for any strata on CTICU arrival, while there were significant differences between radial artery POCT and central lab testing means for both vasoconstrictor strata as well as for patients with core temperature > 36.1°C (Table 2). At 6 hours, there were no statistically significant differences when stratified for receipt of vasoconstrictor or presence of diabetes. Stratification for anemia or core body temperature was not done for patients at the 6-hour post CTICU arrival time because no hemoglobin value was available and all patients except 1 reached a core temperature of 36.1°C.

Although we measured POCT values obtained using 2 different blood sample sources, fingerstick POCT performed better than radial artery POCT testing with regard to the mean values when compared with the central lab. However, radial artery POCT performed better with regard to correlation with the central lab value. In other words, fingerstick POCT values were less significantly different than radial artery POCT values when compared with the central lab, while radial artery POCT values correlated better with values from the central lab. In spite of this unexplained variability in differences and correlation, the blood glucose values stayed in the target goal range (Figures 1-4).

Discussion

The accuracy of glucose POCT in the critical care setting has been called into question.^4,5 The clinical demands of glucose management using CII include timely and accurate guidance in postoperaptive cardiac surgery, in this case, CABG. A previous study compared POCT and central laboratory blood glucose values in medical intensive care unit patients,⁸ but not in patients who have had CABG surgery. Another study has reviewed the difference in glucose values from POCT and central lab analysis in the critically ill population, but not in the post cardiac surgical population.⁹ We have shown that the POCT blood glucose values correlate well with the clinical lab values, but the values are statistically different. Our study adds an additional observation in that, although the POCT inconsistencies were statistically significant, they were not clinically significant. That is, POCT of blood glucose was inaccurate, but it still helped guide care by providing enough information to keep the blood glucose in range (most of the time) and allowing the bedside nurse to detect trends and make appropriate adjustments to the infusion. However, given these inconsistencies, we recommend a low threshold for sending additional samples to the central lab to double-check the glucose values, especially when they are outside the prescribed range. Our analysis provides some measure of reassurance with regard to current postoperative CABG glucose management by showing that the limitations of the blood glucose meter do not jeopardize the safety of patients. Nonetheless, we look forward to advances in the accuracy of POCT blood glucose technology so that critical care patients can be better managed when blood glucose is outside the prescribed range.

This analysis of 116 CABG patients points out both the inaccuracy and the utility of a representative POCT glucometer (in this case, the FreeStyle Precision Pro) used at the bedside to manage CIIs in postoperative CABG patients, keeping the blood glucose level in the moderate control range (110-150 mg/dL). The correlation plot shows that in this population the bedside nurses were able to keep blood glucose in range most of the time, in spite of the inaccuracy of POCT of blood glucose, given that the error of the test fits in the wide margin of 40 mg/dL. The fact that the 6-hour values were slightly less variable than the admission values indicates that sequential determinations of blood glucose over the 6-hour period to detect trends allowed good clinical management even in the face of such inaccuracy. The correlation allows the inaccurate number (blood glucose value) to indicate direction, and frequent determinations allow the bedside nurse to keep that number in the prescribed range most of the time in this population of patients.

Conclusion

We have found that glucometer blood glucose determinations in our center used on a homogenous population (CABG surgery) utilizing a single type of glucometer correlated well with those of the central lab, but were not always accurate. In spite of the inaccuracies, experienced bedside nurses were able to use the instrument successfully and safely, as it informed them if the blood glucose was in or out of a predetermined range and in which direction it was going.

Acknowledgment: The authors are indebted to the nurses of the Cardiothoracic Surgery Intensive Care Unit at Maine Medical Center for their support and assistance, without which this analysis would not have been possible.

Corresponding author: Robert S. Kramer, MD, Division of Cardiothoracic Surgery, Maine Medical Center Cardiovascular Institute, 22 Bramhall St., Portland ME 04102; [email protected].

Financial disclosures: None.

From the Department of Surgery and Perioperative Care, Dell Medical School, The University of Texas at Austin, Austin, TX (Dr. Kortlever, Ms. Haidar, Dr. Reichel, Dr. Driscoll, Dr. Ring, and Dr. Vagner) and University Medical Center Utrecht, Utrecht, The Netherlands (Dr. Teunis).

Abstract

• Objective: Patient satisfaction is considered a quality measure. Satisfaction is typically measured directly after an in-person visit in research and 2 weeks later in practice surveys. We assessed if there was a difference in immediate and delayed measurement of satisfaction.
• Questions: (1) There is no difference in patient satisfaction (measured by Numerical Rating Scale [NRS]) and (2) perceived empathy (measured by the Jefferson Scale of Patient Perceptions of Physician Empathy [JSPPPE]) immediately after the initial visit compared to 2 weeks later. (3) Change in disability (measured by the Patient-Reported Outcome Measurement Information System Physical Function-Upper Extremity [PROMIS PF-UE]) is not independently associated with change in satisfaction and (4) empathy after the initial visit compared to 2 weeks later.
• Methods: 150 new patients completed a survey of demographics, satisfaction with the surgeon, rating of the surgeon’s empathy, and upper extremity specific limitations. The satisfaction, empathy, and limitation questionnaires were repeated 2 weeks later.
• Results: We found a slight but significant decrease in satisfaction 2 weeks after the in-person visit (–0.41, P = 0.001). There was no significant change in perceived empathy (–0.71, P = 0.19). Change in limitations did not account for a change in satisfaction (P = 0.79) or perceived empathy (P = 0.93).
• Conclusion: Satisfaction and perceived empathy are relatively stable constructs that can be measured immediately after the visit.

Keywords: satisfaction, empathy, change, upper extremity, disability.

Patient satisfaction is increasingly being used as a performance measure to evaluate quality of care.^1-8 Patient satisfaction correlates with adherence with recommended treatment.^1,6,8-10 Satisfaction measured on an 11-point ordinal scale immediately after the visit correlates strongly with the perception of clinician empathy.^2,3 Indeed, some satisfaction questionnaires such as the Medical Interview Satisfaction Scale (MISS)^11,12 have questions very similar to empathy questionnaires. It may be that satisfaction is a construct similar to feeling that your doctor listened and cared about you as an individual (perceived physician empathy).

Higher ratings of satisfaction also seem to be related to a physician’s communication style.^1,4,7-10 One study of 13 fertility doctors found that training in effective communication strategies led to improved patient satisfaction.⁷ A qualitative study of 36 patients, health professionals, and clinical support staff in an orthopaedic outpatient setting held interviews and focus group sessions to identify themes influencing patient satisfaction.⁴ Communication and expectation were among the 7 themes identified. We have noticed a high ceiling effect (maximum scores) with measures of patient satisfaction and perceived empathy.^2,3 Another study also noted a high ceiling effect when using an ordinal scale.⁵ It may be that people with a positive feeling shortly after a health care encounter give top ratings out of politeness or gratefulness. It is also possible they will feel differently a few weeks after they leave the office. Furthermore, ratings of satisfaction gathered by a practice or health care system for practice assessment/improvement are often obtained several days to weeks after the visit, while research often obtains satisfaction ratings immediately after the visit for practical reasons. There may be differences between immediate and delayed measurement of satisfaction beyond the mentioned social norms.

Therefore, this study tested the primary null hypothesis that there is no difference in patient satisfaction (measured by Numerical Rating Scale [NRS]) immediately after the initial visit compared to 2 weeks later. Additionally, we assessed the difference in perceived empathy immediately after the initial visit compared to 2 weeks later, and whether change in disability was independently associated with change in satisfaction and empathy after the initial visit compared to 2 weeks later.

Methods

Study Design

After Institutional Review Board approval of this prospective, longitudinal, observational cohort study, we prospectively enrolled 150 adult patients between November 29, 2017 and January 10, 2018. Patients were seen at 5 orthopaedic clinics in a large urban area. We included all new English-speaking patients aged 18 to 89 years who were visiting 1 of 6 participating orthopaedic surgeons for any upper extremity problem and who were able to provide informed consent. We excluded follow-up visits and patients who were unable to speak and understand English. Four research assistants who were not involved with patient treatment described the study to patients before or after the visit with the surgeon. We were granted a waiver of written informed consent; patients indicated their consent by completing the surveys.

Patients could choose either phone or email as their preferred mode of contact for follow-up in this study. For patients who selected email as the preferred mode of contact, the follow-up survey was sent automatically 2 weeks after completion date, and a maximum of 3 reminder emails with 2-day time intervals between them were sent to those who did not respond to the initial invitation. For patients who selected phone as the preferred mode of contact, the follow-up survey was done by an English-speaking research assistant who was not involved with patient treatment. When a response was not obtained on the initial phone call, 3 additional phone calls were made (1 later that same day and 2 the next day). One patient declined participation because he was not interested in the study and had no time after his visit.

Measurements

Patients were asked to complete a set of questionnaires at the end of their visit:

1. A demographic questionnaire consisting of preferred mode of contact for follow-up (phone or email), age, sex, race/ethnicity, marital status, education status, work status, insurance status, and type of visit (first visit or second opinion);

2. An 11-point ordinal measure of satisfaction with the surgeon, with scores ranging from 0 (Worst Surgeon Possible) to 10 (Best Surgeon Possible);

3. The patient’s rating of the surgeon’s empathy, measured by the Jefferson Scale of Patient Perceptions of Physician Empathy (JSPPPE).¹³ The JSPPPE is a 5-item questionnaire, measured on a 7-point Likert scale, with scores ranging from 1 (Strongly Disagree) to 7 (Strongly Agree), that assesses agreement with statements about the physician. The total score is the sum of all item scores (5-35), with higher scores representing a higher degree of perceived physician empathy.

4. Upper extremity disability, measured by the Patient-Reported Outcomes Measurement Information System Physical Function-Upper Extremity (PROMIS PF-UE) Computer Adaptive Test (CAT).^14-16 This is a measure of physical limitations in the upper extremity. It can be completed with as few as 4 questions while still achieving high precision in scoring and thereby decreasing survey burden. PROMIS presents a continuous T-score with a mean of 50 and standard deviation (SD) of 10, with higher scores reflecting better physical function compared to the average of the US general population.¹⁵

After completing the initial questionnaire, the research assistant filled out the office and surgeon name and asked the surgeon to complete the diagnosis. All questionnaires were administered on an encrypted tablet via the secure, HIPAA-compliant electronic platform REDCap (Research Electronic Data Capture), a web-based application for building and managing online surveys and databases.¹⁷ The follow-up survey was sent automatically or was done by phone call as previously described. The follow-up survey consisted of (1) the 11-point ordinal measure of satisfaction with the surgeon, (2) the JSPPPE for perceived empathy, and (3) the PROMIS PF-UE for physical limitations in the upper extremity.

Analysis

Continuous variables are presented as mean ± SD and discrete data as proportions. We used Student’s t-tests to assess baseline differences between continuous variables and Fisher’s exact tests for discrete variables. To assess differences in satisfaction and perceived empathy after 2 weeks, we used Student’s paired t-tests. We created 2 multilevel multivariable linear regression models to assess factors associated with (1) change in satisfaction with the surgeon and (2) change in perceived physician empathy. These models account for correlation of patients treated by the same surgeon. We selected variables to be included in the final models by running multilevel models with only 1 independent variable of interest (Appendix 1). Variables with P < 0.10 were included in our final models. We also included change in PROMIS PF-UE in both models because this was our variable of interest. We considered P < 0.05 significant.

We performed a power analysis for the difference in patient satisfaction immediately after the first visit compared to 2 weeks later. Based on our pilot data where we found an initial mean satisfaction score of 9.4 and mean satisfaction score after 2 weeks of 9.1 (SD of difference 1.0), a priori power analysis showed that we needed a minimum sample size of 90 patients to detect a difference with power set at 0.80 and alpha set at 0.05. In order to account for loss to follow-up as previously noted,¹⁸ we enrolled 67% more patients (total of 150).

Results

Respondent Characteristics

None of the 150 patients were excluded from the analysis. The study patients’ mean age was 51 ± 16 years (range, 18-87 years), and 73 (49%) were men (Table 1). Mean scores directly after the visit were 9.4 ± 1.2 (range, 2-10) for satisfaction with the surgeon, 31 ± 5.2 (range, 9-35) for perceived physician empathy, and 40 ± 10 (range 15-56) for upper extremity disability. Most patients (n = 130, 87%) were seen in 2 of 5 offices, and 106 (71%) were seen by 2 out of 6 participating surgeons.

Ninety-seven (65%) patients completed their follow-up assessment 2 weeks after their initial visit, 49 (51%) by phone and 48 (49%) by email. This is a slightly better rate than the 36% rate reported in previous research.¹⁸ After 2 weeks, the mean score for satisfaction with the surgeon was 9.1 ± 1.5 (range, 0-10), the mean perceived empathy score was 31 ± 5.1 (range, 6-35), and the mean upper extremity disability score was 40 ± 8.7 (range, 23-56). Responders did not differ from nonresponders based on demographic data (Table 2). However, nonresponders had lower perceived empathy scores directly after their visit (P = 0.03) and none had initially chosen phone as their preferred mode of contact for follow-up (P < 0.001). A list of all diagnoses with frequencies the surgeons stated is listed in Appendix 2.

Difference in Satisfaction with the Surgeon

Satisfaction with the surgeon 2 weeks after the in-person visit was slightly, but significantly, lower on bivariate analysis compared to satisfaction with the surgeon immediately after the initial visit (–0.41 ± 1.2, P = 0.001; Table 3).

Difference in Perceived Physician Empathy

Perceived physician empathy 2 weeks after the in-person visit was not significantly lower on bivariate analysis compared to perceived physician empathy immediately after the initial visit (–0.71 ± 5.3, P = 0.19; Table 3).

Factors Associated with Change in Satisfaction with the Surgeon

Accounting for potential interaction of variables using multilevel multivariable analysis, change in disability of the upper extremity was not associated with change in satisfaction with the surgeon (regression coefficient [beta], 0.00 [95% confidence interval {CI}, –0.02 to 0.03]; standard error [SE], 0.01; P = 0.79 [Table 4]). Being Latino was independently associated with less change in satisfaction with the surgeon (beta coefficient, –0.57 [95% CI, –1.1 to 0.00]; SE, 0.29; P = 0.049).

Factors Associated with Change in Perceived Physician Empathy

Accounting for potential interaction of variables using multilevel multivariable analysis, change in disability of the upper extremity was not associated with change in perceived physician empathy (beta coefficient = 0.00 [95% CI, –0.10 to 0.11]; SE, 0.06; P = 0.93 [Table 4]). Race/ethnicity other than white or Latino was independently associated with more change in perceived physician empathy (beta coefficient, 3.5 [95% CI, 0.34 to 6.6]; SE, 1.6; P = 0.030), and preferring email as mode of contact for follow-up was independently associated with less change in perceived physician empathy (beta coefficient, –3.2 [95% CI, –5.2 to –1.3]; SE, 1.0; P = 0.001).

Discussion

Patient satisfaction is considered a quality measure^1-8 and is typically measured directly after an in-person visit. This study tested differences in patient satisfaction and perceived empathy immediately after the initial visit compared to 2 weeks later. In addition, we assessed whether change in disability was independently associated with change in satisfaction and empathy after the initial visit compared to 2 weeks later.

We acknowledge some study limitations. First, we only measured satisfaction based on 1 visit rather than multiple visits over time. It might be that satisfaction ratings differ when the physician-patient relationship is more established. However, we found overall high satisfaction ratings and a well-established relationship might not add to this finding. Second, surgeons were aware of the study and its purpose, which might have resulted in subconsciously altering the behavior to improve satisfaction. The effect of people acting differently as a result of being observed is called the Hawthorne effect.¹⁹ Third, we only used 1 simple ordinal measure to assess patient satisfaction with the surgeon. There is a wide variety of satisfaction measures,²⁰ though the focus of this study was not to test the best possible satisfaction measure but to assess changes in satisfaction over time and its predictors. The simple 11-point ordinal satisfaction measure has proved reliable.⁶ Fourth, 35% of patients did not make a second rating. This is not unusual for phone or email studies. Our response rate was relatively high compared to other studies in our field,¹⁸ perhaps because the time to the second assessment was only 2 weeks and all people were available for follow-up by phone. Fifth, we analyzed 4 surgeons as 1 group and 3 offices as 1 group since we did not enroll enough patients per surgeon and office for individual analysis. However, multilevel linear analysis takes surgeon specific factors into account within that group.

The finding that satisfaction with the surgeon after 2 weeks was significantly lower on bivariate analysis compared to immediately after the initial visit is different from a study that found small increases in satisfaction after 2 weeks and 3 months,¹ but comparable to another study in our field.²¹ Although significant, we believe the decrease in satisfaction is probably not clinically relevant. It might also be that satisfaction at follow-up is lower than measured, but that the least satisfied people did not respond on the follow-up survey.

We found no significant change in perceived empathy after 2 weeks. Since empathy is a strong driver of satisfaction,^2,4-7 we did not expect to find differing results for empathy and for satisfaction over time. Both satisfaction and empathy seem to be relatively durable measures with current measurement tools.

The finding that change in disability was neither independently associated with change in satisfaction nor change in empathy is consistent with prior research.^2,3,21 We cannot adequately study the impact of changes since we did not find an important change in either satisfaction or empathy over time. Jackson et al found higher satisfaction ratings over time in patients who had an increase in physical function and a decrease in symptoms.¹ They also found that met expectations was associated with higher satisfaction immediately after the visit, after 2 weeks, and after 3 months.¹ We feel that met expectations and fewer symptoms and limitations are likely highly co-linear with satisfaction. We therefore may not be able to learn much about one from the others.

The slight change we found in satisfaction with the surgeon among Latino patients was significantly less than the change among white patients. This suggests Latino patients might have a more stable opinion over time (a cultural phenomenon), or it might be spurious given the small number of Latino patients included in the study. The same can be said for the finding that race/ethnicity other than white or Latino was independently associated with greater change in empathy. Providing email as the preferred mode of contact was found to be independently associated with less change in perceived empathy compared to follow-up by phone. We had a 100% success rate for our follow-ups by phone. Our findings suggest that patients might more easily switch ratings on an 11-point ordinal scale than on a 5-item Likert scale. However, both measures are often rated at the ceiling of the scale.^2,21

Conclusion

Satisfaction and perceived empathy are relatively stable constructs, are not clearly associated with other factors, and are strongly correlated with one another. This study supports the research practice of measuring satisfaction immediately after the visit, which is more convenient for both participant and researcher and avoids the loss of more than one third of the patients, and those with a worse experience in particular. To improve the utility and interpretation of patient-reported experience measures such as these, we might direct our efforts to developing scales with less ceiling effect.

Corresponding author: David Ring, MD, PhD, Dell Medical School, The University of Texas at Austin, Health Discovery Building HDB 6.706, 1701 Trinity St., Austin, TX 78705; [email protected].

Financial disclosures: Dr. Ring has or may receive payment or benefits from Skeletal Dynamics, Wright Medical for elbow implants, Deputy Editor for Clinical Orthopaedics and Related Research, Universities and Hospitals, Lawyers outside the submitted work.

Dr. Teunis has or may receive payment or benefits from VCC, PATIENT+, and AO Trauma TK network unrelated to this work and consultant fees from Synthes.

From the Department of Surgery and Perioperative Care, Dell Medical School, The University of Texas at Austin, Austin, TX (Dr. Kortlever, Ms. Haidar, Dr. Reichel, Dr. Driscoll, Dr. Ring, and Dr. Vagner) and University Medical Center Utrecht, Utrecht, The Netherlands (Dr. Teunis).

Abstract

• Objective: Patient satisfaction is considered a quality measure. Satisfaction is typically measured directly after an in-person visit in research and 2 weeks later in practice surveys. We assessed if there was a difference in immediate and delayed measurement of satisfaction.
• Questions: (1) There is no difference in patient satisfaction (measured by Numerical Rating Scale [NRS]) and (2) perceived empathy (measured by the Jefferson Scale of Patient Perceptions of Physician Empathy [JSPPPE]) immediately after the initial visit compared to 2 weeks later. (3) Change in disability (measured by the Patient-Reported Outcome Measurement Information System Physical Function-Upper Extremity [PROMIS PF-UE]) is not independently associated with change in satisfaction and (4) empathy after the initial visit compared to 2 weeks later.
• Methods: 150 new patients completed a survey of demographics, satisfaction with the surgeon, rating of the surgeon’s empathy, and upper extremity specific limitations. The satisfaction, empathy, and limitation questionnaires were repeated 2 weeks later.
• Results: We found a slight but significant decrease in satisfaction 2 weeks after the in-person visit (–0.41, P = 0.001). There was no significant change in perceived empathy (–0.71, P = 0.19). Change in limitations did not account for a change in satisfaction (P = 0.79) or perceived empathy (P = 0.93).
• Conclusion: Satisfaction and perceived empathy are relatively stable constructs that can be measured immediately after the visit.

Keywords: satisfaction, empathy, change, upper extremity, disability.

Patient satisfaction is increasingly being used as a performance measure to evaluate quality of care.^1-8 Patient satisfaction correlates with adherence with recommended treatment.^1,6,8-10 Satisfaction measured on an 11-point ordinal scale immediately after the visit correlates strongly with the perception of clinician empathy.^2,3 Indeed, some satisfaction questionnaires such as the Medical Interview Satisfaction Scale (MISS)^11,12 have questions very similar to empathy questionnaires. It may be that satisfaction is a construct similar to feeling that your doctor listened and cared about you as an individual (perceived physician empathy).

Higher ratings of satisfaction also seem to be related to a physician’s communication style.^1,4,7-10 One study of 13 fertility doctors found that training in effective communication strategies led to improved patient satisfaction.⁷ A qualitative study of 36 patients, health professionals, and clinical support staff in an orthopaedic outpatient setting held interviews and focus group sessions to identify themes influencing patient satisfaction.⁴ Communication and expectation were among the 7 themes identified. We have noticed a high ceiling effect (maximum scores) with measures of patient satisfaction and perceived empathy.^2,3 Another study also noted a high ceiling effect when using an ordinal scale.⁵ It may be that people with a positive feeling shortly after a health care encounter give top ratings out of politeness or gratefulness. It is also possible they will feel differently a few weeks after they leave the office. Furthermore, ratings of satisfaction gathered by a practice or health care system for practice assessment/improvement are often obtained several days to weeks after the visit, while research often obtains satisfaction ratings immediately after the visit for practical reasons. There may be differences between immediate and delayed measurement of satisfaction beyond the mentioned social norms.

Therefore, this study tested the primary null hypothesis that there is no difference in patient satisfaction (measured by Numerical Rating Scale [NRS]) immediately after the initial visit compared to 2 weeks later. Additionally, we assessed the difference in perceived empathy immediately after the initial visit compared to 2 weeks later, and whether change in disability was independently associated with change in satisfaction and empathy after the initial visit compared to 2 weeks later.

Methods

Study Design

After Institutional Review Board approval of this prospective, longitudinal, observational cohort study, we prospectively enrolled 150 adult patients between November 29, 2017 and January 10, 2018. Patients were seen at 5 orthopaedic clinics in a large urban area. We included all new English-speaking patients aged 18 to 89 years who were visiting 1 of 6 participating orthopaedic surgeons for any upper extremity problem and who were able to provide informed consent. We excluded follow-up visits and patients who were unable to speak and understand English. Four research assistants who were not involved with patient treatment described the study to patients before or after the visit with the surgeon. We were granted a waiver of written informed consent; patients indicated their consent by completing the surveys.

Patients could choose either phone or email as their preferred mode of contact for follow-up in this study. For patients who selected email as the preferred mode of contact, the follow-up survey was sent automatically 2 weeks after completion date, and a maximum of 3 reminder emails with 2-day time intervals between them were sent to those who did not respond to the initial invitation. For patients who selected phone as the preferred mode of contact, the follow-up survey was done by an English-speaking research assistant who was not involved with patient treatment. When a response was not obtained on the initial phone call, 3 additional phone calls were made (1 later that same day and 2 the next day). One patient declined participation because he was not interested in the study and had no time after his visit.

Measurements

Patients were asked to complete a set of questionnaires at the end of their visit:

1. A demographic questionnaire consisting of preferred mode of contact for follow-up (phone or email), age, sex, race/ethnicity, marital status, education status, work status, insurance status, and type of visit (first visit or second opinion);

2. An 11-point ordinal measure of satisfaction with the surgeon, with scores ranging from 0 (Worst Surgeon Possible) to 10 (Best Surgeon Possible);

3. The patient’s rating of the surgeon’s empathy, measured by the Jefferson Scale of Patient Perceptions of Physician Empathy (JSPPPE).¹³ The JSPPPE is a 5-item questionnaire, measured on a 7-point Likert scale, with scores ranging from 1 (Strongly Disagree) to 7 (Strongly Agree), that assesses agreement with statements about the physician. The total score is the sum of all item scores (5-35), with higher scores representing a higher degree of perceived physician empathy.

4. Upper extremity disability, measured by the Patient-Reported Outcomes Measurement Information System Physical Function-Upper Extremity (PROMIS PF-UE) Computer Adaptive Test (CAT).^14-16 This is a measure of physical limitations in the upper extremity. It can be completed with as few as 4 questions while still achieving high precision in scoring and thereby decreasing survey burden. PROMIS presents a continuous T-score with a mean of 50 and standard deviation (SD) of 10, with higher scores reflecting better physical function compared to the average of the US general population.¹⁵

After completing the initial questionnaire, the research assistant filled out the office and surgeon name and asked the surgeon to complete the diagnosis. All questionnaires were administered on an encrypted tablet via the secure, HIPAA-compliant electronic platform REDCap (Research Electronic Data Capture), a web-based application for building and managing online surveys and databases.¹⁷ The follow-up survey was sent automatically or was done by phone call as previously described. The follow-up survey consisted of (1) the 11-point ordinal measure of satisfaction with the surgeon, (2) the JSPPPE for perceived empathy, and (3) the PROMIS PF-UE for physical limitations in the upper extremity.

Analysis

Continuous variables are presented as mean ± SD and discrete data as proportions. We used Student’s t-tests to assess baseline differences between continuous variables and Fisher’s exact tests for discrete variables. To assess differences in satisfaction and perceived empathy after 2 weeks, we used Student’s paired t-tests. We created 2 multilevel multivariable linear regression models to assess factors associated with (1) change in satisfaction with the surgeon and (2) change in perceived physician empathy. These models account for correlation of patients treated by the same surgeon. We selected variables to be included in the final models by running multilevel models with only 1 independent variable of interest (Appendix 1). Variables with P < 0.10 were included in our final models. We also included change in PROMIS PF-UE in both models because this was our variable of interest. We considered P < 0.05 significant.

We performed a power analysis for the difference in patient satisfaction immediately after the first visit compared to 2 weeks later. Based on our pilot data where we found an initial mean satisfaction score of 9.4 and mean satisfaction score after 2 weeks of 9.1 (SD of difference 1.0), a priori power analysis showed that we needed a minimum sample size of 90 patients to detect a difference with power set at 0.80 and alpha set at 0.05. In order to account for loss to follow-up as previously noted,¹⁸ we enrolled 67% more patients (total of 150).

Results

Respondent Characteristics

None of the 150 patients were excluded from the analysis. The study patients’ mean age was 51 ± 16 years (range, 18-87 years), and 73 (49%) were men (Table 1). Mean scores directly after the visit were 9.4 ± 1.2 (range, 2-10) for satisfaction with the surgeon, 31 ± 5.2 (range, 9-35) for perceived physician empathy, and 40 ± 10 (range 15-56) for upper extremity disability. Most patients (n = 130, 87%) were seen in 2 of 5 offices, and 106 (71%) were seen by 2 out of 6 participating surgeons.

Ninety-seven (65%) patients completed their follow-up assessment 2 weeks after their initial visit, 49 (51%) by phone and 48 (49%) by email. This is a slightly better rate than the 36% rate reported in previous research.¹⁸ After 2 weeks, the mean score for satisfaction with the surgeon was 9.1 ± 1.5 (range, 0-10), the mean perceived empathy score was 31 ± 5.1 (range, 6-35), and the mean upper extremity disability score was 40 ± 8.7 (range, 23-56). Responders did not differ from nonresponders based on demographic data (Table 2). However, nonresponders had lower perceived empathy scores directly after their visit (P = 0.03) and none had initially chosen phone as their preferred mode of contact for follow-up (P < 0.001). A list of all diagnoses with frequencies the surgeons stated is listed in Appendix 2.

Difference in Satisfaction with the Surgeon

Satisfaction with the surgeon 2 weeks after the in-person visit was slightly, but significantly, lower on bivariate analysis compared to satisfaction with the surgeon immediately after the initial visit (–0.41 ± 1.2, P = 0.001; Table 3).

Difference in Perceived Physician Empathy

Perceived physician empathy 2 weeks after the in-person visit was not significantly lower on bivariate analysis compared to perceived physician empathy immediately after the initial visit (–0.71 ± 5.3, P = 0.19; Table 3).

Factors Associated with Change in Satisfaction with the Surgeon

Accounting for potential interaction of variables using multilevel multivariable analysis, change in disability of the upper extremity was not associated with change in satisfaction with the surgeon (regression coefficient [beta], 0.00 [95% confidence interval {CI}, –0.02 to 0.03]; standard error [SE], 0.01; P = 0.79 [Table 4]). Being Latino was independently associated with less change in satisfaction with the surgeon (beta coefficient, –0.57 [95% CI, –1.1 to 0.00]; SE, 0.29; P = 0.049).

Factors Associated with Change in Perceived Physician Empathy

Accounting for potential interaction of variables using multilevel multivariable analysis, change in disability of the upper extremity was not associated with change in perceived physician empathy (beta coefficient = 0.00 [95% CI, –0.10 to 0.11]; SE, 0.06; P = 0.93 [Table 4]). Race/ethnicity other than white or Latino was independently associated with more change in perceived physician empathy (beta coefficient, 3.5 [95% CI, 0.34 to 6.6]; SE, 1.6; P = 0.030), and preferring email as mode of contact for follow-up was independently associated with less change in perceived physician empathy (beta coefficient, –3.2 [95% CI, –5.2 to –1.3]; SE, 1.0; P = 0.001).

Discussion

Patient satisfaction is considered a quality measure^1-8 and is typically measured directly after an in-person visit. This study tested differences in patient satisfaction and perceived empathy immediately after the initial visit compared to 2 weeks later. In addition, we assessed whether change in disability was independently associated with change in satisfaction and empathy after the initial visit compared to 2 weeks later.

We acknowledge some study limitations. First, we only measured satisfaction based on 1 visit rather than multiple visits over time. It might be that satisfaction ratings differ when the physician-patient relationship is more established. However, we found overall high satisfaction ratings and a well-established relationship might not add to this finding. Second, surgeons were aware of the study and its purpose, which might have resulted in subconsciously altering the behavior to improve satisfaction. The effect of people acting differently as a result of being observed is called the Hawthorne effect.¹⁹ Third, we only used 1 simple ordinal measure to assess patient satisfaction with the surgeon. There is a wide variety of satisfaction measures,²⁰ though the focus of this study was not to test the best possible satisfaction measure but to assess changes in satisfaction over time and its predictors. The simple 11-point ordinal satisfaction measure has proved reliable.⁶ Fourth, 35% of patients did not make a second rating. This is not unusual for phone or email studies. Our response rate was relatively high compared to other studies in our field,¹⁸ perhaps because the time to the second assessment was only 2 weeks and all people were available for follow-up by phone. Fifth, we analyzed 4 surgeons as 1 group and 3 offices as 1 group since we did not enroll enough patients per surgeon and office for individual analysis. However, multilevel linear analysis takes surgeon specific factors into account within that group.

The finding that satisfaction with the surgeon after 2 weeks was significantly lower on bivariate analysis compared to immediately after the initial visit is different from a study that found small increases in satisfaction after 2 weeks and 3 months,¹ but comparable to another study in our field.²¹ Although significant, we believe the decrease in satisfaction is probably not clinically relevant. It might also be that satisfaction at follow-up is lower than measured, but that the least satisfied people did not respond on the follow-up survey.

We found no significant change in perceived empathy after 2 weeks. Since empathy is a strong driver of satisfaction,^2,4-7 we did not expect to find differing results for empathy and for satisfaction over time. Both satisfaction and empathy seem to be relatively durable measures with current measurement tools.

The finding that change in disability was neither independently associated with change in satisfaction nor change in empathy is consistent with prior research.^2,3,21 We cannot adequately study the impact of changes since we did not find an important change in either satisfaction or empathy over time. Jackson et al found higher satisfaction ratings over time in patients who had an increase in physical function and a decrease in symptoms.¹ They also found that met expectations was associated with higher satisfaction immediately after the visit, after 2 weeks, and after 3 months.¹ We feel that met expectations and fewer symptoms and limitations are likely highly co-linear with satisfaction. We therefore may not be able to learn much about one from the others.

The slight change we found in satisfaction with the surgeon among Latino patients was significantly less than the change among white patients. This suggests Latino patients might have a more stable opinion over time (a cultural phenomenon), or it might be spurious given the small number of Latino patients included in the study. The same can be said for the finding that race/ethnicity other than white or Latino was independently associated with greater change in empathy. Providing email as the preferred mode of contact was found to be independently associated with less change in perceived empathy compared to follow-up by phone. We had a 100% success rate for our follow-ups by phone. Our findings suggest that patients might more easily switch ratings on an 11-point ordinal scale than on a 5-item Likert scale. However, both measures are often rated at the ceiling of the scale.^2,21

Conclusion

Satisfaction and perceived empathy are relatively stable constructs, are not clearly associated with other factors, and are strongly correlated with one another. This study supports the research practice of measuring satisfaction immediately after the visit, which is more convenient for both participant and researcher and avoids the loss of more than one third of the patients, and those with a worse experience in particular. To improve the utility and interpretation of patient-reported experience measures such as these, we might direct our efforts to developing scales with less ceiling effect.

Corresponding author: David Ring, MD, PhD, Dell Medical School, The University of Texas at Austin, Health Discovery Building HDB 6.706, 1701 Trinity St., Austin, TX 78705; [email protected].

Financial disclosures: Dr. Ring has or may receive payment or benefits from Skeletal Dynamics, Wright Medical for elbow implants, Deputy Editor for Clinical Orthopaedics and Related Research, Universities and Hospitals, Lawyers outside the submitted work.

Dr. Teunis has or may receive payment or benefits from VCC, PATIENT+, and AO Trauma TK network unrelated to this work and consultant fees from Synthes.

From the Department of Surgery and Perioperative Care, Dell Medical School, The University of Texas at Austin, Austin, TX (Dr. Kortlever, Ms. Haidar, Dr. Reichel, Dr. Driscoll, Dr. Ring, and Dr. Vagner) and University Medical Center Utrecht, Utrecht, The Netherlands (Dr. Teunis).

Abstract

• Objective: Patient satisfaction is considered a quality measure. Satisfaction is typically measured directly after an in-person visit in research and 2 weeks later in practice surveys. We assessed if there was a difference in immediate and delayed measurement of satisfaction.
• Questions: (1) There is no difference in patient satisfaction (measured by Numerical Rating Scale [NRS]) and (2) perceived empathy (measured by the Jefferson Scale of Patient Perceptions of Physician Empathy [JSPPPE]) immediately after the initial visit compared to 2 weeks later. (3) Change in disability (measured by the Patient-Reported Outcome Measurement Information System Physical Function-Upper Extremity [PROMIS PF-UE]) is not independently associated with change in satisfaction and (4) empathy after the initial visit compared to 2 weeks later.
• Methods: 150 new patients completed a survey of demographics, satisfaction with the surgeon, rating of the surgeon’s empathy, and upper extremity specific limitations. The satisfaction, empathy, and limitation questionnaires were repeated 2 weeks later.
• Results: We found a slight but significant decrease in satisfaction 2 weeks after the in-person visit (–0.41, P = 0.001). There was no significant change in perceived empathy (–0.71, P = 0.19). Change in limitations did not account for a change in satisfaction (P = 0.79) or perceived empathy (P = 0.93).
• Conclusion: Satisfaction and perceived empathy are relatively stable constructs that can be measured immediately after the visit.

Keywords: satisfaction, empathy, change, upper extremity, disability.

Patient satisfaction is increasingly being used as a performance measure to evaluate quality of care.^1-8 Patient satisfaction correlates with adherence with recommended treatment.^1,6,8-10 Satisfaction measured on an 11-point ordinal scale immediately after the visit correlates strongly with the perception of clinician empathy.^2,3 Indeed, some satisfaction questionnaires such as the Medical Interview Satisfaction Scale (MISS)^11,12 have questions very similar to empathy questionnaires. It may be that satisfaction is a construct similar to feeling that your doctor listened and cared about you as an individual (perceived physician empathy).

Higher ratings of satisfaction also seem to be related to a physician’s communication style.^1,4,7-10 One study of 13 fertility doctors found that training in effective communication strategies led to improved patient satisfaction.⁷ A qualitative study of 36 patients, health professionals, and clinical support staff in an orthopaedic outpatient setting held interviews and focus group sessions to identify themes influencing patient satisfaction.⁴ Communication and expectation were among the 7 themes identified. We have noticed a high ceiling effect (maximum scores) with measures of patient satisfaction and perceived empathy.^2,3 Another study also noted a high ceiling effect when using an ordinal scale.⁵ It may be that people with a positive feeling shortly after a health care encounter give top ratings out of politeness or gratefulness. It is also possible they will feel differently a few weeks after they leave the office. Furthermore, ratings of satisfaction gathered by a practice or health care system for practice assessment/improvement are often obtained several days to weeks after the visit, while research often obtains satisfaction ratings immediately after the visit for practical reasons. There may be differences between immediate and delayed measurement of satisfaction beyond the mentioned social norms.

Therefore, this study tested the primary null hypothesis that there is no difference in patient satisfaction (measured by Numerical Rating Scale [NRS]) immediately after the initial visit compared to 2 weeks later. Additionally, we assessed the difference in perceived empathy immediately after the initial visit compared to 2 weeks later, and whether change in disability was independently associated with change in satisfaction and empathy after the initial visit compared to 2 weeks later.

Methods

Study Design

After Institutional Review Board approval of this prospective, longitudinal, observational cohort study, we prospectively enrolled 150 adult patients between November 29, 2017 and January 10, 2018. Patients were seen at 5 orthopaedic clinics in a large urban area. We included all new English-speaking patients aged 18 to 89 years who were visiting 1 of 6 participating orthopaedic surgeons for any upper extremity problem and who were able to provide informed consent. We excluded follow-up visits and patients who were unable to speak and understand English. Four research assistants who were not involved with patient treatment described the study to patients before or after the visit with the surgeon. We were granted a waiver of written informed consent; patients indicated their consent by completing the surveys.

Patients could choose either phone or email as their preferred mode of contact for follow-up in this study. For patients who selected email as the preferred mode of contact, the follow-up survey was sent automatically 2 weeks after completion date, and a maximum of 3 reminder emails with 2-day time intervals between them were sent to those who did not respond to the initial invitation. For patients who selected phone as the preferred mode of contact, the follow-up survey was done by an English-speaking research assistant who was not involved with patient treatment. When a response was not obtained on the initial phone call, 3 additional phone calls were made (1 later that same day and 2 the next day). One patient declined participation because he was not interested in the study and had no time after his visit.

Measurements

Patients were asked to complete a set of questionnaires at the end of their visit:

1. A demographic questionnaire consisting of preferred mode of contact for follow-up (phone or email), age, sex, race/ethnicity, marital status, education status, work status, insurance status, and type of visit (first visit or second opinion);

2. An 11-point ordinal measure of satisfaction with the surgeon, with scores ranging from 0 (Worst Surgeon Possible) to 10 (Best Surgeon Possible);

3. The patient’s rating of the surgeon’s empathy, measured by the Jefferson Scale of Patient Perceptions of Physician Empathy (JSPPPE).¹³ The JSPPPE is a 5-item questionnaire, measured on a 7-point Likert scale, with scores ranging from 1 (Strongly Disagree) to 7 (Strongly Agree), that assesses agreement with statements about the physician. The total score is the sum of all item scores (5-35), with higher scores representing a higher degree of perceived physician empathy.

4. Upper extremity disability, measured by the Patient-Reported Outcomes Measurement Information System Physical Function-Upper Extremity (PROMIS PF-UE) Computer Adaptive Test (CAT).^14-16 This is a measure of physical limitations in the upper extremity. It can be completed with as few as 4 questions while still achieving high precision in scoring and thereby decreasing survey burden. PROMIS presents a continuous T-score with a mean of 50 and standard deviation (SD) of 10, with higher scores reflecting better physical function compared to the average of the US general population.¹⁵

After completing the initial questionnaire, the research assistant filled out the office and surgeon name and asked the surgeon to complete the diagnosis. All questionnaires were administered on an encrypted tablet via the secure, HIPAA-compliant electronic platform REDCap (Research Electronic Data Capture), a web-based application for building and managing online surveys and databases.¹⁷ The follow-up survey was sent automatically or was done by phone call as previously described. The follow-up survey consisted of (1) the 11-point ordinal measure of satisfaction with the surgeon, (2) the JSPPPE for perceived empathy, and (3) the PROMIS PF-UE for physical limitations in the upper extremity.

Analysis

Continuous variables are presented as mean ± SD and discrete data as proportions. We used Student’s t-tests to assess baseline differences between continuous variables and Fisher’s exact tests for discrete variables. To assess differences in satisfaction and perceived empathy after 2 weeks, we used Student’s paired t-tests. We created 2 multilevel multivariable linear regression models to assess factors associated with (1) change in satisfaction with the surgeon and (2) change in perceived physician empathy. These models account for correlation of patients treated by the same surgeon. We selected variables to be included in the final models by running multilevel models with only 1 independent variable of interest (Appendix 1). Variables with P < 0.10 were included in our final models. We also included change in PROMIS PF-UE in both models because this was our variable of interest. We considered P < 0.05 significant.

We performed a power analysis for the difference in patient satisfaction immediately after the first visit compared to 2 weeks later. Based on our pilot data where we found an initial mean satisfaction score of 9.4 and mean satisfaction score after 2 weeks of 9.1 (SD of difference 1.0), a priori power analysis showed that we needed a minimum sample size of 90 patients to detect a difference with power set at 0.80 and alpha set at 0.05. In order to account for loss to follow-up as previously noted,¹⁸ we enrolled 67% more patients (total of 150).

Results

Respondent Characteristics

None of the 150 patients were excluded from the analysis. The study patients’ mean age was 51 ± 16 years (range, 18-87 years), and 73 (49%) were men (Table 1). Mean scores directly after the visit were 9.4 ± 1.2 (range, 2-10) for satisfaction with the surgeon, 31 ± 5.2 (range, 9-35) for perceived physician empathy, and 40 ± 10 (range 15-56) for upper extremity disability. Most patients (n = 130, 87%) were seen in 2 of 5 offices, and 106 (71%) were seen by 2 out of 6 participating surgeons.

Ninety-seven (65%) patients completed their follow-up assessment 2 weeks after their initial visit, 49 (51%) by phone and 48 (49%) by email. This is a slightly better rate than the 36% rate reported in previous research.¹⁸ After 2 weeks, the mean score for satisfaction with the surgeon was 9.1 ± 1.5 (range, 0-10), the mean perceived empathy score was 31 ± 5.1 (range, 6-35), and the mean upper extremity disability score was 40 ± 8.7 (range, 23-56). Responders did not differ from nonresponders based on demographic data (Table 2). However, nonresponders had lower perceived empathy scores directly after their visit (P = 0.03) and none had initially chosen phone as their preferred mode of contact for follow-up (P < 0.001). A list of all diagnoses with frequencies the surgeons stated is listed in Appendix 2.

Difference in Satisfaction with the Surgeon

Satisfaction with the surgeon 2 weeks after the in-person visit was slightly, but significantly, lower on bivariate analysis compared to satisfaction with the surgeon immediately after the initial visit (–0.41 ± 1.2, P = 0.001; Table 3).

Difference in Perceived Physician Empathy

Perceived physician empathy 2 weeks after the in-person visit was not significantly lower on bivariate analysis compared to perceived physician empathy immediately after the initial visit (–0.71 ± 5.3, P = 0.19; Table 3).

Factors Associated with Change in Satisfaction with the Surgeon

Accounting for potential interaction of variables using multilevel multivariable analysis, change in disability of the upper extremity was not associated with change in satisfaction with the surgeon (regression coefficient [beta], 0.00 [95% confidence interval {CI}, –0.02 to 0.03]; standard error [SE], 0.01; P = 0.79 [Table 4]). Being Latino was independently associated with less change in satisfaction with the surgeon (beta coefficient, –0.57 [95% CI, –1.1 to 0.00]; SE, 0.29; P = 0.049).

Factors Associated with Change in Perceived Physician Empathy

Accounting for potential interaction of variables using multilevel multivariable analysis, change in disability of the upper extremity was not associated with change in perceived physician empathy (beta coefficient = 0.00 [95% CI, –0.10 to 0.11]; SE, 0.06; P = 0.93 [Table 4]). Race/ethnicity other than white or Latino was independently associated with more change in perceived physician empathy (beta coefficient, 3.5 [95% CI, 0.34 to 6.6]; SE, 1.6; P = 0.030), and preferring email as mode of contact for follow-up was independently associated with less change in perceived physician empathy (beta coefficient, –3.2 [95% CI, –5.2 to –1.3]; SE, 1.0; P = 0.001).

Discussion

Patient satisfaction is considered a quality measure^1-8 and is typically measured directly after an in-person visit. This study tested differences in patient satisfaction and perceived empathy immediately after the initial visit compared to 2 weeks later. In addition, we assessed whether change in disability was independently associated with change in satisfaction and empathy after the initial visit compared to 2 weeks later.

We acknowledge some study limitations. First, we only measured satisfaction based on 1 visit rather than multiple visits over time. It might be that satisfaction ratings differ when the physician-patient relationship is more established. However, we found overall high satisfaction ratings and a well-established relationship might not add to this finding. Second, surgeons were aware of the study and its purpose, which might have resulted in subconsciously altering the behavior to improve satisfaction. The effect of people acting differently as a result of being observed is called the Hawthorne effect.¹⁹ Third, we only used 1 simple ordinal measure to assess patient satisfaction with the surgeon. There is a wide variety of satisfaction measures,²⁰ though the focus of this study was not to test the best possible satisfaction measure but to assess changes in satisfaction over time and its predictors. The simple 11-point ordinal satisfaction measure has proved reliable.⁶ Fourth, 35% of patients did not make a second rating. This is not unusual for phone or email studies. Our response rate was relatively high compared to other studies in our field,¹⁸ perhaps because the time to the second assessment was only 2 weeks and all people were available for follow-up by phone. Fifth, we analyzed 4 surgeons as 1 group and 3 offices as 1 group since we did not enroll enough patients per surgeon and office for individual analysis. However, multilevel linear analysis takes surgeon specific factors into account within that group.

The finding that satisfaction with the surgeon after 2 weeks was significantly lower on bivariate analysis compared to immediately after the initial visit is different from a study that found small increases in satisfaction after 2 weeks and 3 months,¹ but comparable to another study in our field.²¹ Although significant, we believe the decrease in satisfaction is probably not clinically relevant. It might also be that satisfaction at follow-up is lower than measured, but that the least satisfied people did not respond on the follow-up survey.

We found no significant change in perceived empathy after 2 weeks. Since empathy is a strong driver of satisfaction,^2,4-7 we did not expect to find differing results for empathy and for satisfaction over time. Both satisfaction and empathy seem to be relatively durable measures with current measurement tools.

The finding that change in disability was neither independently associated with change in satisfaction nor change in empathy is consistent with prior research.^2,3,21 We cannot adequately study the impact of changes since we did not find an important change in either satisfaction or empathy over time. Jackson et al found higher satisfaction ratings over time in patients who had an increase in physical function and a decrease in symptoms.¹ They also found that met expectations was associated with higher satisfaction immediately after the visit, after 2 weeks, and after 3 months.¹ We feel that met expectations and fewer symptoms and limitations are likely highly co-linear with satisfaction. We therefore may not be able to learn much about one from the others.

The slight change we found in satisfaction with the surgeon among Latino patients was significantly less than the change among white patients. This suggests Latino patients might have a more stable opinion over time (a cultural phenomenon), or it might be spurious given the small number of Latino patients included in the study. The same can be said for the finding that race/ethnicity other than white or Latino was independently associated with greater change in empathy. Providing email as the preferred mode of contact was found to be independently associated with less change in perceived empathy compared to follow-up by phone. We had a 100% success rate for our follow-ups by phone. Our findings suggest that patients might more easily switch ratings on an 11-point ordinal scale than on a 5-item Likert scale. However, both measures are often rated at the ceiling of the scale.^2,21

Conclusion

Satisfaction and perceived empathy are relatively stable constructs, are not clearly associated with other factors, and are strongly correlated with one another. This study supports the research practice of measuring satisfaction immediately after the visit, which is more convenient for both participant and researcher and avoids the loss of more than one third of the patients, and those with a worse experience in particular. To improve the utility and interpretation of patient-reported experience measures such as these, we might direct our efforts to developing scales with less ceiling effect.

Corresponding author: David Ring, MD, PhD, Dell Medical School, The University of Texas at Austin, Health Discovery Building HDB 6.706, 1701 Trinity St., Austin, TX 78705; [email protected].

Financial disclosures: Dr. Ring has or may receive payment or benefits from Skeletal Dynamics, Wright Medical for elbow implants, Deputy Editor for Clinical Orthopaedics and Related Research, Universities and Hospitals, Lawyers outside the submitted work.

Dr. Teunis has or may receive payment or benefits from VCC, PATIENT+, and AO Trauma TK network unrelated to this work and consultant fees from Synthes.

Introduction

Soft tissue sarcomas (STS) are a heterogeneous group of cancerous tumors, comprised of more than 50 histological subtypes that develop from soft tissues of the body (eg, fat, muscles, nerve tissue, deep skin tissue, visceral nonepithelial tissue). Due to many factors, not limited to the heterogeneity of this set of diseases and lack of screening tests, reaching a diagnosis of STS is challenging for the general practitioner as well as for the oncologist. Sarcomas may present with nonspecific and often indolent symptomology, depending on the specific histological subtype. According to the American Cancer Society, the signs and symptoms of a sarcoma include a new or growing lump, worsening abdominal pain, blood in stool or vomit, and black stools (due to abdominal bleeding).¹ Unfortunately, these symptoms could be indicative of any number of other health conditions and are nonspecific to sarcoma.

As with many cancers, the early detection of disease when it may be completely resected could lead to a cure, whereas diagnosis when the disease is no longer amenable to surgery will impact patient survival. Among all forms of STS, early diagnosis when the patient has only localized disease is associated with an 80.8% five-year survival rate, which decreases to 16.4% for patients whose disease has already metastasized to other parts of the body at the time of diagnosis.²

Previous work has evaluated the relationship between duration of symptoms that may lead to a diagnosis of sarcoma and cancer outcomes. A retrospective analysis of a cohort of adults with bone or STS found no correlation between patient recall of duration of prediagnosis symptoms and survival or metastatic disease at diagnosis.^3,4 Little other research was identified that examined the challenges of identifying a potential sarcoma. Despite the gap in knowledge, advocacy and patient-centered organizations emphasize the risk of delayed diagnosis and report high levels of stress and frustration among patients by the time an accurate diagnosis is obtained.⁵ The objective of this study was to quantify the health care experience and misdiagnoses that occurred prior to a sarcoma diagnosis compared to a cohort of matched controls.

Methods

A retrospective observational database study was conducted using detailed resource utilization and cost data from the Truven MarketScan claims database. Truven MarketScan^® is a HIPAA-compliant, fully integrated patient-level database containing inpatient, outpatient, drug, lab, health risk assessment, and benefit design information from commercial and Medicare supplemental insurance plans. Additionally, the Health and Productivity Management (HPM) database, containing workplace absence, short-term disability, long-term disability, and worker’s compensation data, is linked at the individual patient level. The linkage of the claims and HPM database was used for this study.

Patients were eligible for inclusion in the cohort of a sarcoma if they had at least two ICD-9 codes of 171.x on two different days between July 1, 2004, and March 30, 2014. The date of the first eligible code was considered the index date. Patients were required to have at least 6 months of health care plan enrollment prior to the first eligible ICD-9 code to allow for prediagnosis activity to be identified in the database. Patients were also required to be 18 years of age or older on the first eligible ICD-9 code date. Patients were excluded who had evidence suggesting a diagnosis of osteosarcoma, Kaposi’s sarcoma, or gastrointestinal stromal tumors (treatment with methotrexate, ICD-9 codes of 176.x, 171.x, or 238.1), a history of any cancer before the eligible sarcoma ICD-9 code, or history of systemic anticancer therapy during the 6-month pre-index period. All patients meeting eligibility criteria were included in the matching algorithm to identify the control cohort.

The matched control cohort was required to have at least the same duration of follow-up at the case level as the matched sarcoma patient, could not have any evidence of any malignancy at any time in the database, nor could have received any systemic anticancer therapy at any time. Controls were randomly selected from the more than 100 million individual patient cases in the MarketScan database to be matched to the eligible sarcoma patient cohort exactly on age, geographic region of residence, health insurance plan type, gender, noncancer comorbid conditions (measured by Charlson Comorbidity Index items), and employment status. All factors were exact matched at the sarcoma cohort index diagnosis date. In the case of missing variables, patients were matched on missingness (eg, a case with missing employment status would be matched to a control with missing employment status).

The eligible time period for the index date of the possible sarcoma cohort and matched controls was between July 1, 2004, and March 30, 2014, which allowed for a minimum of 1-year follow-up through the end of the database available at the time of analysis.

All ICD-9 diagnostic and procedure codes present in the matched 6-month time period pre-index diagnosis were compared to explore factors that may be more likely to be present in the sarcoma cohort compared to matched controls. Univariate analysis was conducted for each prediagnosis variable. Analyses were conducted using T test for continuous variables, and Chi-square or Fisher’s exact test was used for categorical variables.

Number of physician visits, inpatient hospital stays, surgical procedures, and emergency room visits were compared between those in the sarcoma cohort and matched controls during the matched 6-month pre-index period. The post-index diagnosis employment status was also compared between groups using the HPM database. Comparisons between the sarcoma cohort and control cohort were made among the actively employed patients at baseline related to the proportion of patients who continued active employment, the proportion who permanently discontinued work, and the proportion who initially discontinued work and then returned to work at a later time. No adjustments were made for multiple comparisons.

Results

A total of 7826 controls were each matched to patients in the sarcoma cohort. The baseline characteristics of the study cohorts are provided in Table 1. 

During the 6-month period before the sarcoma diagnosis (prediagnosis period), patients had significantly greater frequency of diagnoses identified than controls for uncertain neoplasms, limb pain, and hypertension (all P<.001, Table 2). 

Similarly, the majority of health care resource utilization factors evaluated showed statistically higher health care use among patients later suspected of having sarcoma than matched controls (Table 3). 

Discussion

The symptoms experienced by patients that were recorded in claims were significantly higher across multiple categories than matched controls. However, the rates were relatively low, demonstrating the wide variability in the presentation of sarcoma. Patients had a variety of recorded problems, not limited to a lump or pain, but including hematologic, gastric, and cardiac concerns, that differed from those who had no suspected sarcoma. These factors highlight the challenges that may be facing patients who have an undetected sarcoma.

An expected finding was the difference in duration of follow-up between cohorts. This could be due to longer survival of those without a sarcoma diagnosis or due to insurance changes among those who had a sarcoma diagnosis. The absence of death data did not allow for further exploration of this finding within this study. Future research may wish to identify more comprehensive datasets to allow for the objective evaluation of the differences in time to diagnosis and stage of disease and survival, which would be the ultimate goal in order to develop potential strategies to improve patient outcomes.

This study was limited in that the sarcoma diagnosis could not be verified in a clinical record due to the de-identified nature of the claims data used for this study. Prior work has shown that the ICD coding for sarcoma is incomplete^6,7; therefore it is likely there are many other patients in the claims dataset who had a suspected sarcoma but who did not have a 171.x code recorded. Hence, this study is limited to a comparison of a cohort for whom the provider specified a sarcoma code in their billing records. While there are gaps in the ability to identify the entire population of sarcoma patients, the patients with ICD codes used in this study are likely true sarcoma cases. Prior work has demonstrated that the presence of these codes accurately reflects a true sarcoma diagnosis.⁷ However, given the concerns with ICD coding, two sarcoma codes were required on unique days to reduce the risk of single rule-out codes or data entry error. Patients diagnosed with sarcoma demonstrate significantly greater health care resource use across variables as matched controls during the 6-month period leading to diagnosis, supporting the observations within advocacy and patient reports of the challenges faced during the process to reach an accurate diagnosis. This work may provide the initial basis for the development of strategies to more rapidly identify a potential sarcoma. Future research could also evaluate more than 6 months prior to diagnosis, to quantify the duration of time during which these differences versus controls may exist. Additionally, the cost of care may be of interest to future research to better quantify the burden of misdiagnosis on the health care system.

Acknowledgement
The authors would like to acknowledge Yun Fang, MS, for her support in the SAS coding for the analysis of this study.

Corresponding Author
Lisa M. Hess, PhD, Eli Lilly and Company. [email protected]

Disclosures
No funding was received or exchanged in the conceptualization, conduct, data collection, analysis, interpretation, or writing related to this study. This unfunded study was conducted by employees of Eli Lilly and Company.

Introduction

Soft tissue sarcomas (STS) are a heterogeneous group of cancerous tumors, comprised of more than 50 histological subtypes that develop from soft tissues of the body (eg, fat, muscles, nerve tissue, deep skin tissue, visceral nonepithelial tissue). Due to many factors, not limited to the heterogeneity of this set of diseases and lack of screening tests, reaching a diagnosis of STS is challenging for the general practitioner as well as for the oncologist. Sarcomas may present with nonspecific and often indolent symptomology, depending on the specific histological subtype. According to the American Cancer Society, the signs and symptoms of a sarcoma include a new or growing lump, worsening abdominal pain, blood in stool or vomit, and black stools (due to abdominal bleeding).¹ Unfortunately, these symptoms could be indicative of any number of other health conditions and are nonspecific to sarcoma.

As with many cancers, the early detection of disease when it may be completely resected could lead to a cure, whereas diagnosis when the disease is no longer amenable to surgery will impact patient survival. Among all forms of STS, early diagnosis when the patient has only localized disease is associated with an 80.8% five-year survival rate, which decreases to 16.4% for patients whose disease has already metastasized to other parts of the body at the time of diagnosis.²

Previous work has evaluated the relationship between duration of symptoms that may lead to a diagnosis of sarcoma and cancer outcomes. A retrospective analysis of a cohort of adults with bone or STS found no correlation between patient recall of duration of prediagnosis symptoms and survival or metastatic disease at diagnosis.^3,4 Little other research was identified that examined the challenges of identifying a potential sarcoma. Despite the gap in knowledge, advocacy and patient-centered organizations emphasize the risk of delayed diagnosis and report high levels of stress and frustration among patients by the time an accurate diagnosis is obtained.⁵ The objective of this study was to quantify the health care experience and misdiagnoses that occurred prior to a sarcoma diagnosis compared to a cohort of matched controls.

Methods

A retrospective observational database study was conducted using detailed resource utilization and cost data from the Truven MarketScan claims database. Truven MarketScan^® is a HIPAA-compliant, fully integrated patient-level database containing inpatient, outpatient, drug, lab, health risk assessment, and benefit design information from commercial and Medicare supplemental insurance plans. Additionally, the Health and Productivity Management (HPM) database, containing workplace absence, short-term disability, long-term disability, and worker’s compensation data, is linked at the individual patient level. The linkage of the claims and HPM database was used for this study.

Patients were eligible for inclusion in the cohort of a sarcoma if they had at least two ICD-9 codes of 171.x on two different days between July 1, 2004, and March 30, 2014. The date of the first eligible code was considered the index date. Patients were required to have at least 6 months of health care plan enrollment prior to the first eligible ICD-9 code to allow for prediagnosis activity to be identified in the database. Patients were also required to be 18 years of age or older on the first eligible ICD-9 code date. Patients were excluded who had evidence suggesting a diagnosis of osteosarcoma, Kaposi’s sarcoma, or gastrointestinal stromal tumors (treatment with methotrexate, ICD-9 codes of 176.x, 171.x, or 238.1), a history of any cancer before the eligible sarcoma ICD-9 code, or history of systemic anticancer therapy during the 6-month pre-index period. All patients meeting eligibility criteria were included in the matching algorithm to identify the control cohort.

The matched control cohort was required to have at least the same duration of follow-up at the case level as the matched sarcoma patient, could not have any evidence of any malignancy at any time in the database, nor could have received any systemic anticancer therapy at any time. Controls were randomly selected from the more than 100 million individual patient cases in the MarketScan database to be matched to the eligible sarcoma patient cohort exactly on age, geographic region of residence, health insurance plan type, gender, noncancer comorbid conditions (measured by Charlson Comorbidity Index items), and employment status. All factors were exact matched at the sarcoma cohort index diagnosis date. In the case of missing variables, patients were matched on missingness (eg, a case with missing employment status would be matched to a control with missing employment status).

The eligible time period for the index date of the possible sarcoma cohort and matched controls was between July 1, 2004, and March 30, 2014, which allowed for a minimum of 1-year follow-up through the end of the database available at the time of analysis.

All ICD-9 diagnostic and procedure codes present in the matched 6-month time period pre-index diagnosis were compared to explore factors that may be more likely to be present in the sarcoma cohort compared to matched controls. Univariate analysis was conducted for each prediagnosis variable. Analyses were conducted using T test for continuous variables, and Chi-square or Fisher’s exact test was used for categorical variables.

Number of physician visits, inpatient hospital stays, surgical procedures, and emergency room visits were compared between those in the sarcoma cohort and matched controls during the matched 6-month pre-index period. The post-index diagnosis employment status was also compared between groups using the HPM database. Comparisons between the sarcoma cohort and control cohort were made among the actively employed patients at baseline related to the proportion of patients who continued active employment, the proportion who permanently discontinued work, and the proportion who initially discontinued work and then returned to work at a later time. No adjustments were made for multiple comparisons.

Results

A total of 7826 controls were each matched to patients in the sarcoma cohort. The baseline characteristics of the study cohorts are provided in Table 1. 

During the 6-month period before the sarcoma diagnosis (prediagnosis period), patients had significantly greater frequency of diagnoses identified than controls for uncertain neoplasms, limb pain, and hypertension (all P<.001, Table 2). 

Similarly, the majority of health care resource utilization factors evaluated showed statistically higher health care use among patients later suspected of having sarcoma than matched controls (Table 3). 

Discussion

The symptoms experienced by patients that were recorded in claims were significantly higher across multiple categories than matched controls. However, the rates were relatively low, demonstrating the wide variability in the presentation of sarcoma. Patients had a variety of recorded problems, not limited to a lump or pain, but including hematologic, gastric, and cardiac concerns, that differed from those who had no suspected sarcoma. These factors highlight the challenges that may be facing patients who have an undetected sarcoma.

An expected finding was the difference in duration of follow-up between cohorts. This could be due to longer survival of those without a sarcoma diagnosis or due to insurance changes among those who had a sarcoma diagnosis. The absence of death data did not allow for further exploration of this finding within this study. Future research may wish to identify more comprehensive datasets to allow for the objective evaluation of the differences in time to diagnosis and stage of disease and survival, which would be the ultimate goal in order to develop potential strategies to improve patient outcomes.

This study was limited in that the sarcoma diagnosis could not be verified in a clinical record due to the de-identified nature of the claims data used for this study. Prior work has shown that the ICD coding for sarcoma is incomplete^6,7; therefore it is likely there are many other patients in the claims dataset who had a suspected sarcoma but who did not have a 171.x code recorded. Hence, this study is limited to a comparison of a cohort for whom the provider specified a sarcoma code in their billing records. While there are gaps in the ability to identify the entire population of sarcoma patients, the patients with ICD codes used in this study are likely true sarcoma cases. Prior work has demonstrated that the presence of these codes accurately reflects a true sarcoma diagnosis.⁷ However, given the concerns with ICD coding, two sarcoma codes were required on unique days to reduce the risk of single rule-out codes or data entry error. Patients diagnosed with sarcoma demonstrate significantly greater health care resource use across variables as matched controls during the 6-month period leading to diagnosis, supporting the observations within advocacy and patient reports of the challenges faced during the process to reach an accurate diagnosis. This work may provide the initial basis for the development of strategies to more rapidly identify a potential sarcoma. Future research could also evaluate more than 6 months prior to diagnosis, to quantify the duration of time during which these differences versus controls may exist. Additionally, the cost of care may be of interest to future research to better quantify the burden of misdiagnosis on the health care system.

Acknowledgement
The authors would like to acknowledge Yun Fang, MS, for her support in the SAS coding for the analysis of this study.

Corresponding Author
Lisa M. Hess, PhD, Eli Lilly and Company. [email protected]

Disclosures
No funding was received or exchanged in the conceptualization, conduct, data collection, analysis, interpretation, or writing related to this study. This unfunded study was conducted by employees of Eli Lilly and Company.

Introduction

Soft tissue sarcomas (STS) are a heterogeneous group of cancerous tumors, comprised of more than 50 histological subtypes that develop from soft tissues of the body (eg, fat, muscles, nerve tissue, deep skin tissue, visceral nonepithelial tissue). Due to many factors, not limited to the heterogeneity of this set of diseases and lack of screening tests, reaching a diagnosis of STS is challenging for the general practitioner as well as for the oncologist. Sarcomas may present with nonspecific and often indolent symptomology, depending on the specific histological subtype. According to the American Cancer Society, the signs and symptoms of a sarcoma include a new or growing lump, worsening abdominal pain, blood in stool or vomit, and black stools (due to abdominal bleeding).¹ Unfortunately, these symptoms could be indicative of any number of other health conditions and are nonspecific to sarcoma.

As with many cancers, the early detection of disease when it may be completely resected could lead to a cure, whereas diagnosis when the disease is no longer amenable to surgery will impact patient survival. Among all forms of STS, early diagnosis when the patient has only localized disease is associated with an 80.8% five-year survival rate, which decreases to 16.4% for patients whose disease has already metastasized to other parts of the body at the time of diagnosis.²

Previous work has evaluated the relationship between duration of symptoms that may lead to a diagnosis of sarcoma and cancer outcomes. A retrospective analysis of a cohort of adults with bone or STS found no correlation between patient recall of duration of prediagnosis symptoms and survival or metastatic disease at diagnosis.^3,4 Little other research was identified that examined the challenges of identifying a potential sarcoma. Despite the gap in knowledge, advocacy and patient-centered organizations emphasize the risk of delayed diagnosis and report high levels of stress and frustration among patients by the time an accurate diagnosis is obtained.⁵ The objective of this study was to quantify the health care experience and misdiagnoses that occurred prior to a sarcoma diagnosis compared to a cohort of matched controls.

Methods

A retrospective observational database study was conducted using detailed resource utilization and cost data from the Truven MarketScan claims database. Truven MarketScan^® is a HIPAA-compliant, fully integrated patient-level database containing inpatient, outpatient, drug, lab, health risk assessment, and benefit design information from commercial and Medicare supplemental insurance plans. Additionally, the Health and Productivity Management (HPM) database, containing workplace absence, short-term disability, long-term disability, and worker’s compensation data, is linked at the individual patient level. The linkage of the claims and HPM database was used for this study.

Patients were eligible for inclusion in the cohort of a sarcoma if they had at least two ICD-9 codes of 171.x on two different days between July 1, 2004, and March 30, 2014. The date of the first eligible code was considered the index date. Patients were required to have at least 6 months of health care plan enrollment prior to the first eligible ICD-9 code to allow for prediagnosis activity to be identified in the database. Patients were also required to be 18 years of age or older on the first eligible ICD-9 code date. Patients were excluded who had evidence suggesting a diagnosis of osteosarcoma, Kaposi’s sarcoma, or gastrointestinal stromal tumors (treatment with methotrexate, ICD-9 codes of 176.x, 171.x, or 238.1), a history of any cancer before the eligible sarcoma ICD-9 code, or history of systemic anticancer therapy during the 6-month pre-index period. All patients meeting eligibility criteria were included in the matching algorithm to identify the control cohort.

The matched control cohort was required to have at least the same duration of follow-up at the case level as the matched sarcoma patient, could not have any evidence of any malignancy at any time in the database, nor could have received any systemic anticancer therapy at any time. Controls were randomly selected from the more than 100 million individual patient cases in the MarketScan database to be matched to the eligible sarcoma patient cohort exactly on age, geographic region of residence, health insurance plan type, gender, noncancer comorbid conditions (measured by Charlson Comorbidity Index items), and employment status. All factors were exact matched at the sarcoma cohort index diagnosis date. In the case of missing variables, patients were matched on missingness (eg, a case with missing employment status would be matched to a control with missing employment status).

The eligible time period for the index date of the possible sarcoma cohort and matched controls was between July 1, 2004, and March 30, 2014, which allowed for a minimum of 1-year follow-up through the end of the database available at the time of analysis.

All ICD-9 diagnostic and procedure codes present in the matched 6-month time period pre-index diagnosis were compared to explore factors that may be more likely to be present in the sarcoma cohort compared to matched controls. Univariate analysis was conducted for each prediagnosis variable. Analyses were conducted using T test for continuous variables, and Chi-square or Fisher’s exact test was used for categorical variables.

Number of physician visits, inpatient hospital stays, surgical procedures, and emergency room visits were compared between those in the sarcoma cohort and matched controls during the matched 6-month pre-index period. The post-index diagnosis employment status was also compared between groups using the HPM database. Comparisons between the sarcoma cohort and control cohort were made among the actively employed patients at baseline related to the proportion of patients who continued active employment, the proportion who permanently discontinued work, and the proportion who initially discontinued work and then returned to work at a later time. No adjustments were made for multiple comparisons.

Results

A total of 7826 controls were each matched to patients in the sarcoma cohort. The baseline characteristics of the study cohorts are provided in Table 1. 

During the 6-month period before the sarcoma diagnosis (prediagnosis period), patients had significantly greater frequency of diagnoses identified than controls for uncertain neoplasms, limb pain, and hypertension (all P<.001, Table 2). 

Similarly, the majority of health care resource utilization factors evaluated showed statistically higher health care use among patients later suspected of having sarcoma than matched controls (Table 3). 

Discussion

The symptoms experienced by patients that were recorded in claims were significantly higher across multiple categories than matched controls. However, the rates were relatively low, demonstrating the wide variability in the presentation of sarcoma. Patients had a variety of recorded problems, not limited to a lump or pain, but including hematologic, gastric, and cardiac concerns, that differed from those who had no suspected sarcoma. These factors highlight the challenges that may be facing patients who have an undetected sarcoma.

An expected finding was the difference in duration of follow-up between cohorts. This could be due to longer survival of those without a sarcoma diagnosis or due to insurance changes among those who had a sarcoma diagnosis. The absence of death data did not allow for further exploration of this finding within this study. Future research may wish to identify more comprehensive datasets to allow for the objective evaluation of the differences in time to diagnosis and stage of disease and survival, which would be the ultimate goal in order to develop potential strategies to improve patient outcomes.

This study was limited in that the sarcoma diagnosis could not be verified in a clinical record due to the de-identified nature of the claims data used for this study. Prior work has shown that the ICD coding for sarcoma is incomplete^6,7; therefore it is likely there are many other patients in the claims dataset who had a suspected sarcoma but who did not have a 171.x code recorded. Hence, this study is limited to a comparison of a cohort for whom the provider specified a sarcoma code in their billing records. While there are gaps in the ability to identify the entire population of sarcoma patients, the patients with ICD codes used in this study are likely true sarcoma cases. Prior work has demonstrated that the presence of these codes accurately reflects a true sarcoma diagnosis.⁷ However, given the concerns with ICD coding, two sarcoma codes were required on unique days to reduce the risk of single rule-out codes or data entry error. Patients diagnosed with sarcoma demonstrate significantly greater health care resource use across variables as matched controls during the 6-month period leading to diagnosis, supporting the observations within advocacy and patient reports of the challenges faced during the process to reach an accurate diagnosis. This work may provide the initial basis for the development of strategies to more rapidly identify a potential sarcoma. Future research could also evaluate more than 6 months prior to diagnosis, to quantify the duration of time during which these differences versus controls may exist. Additionally, the cost of care may be of interest to future research to better quantify the burden of misdiagnosis on the health care system.

Acknowledgement
The authors would like to acknowledge Yun Fang, MS, for her support in the SAS coding for the analysis of this study.

Corresponding Author
Lisa M. Hess, PhD, Eli Lilly and Company. [email protected]

Disclosures
No funding was received or exchanged in the conceptualization, conduct, data collection, analysis, interpretation, or writing related to this study. This unfunded study was conducted by employees of Eli Lilly and Company.

In recent years, hospitals have made considerable efforts to improve transitions of care, in part due to financial incentives from the Medicare Hospital Readmission Reduction Program (HRRP).¹ Initially focusing on three medical conditions, the HRRP has been associated with significant reductions in readmission rates.² Importantly, a small proportion of patients accounts for a very large proportion of hospital readmissions and hospital use.^3,4 Frequently hospitalized patients often have multiple chronic conditions and unique needs which may not be met by conventional approaches to healthcare delivery, including those influenced by the HRRP.^4-6 In light of this challenge, some hospitals have developed programs specifically focused on frequently hospitalized patients. A recent systematic review of these programs found relatively few studies of high quality, providing only limited insight in designing interventions to support this population.⁷ Moreover, no studies appear to have incorporated the patients’ perspectives into the design or adaptation of the model. Members of our research team developed and implemented the Complex High Admission Management Program (CHAMP) in January 2016 to address the needs of frequently hospitalized patients in our hospital. To enhance CHAMP and inform the design of programs serving similar populations in other health systems, we sought to identify factors associated with the onset and continuation of high hospital use. Our research question was, from the patients’ perspective, what factors contribute to patients’ becoming and continuing to be high users of hospital care.

Setting, Study Design, and Participants

This qualitative study took place at Northwestern Memorial Hospital (NMH), an 894-bed urban academic hospital located in Chicago, Illinois. Between December 2016 and September 2017, we recruited adult patients admitted to the general medicine services. Eligible participants were identified with the assistance of a daily Northwestern Medicine Electronic Data Warehouse (EDW) search and included patients with two unplanned 30-day inpatient readmissions to NMH within the prior 12 months, in addition to one or more of the following criteria: (1) at least one readmission in the last six months; (2) a referral from one of the patient’s medical providers; or (3) at least three observation visits. We excluded patients whose preferred language was not English and those disoriented to person, place, or time. Considering NMH data showing that approximately one-third of high-utilizer patients have sickle cell disease, we used purposive sampling with the goal to compare findings within and between two groups of participants; those with and those without sickle cell disease. Our study was deemed exempt by the Northwestern University Institutional Review Board.

Participant Enrollment and Data Collection

We created an interview guide based on the research team’s experience with this population, a literature review, and our research question (See Appendix).^8,9 A research coordinator approached eligible participants during their hospital stay. The coordinator explained the study to eligible participants and obtained verbal consent for participation. The research coordinator then conducted one-on-one semi-structured interviews. Interviews were audio recorded for subsequent transcription and coding. Each interview lasted approximately 45 minutes. Participants were compensated with a $20 gift card for their time.

Analysis

Digital audio recordings from interviews were transcribed verbatim, deidentified, and analyzed using an iterative inductive team-based approach to coding.¹⁰ In our first cycle coding, all coders (KJO, SF, MMC, LO, KAC) independently reviewed and coded three transcripts using descriptive coding and subcoding to generate a preliminary codebook with code definitions.^10,11 Following the meetings to compare and compile our initial coding, each researcher then independently recoded the three transcripts with the developed codebook. The researchers met again to triangulate perspectives and reach a consensus on the final codebook. Using multiple coders is a standard process to control for subjective bias that one coder could bring to the coding process.¹² Following this meeting, the coders split into two teams of two (KJO, SF, and MMC, LO) to complete the coding of the remaining transcripts. Each team member independently coded the assigned transcripts and reconciled their codes with their counterpart; any discrepancies were resolved through discussion. Using this strategy, every transcript was coded by at least two team members. Our second coding cycle utilized pattern coding and involved identifying consistency both within and between transcripts; discovering associations between codes.^10,11,13 Constant comparison was used to compare responses among all participants, as well as between sickle-cell and nonsickle-cell participants.^13,14 Following team coding and reconciling, the analyses were presented to a broader research team for additional feedback and critique. All analyses were conducted using Dedoose version 8.0.35 (Los Angeles, California). Participant recruitment, interviews, and analysis of the transcripts continued until no new codes emerged and thematic saturation was achieved.

Participant Characteristics

Overall, we invited 34 patients to be interviewed; 26 consented and completed interviews (76.5%). Six (17.6%) patients declined participation, one (2.9%) was unable to complete the interview before hospital discharge, and one (2.9%) was excluded due to disorientation. Demographic characteristics of the 26 participants are shown in Table 1.

Four main themes emerged from our analysis. Table 2 summarizes these themes, subthemes, and provides representative quotes.

Major Medical Problem(s) are Universal, but High Hospital Use Varies in Onset

Not surprisingly, all participants described having at least one major medical problem. Some participants, such as those with genetic disorders, had experienced periods of high hospital use throughout their entire lifetime, while other participants experienced an onset of high hospital use as an adult after being previously healthy. Though most participants with genetic disorders had sickle cell anemia; one had a rare genetic disorder which caused chronic gastrointestinal symptoms. Participants typically described having a significant medical condition as well as other medical problems or complications from past surgery. Some participants described having a major medical problem which did not require frequent hospitalization until a complication or other medical problem arose, suggesting these new issues pushed them over a threshold beyond which self-management at home was less successful.

Course Fluctuates over Time and is Related to Psychological, Social, and Economic Factors

Participants identified psychological stress, social support, and financial constraints as factors which influence the course of their illness over time. Deaths in the family, breakups, and concerns about other family members were mentioned as specific forms of psychological stress and directly linked by participants to worsening of symptoms. Social support was present for most, but not all, participants, with no appreciable difference based on whether the participant had sickle cell disease. Social support was generally perceived as helpful, and several participants indicated a benefit to their own health when providing social support to others. Financial pressures also served as stressors and often impeded care due to lack of access to medications, other treatments, and housing.

Onset and Progression of Episodes Vary, but Generally Seem Uncontrollable

Regarding the onset of illness episodes, some participants described the sudden, unpredictable onset of symptoms, others described a more gradual onset which allowed them to attempt self-management. Regardless of the timing, episodes of illness were often perceived as spontaneous or triggered by factors outside of the participant’s control. Several participants, especially those with sickle cell disease, mentioned a relationship between their symptoms and the weather. Participants also noted the inconsistency in factors which may trigger an episode (ie, sometimes the factor exacerbated symptoms, while other times it did not). Participants also described having a high symptom burden with significant limitations in activities of daily living during episodes of illness. Pain was a very common component of symptoms regardless of whether or not the participant had sickle cell disease.

Individuals Seek Care after Self-Management Fails and Prefer to Avoid Hospitalization

Participants tried to control their symptoms with medications and typically sought care only when it was clear that this approach was not working, or they ran out of medications. This finding was consistent across both groups of participants (ie, those with and those without sickle cell disease). Many participants described very strong preferences not to come to the hospital; no participant described being in the hospital as a favorable or positive experience. Some participants mentioned that they had spent major holidays in the hospital and that they missed their family. No participant had a desire to come to the hospital.

In this study of frequently hospitalized patients, we found four major themes that illuminate patient perspectives about factors that contribute to high hospital use. While some of our findings corroborate those of previous studies, other emerging patterns were novel. Herein, we summarize key findings, provide context, and describe implications for the design of models of care for frequently hospitalized patients.

Similar to the findings of previous quantitative research, participants in our study described having a significant medical condition and typically had multiple medical conditions or complications.^4-6 Importantly, some participants described having a major medical problem which did not require frequent hospitalization until another medical problem or complication arose. This finding suggests that there may be an opportunity to identify patients with significant medical problems who are at elevated risk before the onset of high hospital use. Early identification of these high-risk patients could allow for the provision of additional support to prevent potential complications or address other factors which may contribute to the need for frequent hospitalization.

Participants in our study directly linked psychological stress to fluctuations in their course of illness. Previous research by Mautner and colleagues queried participants about childhood experiences and early life stressors and reported that early life instabilities and traumas were prevalent among patients with high levels of emergency and hospital-based healthcare utilization.¹⁵ Our participants identified more recent traumatic events (eg, the death of a loved one and breakups) when reflecting on factors contributing to illness exacerbations; early life trauma did not emerge as an identified contributor. Of note, unlike Mautner et al., we did not ask participants to reflect on childhood determinants of disease and illness specifically. Our findings suggest that psychological stress contributes to illness exacerbation, even for those patients without other significant psychiatric conditions (eg, depressive disorder, schizophrenia). Incorporating mental health professionals into programs for this patient population may improve health by teaching specific coping strategies, including cognitive-behavioral therapy for an acute stress disorder.^16,17

Social support was also a factor related to illness fluctuations over time. Notably, several participants indicated a benefit to their own health when providing social support to others, suggesting a role for peer support that may be reciprocally beneficial. This approach is supported by the literature. Williams and colleagues found that patients with sickle cell anemia experienced symptom improvement with peer support;¹⁸ while Johnson and colleagues recently reported a reduction in readmissions to acute care with the use of peer support for patients with severe mental illness.¹⁹

Financial constraints impeded care for some patients and served as a barrier to accessing medications, other treatments, and housing. Similar to the findings of prior quantitative research, our frequently hospitalized patients had a high proportion of patients with Medicaid and low proportion with private insurance, suggesting low socioeconomic status.^9,20 We did not formally collect data on income or economic status. Interestingly, prior qualitative studies have not identified financial constraints as a major theme, though this may be explained by differences in study populations and the overall objectives of the studies.^15,21 Importantly, the overwhelming majority of programs for frequently hospitalized patients identified in a recent systematic review included social workers.⁷ Our findings support the need to address financial constraints and the use of social workers in models of care for frequently hospitalized patients.

Many participants in our study felt that the factors contributing to exacerbations of illness were either inconsistent in their effect or out of their control. These findings have similarities to those from a qualitative study by Liu and colleagues in which they interviewed 20 “hospital-dependent” patients over 65 years of age.²¹ Though not explicitly focused on factors contributing to exacerbations, participants in their study felt that hospitalizations were generally inevitable. In our study, participants with sickle cell disease often identified changes in the weather as contributing to illness exacerbations. The relationship between weather and sickle cell disease remains incompletely understood, with an inconsistent association found in prior studies.²²

Participants in our study strongly desired to avoid hospitalization and typically sought hospital care when symptoms could not be controlled at home. This finding is in contrast to that from the study by Liu and colleagues where they found that hospital-dependent patients over 65 years had favorable perspectives of hospitalization because they felt safer and more secure in the hospital.²¹ Our participants were younger than those from the study by Liu and colleagues, had a high symptom burden, and may have been more concerned about control of those symptoms than the risk for clinical deterioration. Programs should aim to strengthen their support of patients’ self-management efforts early in the episode of illness and potentially offer home visits or a day hospital to avoid hospitalization. A recent systematic review found evidence that alternatives to inpatient care (eg, hospital-at-home) for low risk medical patients can achieve comparable outcomes at lower costs.²³ Similarly, some health systems have implemented day hospitals to treat low risk patients with uncomplicated sickle cell pain.^24,25

The heavy symptom burden experienced by participants in our study is notable. Pain was especially common. Programs may wish to partner with palliative care and addiction specialists to balance symptom relief with the simultaneous need to address comorbid substance and opioid use disorders when they are present.^4,9

Our study has several limitations. First, participants were recruited from the medicine service at a single academic hospital using criteria we developed to identify frequently hospitalized patients. Populations differ across hospitals and definitions of frequently hospitalized patients vary, limiting the generalizability of our findings. Second, we excluded patients whose preferred language was not English, as well as those disoriented to person, place, or time. It is possible that factors contributing to high hospital use differ for non-English speaking patients and those with cognitive deficits.

In this qualitative study, we identified factors associated with the onset and continuation of high hospital use. Emergent themes pointed to factors which influence patients’ onset of high hospital use, fluctuations in their illness over time, and triggers to seek care during an illness episode. These findings represent an important contribution to the literature because they allow patients’ perspectives to be incorporated into the design and adaptation of programs serving similar populations in other health systems. Programs that integrate patients’ perspectives into their design are likely to be better prepared to address patients’ needs and improve patient outcomes.

Acknowledgments

The authors thank the participants for their time and willingness to share their stories. The authors also thank Claire A. Knoten PhD and Erin Lambers PhD, former research team members who helped in the initial stages of the study.

Disclosures

The authors have nothing to disclose.

Funding

This project was funded by Northwestern Memorial Hospital and the Northwestern Medical Group.

In recent years, hospitals have made considerable efforts to improve transitions of care, in part due to financial incentives from the Medicare Hospital Readmission Reduction Program (HRRP).¹ Initially focusing on three medical conditions, the HRRP has been associated with significant reductions in readmission rates.² Importantly, a small proportion of patients accounts for a very large proportion of hospital readmissions and hospital use.^3,4 Frequently hospitalized patients often have multiple chronic conditions and unique needs which may not be met by conventional approaches to healthcare delivery, including those influenced by the HRRP.^4-6 In light of this challenge, some hospitals have developed programs specifically focused on frequently hospitalized patients. A recent systematic review of these programs found relatively few studies of high quality, providing only limited insight in designing interventions to support this population.⁷ Moreover, no studies appear to have incorporated the patients’ perspectives into the design or adaptation of the model. Members of our research team developed and implemented the Complex High Admission Management Program (CHAMP) in January 2016 to address the needs of frequently hospitalized patients in our hospital. To enhance CHAMP and inform the design of programs serving similar populations in other health systems, we sought to identify factors associated with the onset and continuation of high hospital use. Our research question was, from the patients’ perspective, what factors contribute to patients’ becoming and continuing to be high users of hospital care.

Setting, Study Design, and Participants

This qualitative study took place at Northwestern Memorial Hospital (NMH), an 894-bed urban academic hospital located in Chicago, Illinois. Between December 2016 and September 2017, we recruited adult patients admitted to the general medicine services. Eligible participants were identified with the assistance of a daily Northwestern Medicine Electronic Data Warehouse (EDW) search and included patients with two unplanned 30-day inpatient readmissions to NMH within the prior 12 months, in addition to one or more of the following criteria: (1) at least one readmission in the last six months; (2) a referral from one of the patient’s medical providers; or (3) at least three observation visits. We excluded patients whose preferred language was not English and those disoriented to person, place, or time. Considering NMH data showing that approximately one-third of high-utilizer patients have sickle cell disease, we used purposive sampling with the goal to compare findings within and between two groups of participants; those with and those without sickle cell disease. Our study was deemed exempt by the Northwestern University Institutional Review Board.

Participant Enrollment and Data Collection

We created an interview guide based on the research team’s experience with this population, a literature review, and our research question (See Appendix).^8,9 A research coordinator approached eligible participants during their hospital stay. The coordinator explained the study to eligible participants and obtained verbal consent for participation. The research coordinator then conducted one-on-one semi-structured interviews. Interviews were audio recorded for subsequent transcription and coding. Each interview lasted approximately 45 minutes. Participants were compensated with a $20 gift card for their time.

Analysis

Digital audio recordings from interviews were transcribed verbatim, deidentified, and analyzed using an iterative inductive team-based approach to coding.¹⁰ In our first cycle coding, all coders (KJO, SF, MMC, LO, KAC) independently reviewed and coded three transcripts using descriptive coding and subcoding to generate a preliminary codebook with code definitions.^10,11 Following the meetings to compare and compile our initial coding, each researcher then independently recoded the three transcripts with the developed codebook. The researchers met again to triangulate perspectives and reach a consensus on the final codebook. Using multiple coders is a standard process to control for subjective bias that one coder could bring to the coding process.¹² Following this meeting, the coders split into two teams of two (KJO, SF, and MMC, LO) to complete the coding of the remaining transcripts. Each team member independently coded the assigned transcripts and reconciled their codes with their counterpart; any discrepancies were resolved through discussion. Using this strategy, every transcript was coded by at least two team members. Our second coding cycle utilized pattern coding and involved identifying consistency both within and between transcripts; discovering associations between codes.^10,11,13 Constant comparison was used to compare responses among all participants, as well as between sickle-cell and nonsickle-cell participants.^13,14 Following team coding and reconciling, the analyses were presented to a broader research team for additional feedback and critique. All analyses were conducted using Dedoose version 8.0.35 (Los Angeles, California). Participant recruitment, interviews, and analysis of the transcripts continued until no new codes emerged and thematic saturation was achieved.

Participant Characteristics

Overall, we invited 34 patients to be interviewed; 26 consented and completed interviews (76.5%). Six (17.6%) patients declined participation, one (2.9%) was unable to complete the interview before hospital discharge, and one (2.9%) was excluded due to disorientation. Demographic characteristics of the 26 participants are shown in Table 1.

Four main themes emerged from our analysis. Table 2 summarizes these themes, subthemes, and provides representative quotes.

Major Medical Problem(s) are Universal, but High Hospital Use Varies in Onset

Not surprisingly, all participants described having at least one major medical problem. Some participants, such as those with genetic disorders, had experienced periods of high hospital use throughout their entire lifetime, while other participants experienced an onset of high hospital use as an adult after being previously healthy. Though most participants with genetic disorders had sickle cell anemia; one had a rare genetic disorder which caused chronic gastrointestinal symptoms. Participants typically described having a significant medical condition as well as other medical problems or complications from past surgery. Some participants described having a major medical problem which did not require frequent hospitalization until a complication or other medical problem arose, suggesting these new issues pushed them over a threshold beyond which self-management at home was less successful.

Course Fluctuates over Time and is Related to Psychological, Social, and Economic Factors

Participants identified psychological stress, social support, and financial constraints as factors which influence the course of their illness over time. Deaths in the family, breakups, and concerns about other family members were mentioned as specific forms of psychological stress and directly linked by participants to worsening of symptoms. Social support was present for most, but not all, participants, with no appreciable difference based on whether the participant had sickle cell disease. Social support was generally perceived as helpful, and several participants indicated a benefit to their own health when providing social support to others. Financial pressures also served as stressors and often impeded care due to lack of access to medications, other treatments, and housing.

Onset and Progression of Episodes Vary, but Generally Seem Uncontrollable

Regarding the onset of illness episodes, some participants described the sudden, unpredictable onset of symptoms, others described a more gradual onset which allowed them to attempt self-management. Regardless of the timing, episodes of illness were often perceived as spontaneous or triggered by factors outside of the participant’s control. Several participants, especially those with sickle cell disease, mentioned a relationship between their symptoms and the weather. Participants also noted the inconsistency in factors which may trigger an episode (ie, sometimes the factor exacerbated symptoms, while other times it did not). Participants also described having a high symptom burden with significant limitations in activities of daily living during episodes of illness. Pain was a very common component of symptoms regardless of whether or not the participant had sickle cell disease.

Individuals Seek Care after Self-Management Fails and Prefer to Avoid Hospitalization

Participants tried to control their symptoms with medications and typically sought care only when it was clear that this approach was not working, or they ran out of medications. This finding was consistent across both groups of participants (ie, those with and those without sickle cell disease). Many participants described very strong preferences not to come to the hospital; no participant described being in the hospital as a favorable or positive experience. Some participants mentioned that they had spent major holidays in the hospital and that they missed their family. No participant had a desire to come to the hospital.

In this study of frequently hospitalized patients, we found four major themes that illuminate patient perspectives about factors that contribute to high hospital use. While some of our findings corroborate those of previous studies, other emerging patterns were novel. Herein, we summarize key findings, provide context, and describe implications for the design of models of care for frequently hospitalized patients.

Similar to the findings of previous quantitative research, participants in our study described having a significant medical condition and typically had multiple medical conditions or complications.^4-6 Importantly, some participants described having a major medical problem which did not require frequent hospitalization until another medical problem or complication arose. This finding suggests that there may be an opportunity to identify patients with significant medical problems who are at elevated risk before the onset of high hospital use. Early identification of these high-risk patients could allow for the provision of additional support to prevent potential complications or address other factors which may contribute to the need for frequent hospitalization.

Participants in our study directly linked psychological stress to fluctuations in their course of illness. Previous research by Mautner and colleagues queried participants about childhood experiences and early life stressors and reported that early life instabilities and traumas were prevalent among patients with high levels of emergency and hospital-based healthcare utilization.¹⁵ Our participants identified more recent traumatic events (eg, the death of a loved one and breakups) when reflecting on factors contributing to illness exacerbations; early life trauma did not emerge as an identified contributor. Of note, unlike Mautner et al., we did not ask participants to reflect on childhood determinants of disease and illness specifically. Our findings suggest that psychological stress contributes to illness exacerbation, even for those patients without other significant psychiatric conditions (eg, depressive disorder, schizophrenia). Incorporating mental health professionals into programs for this patient population may improve health by teaching specific coping strategies, including cognitive-behavioral therapy for an acute stress disorder.^16,17

Social support was also a factor related to illness fluctuations over time. Notably, several participants indicated a benefit to their own health when providing social support to others, suggesting a role for peer support that may be reciprocally beneficial. This approach is supported by the literature. Williams and colleagues found that patients with sickle cell anemia experienced symptom improvement with peer support;¹⁸ while Johnson and colleagues recently reported a reduction in readmissions to acute care with the use of peer support for patients with severe mental illness.¹⁹

Financial constraints impeded care for some patients and served as a barrier to accessing medications, other treatments, and housing. Similar to the findings of prior quantitative research, our frequently hospitalized patients had a high proportion of patients with Medicaid and low proportion with private insurance, suggesting low socioeconomic status.^9,20 We did not formally collect data on income or economic status. Interestingly, prior qualitative studies have not identified financial constraints as a major theme, though this may be explained by differences in study populations and the overall objectives of the studies.^15,21 Importantly, the overwhelming majority of programs for frequently hospitalized patients identified in a recent systematic review included social workers.⁷ Our findings support the need to address financial constraints and the use of social workers in models of care for frequently hospitalized patients.

Many participants in our study felt that the factors contributing to exacerbations of illness were either inconsistent in their effect or out of their control. These findings have similarities to those from a qualitative study by Liu and colleagues in which they interviewed 20 “hospital-dependent” patients over 65 years of age.²¹ Though not explicitly focused on factors contributing to exacerbations, participants in their study felt that hospitalizations were generally inevitable. In our study, participants with sickle cell disease often identified changes in the weather as contributing to illness exacerbations. The relationship between weather and sickle cell disease remains incompletely understood, with an inconsistent association found in prior studies.²²

Participants in our study strongly desired to avoid hospitalization and typically sought hospital care when symptoms could not be controlled at home. This finding is in contrast to that from the study by Liu and colleagues where they found that hospital-dependent patients over 65 years had favorable perspectives of hospitalization because they felt safer and more secure in the hospital.²¹ Our participants were younger than those from the study by Liu and colleagues, had a high symptom burden, and may have been more concerned about control of those symptoms than the risk for clinical deterioration. Programs should aim to strengthen their support of patients’ self-management efforts early in the episode of illness and potentially offer home visits or a day hospital to avoid hospitalization. A recent systematic review found evidence that alternatives to inpatient care (eg, hospital-at-home) for low risk medical patients can achieve comparable outcomes at lower costs.²³ Similarly, some health systems have implemented day hospitals to treat low risk patients with uncomplicated sickle cell pain.^24,25

The heavy symptom burden experienced by participants in our study is notable. Pain was especially common. Programs may wish to partner with palliative care and addiction specialists to balance symptom relief with the simultaneous need to address comorbid substance and opioid use disorders when they are present.^4,9

Our study has several limitations. First, participants were recruited from the medicine service at a single academic hospital using criteria we developed to identify frequently hospitalized patients. Populations differ across hospitals and definitions of frequently hospitalized patients vary, limiting the generalizability of our findings. Second, we excluded patients whose preferred language was not English, as well as those disoriented to person, place, or time. It is possible that factors contributing to high hospital use differ for non-English speaking patients and those with cognitive deficits.

In this qualitative study, we identified factors associated with the onset and continuation of high hospital use. Emergent themes pointed to factors which influence patients’ onset of high hospital use, fluctuations in their illness over time, and triggers to seek care during an illness episode. These findings represent an important contribution to the literature because they allow patients’ perspectives to be incorporated into the design and adaptation of programs serving similar populations in other health systems. Programs that integrate patients’ perspectives into their design are likely to be better prepared to address patients’ needs and improve patient outcomes.

Acknowledgments

The authors thank the participants for their time and willingness to share their stories. The authors also thank Claire A. Knoten PhD and Erin Lambers PhD, former research team members who helped in the initial stages of the study.

Disclosures

The authors have nothing to disclose.

Funding

This project was funded by Northwestern Memorial Hospital and the Northwestern Medical Group.

In recent years, hospitals have made considerable efforts to improve transitions of care, in part due to financial incentives from the Medicare Hospital Readmission Reduction Program (HRRP).¹ Initially focusing on three medical conditions, the HRRP has been associated with significant reductions in readmission rates.² Importantly, a small proportion of patients accounts for a very large proportion of hospital readmissions and hospital use.^3,4 Frequently hospitalized patients often have multiple chronic conditions and unique needs which may not be met by conventional approaches to healthcare delivery, including those influenced by the HRRP.^4-6 In light of this challenge, some hospitals have developed programs specifically focused on frequently hospitalized patients. A recent systematic review of these programs found relatively few studies of high quality, providing only limited insight in designing interventions to support this population.⁷ Moreover, no studies appear to have incorporated the patients’ perspectives into the design or adaptation of the model. Members of our research team developed and implemented the Complex High Admission Management Program (CHAMP) in January 2016 to address the needs of frequently hospitalized patients in our hospital. To enhance CHAMP and inform the design of programs serving similar populations in other health systems, we sought to identify factors associated with the onset and continuation of high hospital use. Our research question was, from the patients’ perspective, what factors contribute to patients’ becoming and continuing to be high users of hospital care.

Setting, Study Design, and Participants

This qualitative study took place at Northwestern Memorial Hospital (NMH), an 894-bed urban academic hospital located in Chicago, Illinois. Between December 2016 and September 2017, we recruited adult patients admitted to the general medicine services. Eligible participants were identified with the assistance of a daily Northwestern Medicine Electronic Data Warehouse (EDW) search and included patients with two unplanned 30-day inpatient readmissions to NMH within the prior 12 months, in addition to one or more of the following criteria: (1) at least one readmission in the last six months; (2) a referral from one of the patient’s medical providers; or (3) at least three observation visits. We excluded patients whose preferred language was not English and those disoriented to person, place, or time. Considering NMH data showing that approximately one-third of high-utilizer patients have sickle cell disease, we used purposive sampling with the goal to compare findings within and between two groups of participants; those with and those without sickle cell disease. Our study was deemed exempt by the Northwestern University Institutional Review Board.

Participant Enrollment and Data Collection

We created an interview guide based on the research team’s experience with this population, a literature review, and our research question (See Appendix).^8,9 A research coordinator approached eligible participants during their hospital stay. The coordinator explained the study to eligible participants and obtained verbal consent for participation. The research coordinator then conducted one-on-one semi-structured interviews. Interviews were audio recorded for subsequent transcription and coding. Each interview lasted approximately 45 minutes. Participants were compensated with a $20 gift card for their time.

Analysis

Digital audio recordings from interviews were transcribed verbatim, deidentified, and analyzed using an iterative inductive team-based approach to coding.¹⁰ In our first cycle coding, all coders (KJO, SF, MMC, LO, KAC) independently reviewed and coded three transcripts using descriptive coding and subcoding to generate a preliminary codebook with code definitions.^10,11 Following the meetings to compare and compile our initial coding, each researcher then independently recoded the three transcripts with the developed codebook. The researchers met again to triangulate perspectives and reach a consensus on the final codebook. Using multiple coders is a standard process to control for subjective bias that one coder could bring to the coding process.¹² Following this meeting, the coders split into two teams of two (KJO, SF, and MMC, LO) to complete the coding of the remaining transcripts. Each team member independently coded the assigned transcripts and reconciled their codes with their counterpart; any discrepancies were resolved through discussion. Using this strategy, every transcript was coded by at least two team members. Our second coding cycle utilized pattern coding and involved identifying consistency both within and between transcripts; discovering associations between codes.^10,11,13 Constant comparison was used to compare responses among all participants, as well as between sickle-cell and nonsickle-cell participants.^13,14 Following team coding and reconciling, the analyses were presented to a broader research team for additional feedback and critique. All analyses were conducted using Dedoose version 8.0.35 (Los Angeles, California). Participant recruitment, interviews, and analysis of the transcripts continued until no new codes emerged and thematic saturation was achieved.

Participant Characteristics

Overall, we invited 34 patients to be interviewed; 26 consented and completed interviews (76.5%). Six (17.6%) patients declined participation, one (2.9%) was unable to complete the interview before hospital discharge, and one (2.9%) was excluded due to disorientation. Demographic characteristics of the 26 participants are shown in Table 1.

Four main themes emerged from our analysis. Table 2 summarizes these themes, subthemes, and provides representative quotes.

Major Medical Problem(s) are Universal, but High Hospital Use Varies in Onset

Not surprisingly, all participants described having at least one major medical problem. Some participants, such as those with genetic disorders, had experienced periods of high hospital use throughout their entire lifetime, while other participants experienced an onset of high hospital use as an adult after being previously healthy. Though most participants with genetic disorders had sickle cell anemia; one had a rare genetic disorder which caused chronic gastrointestinal symptoms. Participants typically described having a significant medical condition as well as other medical problems or complications from past surgery. Some participants described having a major medical problem which did not require frequent hospitalization until a complication or other medical problem arose, suggesting these new issues pushed them over a threshold beyond which self-management at home was less successful.

Course Fluctuates over Time and is Related to Psychological, Social, and Economic Factors

Participants identified psychological stress, social support, and financial constraints as factors which influence the course of their illness over time. Deaths in the family, breakups, and concerns about other family members were mentioned as specific forms of psychological stress and directly linked by participants to worsening of symptoms. Social support was present for most, but not all, participants, with no appreciable difference based on whether the participant had sickle cell disease. Social support was generally perceived as helpful, and several participants indicated a benefit to their own health when providing social support to others. Financial pressures also served as stressors and often impeded care due to lack of access to medications, other treatments, and housing.

Onset and Progression of Episodes Vary, but Generally Seem Uncontrollable

Regarding the onset of illness episodes, some participants described the sudden, unpredictable onset of symptoms, others described a more gradual onset which allowed them to attempt self-management. Regardless of the timing, episodes of illness were often perceived as spontaneous or triggered by factors outside of the participant’s control. Several participants, especially those with sickle cell disease, mentioned a relationship between their symptoms and the weather. Participants also noted the inconsistency in factors which may trigger an episode (ie, sometimes the factor exacerbated symptoms, while other times it did not). Participants also described having a high symptom burden with significant limitations in activities of daily living during episodes of illness. Pain was a very common component of symptoms regardless of whether or not the participant had sickle cell disease.

Individuals Seek Care after Self-Management Fails and Prefer to Avoid Hospitalization

Participants tried to control their symptoms with medications and typically sought care only when it was clear that this approach was not working, or they ran out of medications. This finding was consistent across both groups of participants (ie, those with and those without sickle cell disease). Many participants described very strong preferences not to come to the hospital; no participant described being in the hospital as a favorable or positive experience. Some participants mentioned that they had spent major holidays in the hospital and that they missed their family. No participant had a desire to come to the hospital.

In this study of frequently hospitalized patients, we found four major themes that illuminate patient perspectives about factors that contribute to high hospital use. While some of our findings corroborate those of previous studies, other emerging patterns were novel. Herein, we summarize key findings, provide context, and describe implications for the design of models of care for frequently hospitalized patients.

Similar to the findings of previous quantitative research, participants in our study described having a significant medical condition and typically had multiple medical conditions or complications.^4-6 Importantly, some participants described having a major medical problem which did not require frequent hospitalization until another medical problem or complication arose. This finding suggests that there may be an opportunity to identify patients with significant medical problems who are at elevated risk before the onset of high hospital use. Early identification of these high-risk patients could allow for the provision of additional support to prevent potential complications or address other factors which may contribute to the need for frequent hospitalization.

Participants in our study directly linked psychological stress to fluctuations in their course of illness. Previous research by Mautner and colleagues queried participants about childhood experiences and early life stressors and reported that early life instabilities and traumas were prevalent among patients with high levels of emergency and hospital-based healthcare utilization.¹⁵ Our participants identified more recent traumatic events (eg, the death of a loved one and breakups) when reflecting on factors contributing to illness exacerbations; early life trauma did not emerge as an identified contributor. Of note, unlike Mautner et al., we did not ask participants to reflect on childhood determinants of disease and illness specifically. Our findings suggest that psychological stress contributes to illness exacerbation, even for those patients without other significant psychiatric conditions (eg, depressive disorder, schizophrenia). Incorporating mental health professionals into programs for this patient population may improve health by teaching specific coping strategies, including cognitive-behavioral therapy for an acute stress disorder.^16,17

Social support was also a factor related to illness fluctuations over time. Notably, several participants indicated a benefit to their own health when providing social support to others, suggesting a role for peer support that may be reciprocally beneficial. This approach is supported by the literature. Williams and colleagues found that patients with sickle cell anemia experienced symptom improvement with peer support;¹⁸ while Johnson and colleagues recently reported a reduction in readmissions to acute care with the use of peer support for patients with severe mental illness.¹⁹

Financial constraints impeded care for some patients and served as a barrier to accessing medications, other treatments, and housing. Similar to the findings of prior quantitative research, our frequently hospitalized patients had a high proportion of patients with Medicaid and low proportion with private insurance, suggesting low socioeconomic status.^9,20 We did not formally collect data on income or economic status. Interestingly, prior qualitative studies have not identified financial constraints as a major theme, though this may be explained by differences in study populations and the overall objectives of the studies.^15,21 Importantly, the overwhelming majority of programs for frequently hospitalized patients identified in a recent systematic review included social workers.⁷ Our findings support the need to address financial constraints and the use of social workers in models of care for frequently hospitalized patients.

Many participants in our study felt that the factors contributing to exacerbations of illness were either inconsistent in their effect or out of their control. These findings have similarities to those from a qualitative study by Liu and colleagues in which they interviewed 20 “hospital-dependent” patients over 65 years of age.²¹ Though not explicitly focused on factors contributing to exacerbations, participants in their study felt that hospitalizations were generally inevitable. In our study, participants with sickle cell disease often identified changes in the weather as contributing to illness exacerbations. The relationship between weather and sickle cell disease remains incompletely understood, with an inconsistent association found in prior studies.²²

Participants in our study strongly desired to avoid hospitalization and typically sought hospital care when symptoms could not be controlled at home. This finding is in contrast to that from the study by Liu and colleagues where they found that hospital-dependent patients over 65 years had favorable perspectives of hospitalization because they felt safer and more secure in the hospital.²¹ Our participants were younger than those from the study by Liu and colleagues, had a high symptom burden, and may have been more concerned about control of those symptoms than the risk for clinical deterioration. Programs should aim to strengthen their support of patients’ self-management efforts early in the episode of illness and potentially offer home visits or a day hospital to avoid hospitalization. A recent systematic review found evidence that alternatives to inpatient care (eg, hospital-at-home) for low risk medical patients can achieve comparable outcomes at lower costs.²³ Similarly, some health systems have implemented day hospitals to treat low risk patients with uncomplicated sickle cell pain.^24,25

The heavy symptom burden experienced by participants in our study is notable. Pain was especially common. Programs may wish to partner with palliative care and addiction specialists to balance symptom relief with the simultaneous need to address comorbid substance and opioid use disorders when they are present.^4,9

Our study has several limitations. First, participants were recruited from the medicine service at a single academic hospital using criteria we developed to identify frequently hospitalized patients. Populations differ across hospitals and definitions of frequently hospitalized patients vary, limiting the generalizability of our findings. Second, we excluded patients whose preferred language was not English, as well as those disoriented to person, place, or time. It is possible that factors contributing to high hospital use differ for non-English speaking patients and those with cognitive deficits.

In this qualitative study, we identified factors associated with the onset and continuation of high hospital use. Emergent themes pointed to factors which influence patients’ onset of high hospital use, fluctuations in their illness over time, and triggers to seek care during an illness episode. These findings represent an important contribution to the literature because they allow patients’ perspectives to be incorporated into the design and adaptation of programs serving similar populations in other health systems. Programs that integrate patients’ perspectives into their design are likely to be better prepared to address patients’ needs and improve patient outcomes.

Acknowledgments

The authors thank the participants for their time and willingness to share their stories. The authors also thank Claire A. Knoten PhD and Erin Lambers PhD, former research team members who helped in the initial stages of the study.

Disclosures

The authors have nothing to disclose.

Funding

This project was funded by Northwestern Memorial Hospital and the Northwestern Medical Group.

There is an increasing utilization of advanced practice providers (APPs) in the delivery of healthcare in the United States.^1,2 As of 2016, there were 157, 025 nurse practitioners (NPs) and 102,084 physician assistants (PAs) with a projected growth rate of 6.8% and 4.3%, respectively, which exceeds the physician growth rate of 1.1%.² This increased growth rate has been attributed to the expectation that APPs can enhance the quality of physician care, relieve physician shortages, and reduce service costs, as APPs are less expensive to hire than physicians.^3,4 Hospital medicine is the fastest growing medical field in the United States, and approximately 83% of hospitalist groups around the country utilize APPs; however, the demand for hospitalists continues to exceed the supply, and this has led to increased utilization of APPs in hospital medicine.^5-10

APPs receive very limited inpatient training and there is wide variation in their clinical abilities after graduation.¹¹ This is an issue that has become exacerbated in recent years by a change in the training process for PAs. Before 2005, PA programs were typically two to three years long and required the same prerequisite courses as medical schools.¹¹ PA students completed more than 2,000 hours of clinical rotations and then had to pass the Physician Assistant National Certifying Exam before they could practice.¹² Traditionally, PA programs typically attracted students with prior healthcare experience.¹¹ In 2005, PA programs began transitioning from bachelor’s degrees to requiring a master’s level degree for completion of the programs. This has shifted the demographics of the students matriculating to younger students with little-to-no prior healthcare experience; moreover, these fresh graduates lack exposure to hospital medicine.¹¹

NPs usually gain clinical experience working as registered nurses (RNs) for two or more years prior to entry into the NP program. NP programs for baccalaureate-prepared RNs vary in length from two to three years.² There is an acute care focus for NPs in training; however, there is no standardized training or licensure to ensure that hospital medicine competencies are met.^13-15 Some studies have shown that a lack of structured support has been found to affect NP role transition negatively during the first year of practice,¹⁶ and graduating NPs have indicated that they needed more out of their clinical education in terms of content, clinical experience, and competency testing.¹⁷

Hiring new APP graduates as hospitalists requires a longer and more rigorous onboarding process. On‐the‐job training in hospital medicine for new APP graduates can take as long as six to 12 months in order for them to acquire the basic skill set necessary to adequately manage hospitalized patients.¹⁵ This extended onboarding is costly because the APPs are receiving a full hospitalist salary, yet they are not functioning at full capacity. Ideally, there should be an intermediary training step between graduation and employment as hospitalist APPs. Studies have shown that APPs are interested in formal postgraduate hospital medicine training, even if it means having a lower stipend during the first year after graduating from their NP or PA program.^9,15,18

The growing need for hospitalists, driven by residency work-hour reform, increased age and complexity of patients, and the need to improve the quality of inpatient care while simultaneously reducing waste, has contributed to the increasing utilization of and need for highly qualified APPs in hospital medicine.^11,19,20 We established a fellowship to train APPs. The goal of this study was to determine if an APP fellowship is a cost-effective pipeline for filling vacancies within a hospitalist program.

Design and Setting

Johns Hopkins Bayview Medical Center (JHBMC) is a 440 bed hospital in Baltimore Maryland. The hospitalist group was started in 1996 with one physician seeing approximately 500 discharges a year. Over the last 20 years, the group has grown and is now its own division with 57 providers, including 42 physicians, 11 APPs, and four APP fellows. The hospitalist division manages ~7,000 discharges a year, which corresponds to approximately 60% of admissions to general medicine. Hospitalist APPs help staff general medicine by working alongside doctors and admitting patients during the day and night. The APPs also staff the pulmonary step down unit with a pulmonary attending and the chemical dependency unit with an internal medicine addiction specialist.

The growth of the division of hospital medicine at JHBMC is a result of increasing volumes and reduced residency duty hours. The increasing full time equivalents (FTEs) resulted in a need for APPs; however, vacancies went unfilled for an average of 35 weeks due to the time it took to post open positions, interview applicants, and hire applicants through the credentialing process. Further, it took as long as 22 to 34 weeks for a new hire to work independently. The APP vacancies and onboarding resulted in increased costs to the division incurred by physician moonlighting to cover open shifts. The hourly physician moonlighting rate at JHBMC is $150. All costs were calculated on the basis of a 40-hour work week. We performed a pre- and postanalysis of outcomes of interest between January 2009 and June 2018. This study was exempt from institutional review board review.

Intervention

In 2014, a one year APP clinical fellowship in hospital medicine was started. The fellows evaluate and manage patients working one-on-one with an experienced hospitalist faculty member. The program consists of 80% clinical experience in the inpatient setting and 20% didactic instruction (Table 1). Up to four fellows are accepted each year and are eligible for hire after training if vacancies exist. The program is cost neutral and was financed by downsizing, through attrition, two physician FTEs. Four APP fellows’ salaries are the equivalent of two entry-level hospitalist physicians’ salaries at JHBMC. The annual salary for an APP fellow is $69,000.

Downsizing by two physician FTEs meant that one less doctor was scheduled every day. The patient load previously seen by that one doctor (10 patients) was absorbed by the MD–APP fellow dyads. Paired with a fellow, each physician sees a higher cap of 13 patients, and it takes six weeks for the fellows to ramp-up to this patient load. When the fellow first starts, the team sees 10 patients. Every two weeks, the pair’s census increases by one patient to the cap of 13. Collectively, the four APP fellow–MD dyads make it possible for four physicians to see an additional 12 patients. The two extra patients absorbed by the service per day results in a net increase in capacity of up to 730 patient encounters a year.

Outcomes and Analysis

Our main outcomes of interest were duration of onboarding and cost incurred by the division to (1) staff the service during a vacancy and (2) onboard new hires. Secondary outcomes included duration of vacancy and total time spent with the group. We collected basic demographic data on participants, including, age, gender, and race. Demographics and outcomes of interest were compared pre- (2009-2013) and post- (2014-2018) initiation of the APP clinical fellowship using the chi-square test, the t-test for normally distributed data, and the Wilcoxon rank-sum for nonnormally distributed data, as appropriate. The normality of the data distribution was tested using the Shapiro-Wilk W test. Two-tailed P values less than .05 were considered to be statistically significant. Results were analyzed using Stata/MP version 13.0 (StataCorp Inc, College Station, Texas).

Twelve fellows have been recruited, and of these, 10 have graduated. Two chose to leave the program prior to completion. Of the 10 fellows that have graduated, six have been hired into our group, one was hired within our facility, and three were hired as hospitalists at other institutions. The median time from APP school graduation to hire was also not different between the two groups (10.5 vs 3.9 months, P = .069). In addition, the total time that the new APP hires spent with the group was nonstatistically significantly different between the two periods (17.9 vs 18.3 months, P = .735). Both the mean duration of onboarding and the cost to the division were significantly reduced after implementation of the program (25.4 vs 11.0 weeks, P = .017 and $361,714 vs $66,000, P = .004; Table 2).

The yearly cost of an APP vacancy and onboarding is incurred by doctor moonlighting costs (at the rate of $150 per hour) to cover open shifts. The mean duration of vacancies and onboarding each year was 34.9 and 25.4 weeks, respectively, before the fellowship. The yearly cost of onboarding, after the establishment of the fellowship, is a maximum of $66,000, derived from physician moonlighting to cover the six-week ramp-up at the very beginning of the fellowship and the five weeks of orientation to the pulmonary and chemical dependency units after the fellowship (Table 3).

Our APP clinical fellowship in hospital medicine at JHBMC has produced several benefits. First, the fellowship has become a pipeline for filling APP vacancies within our division. We have been able to hire for four consecutive years from the fellowship. Second, the ready availability of high-functioning and efficient APP hospitalists has cut down on the onboarding time for our new APP hires. Many new APP graduates lack confidence in caring for complex hospitalized patients. Following our 12-month clinical fellowship, our matriculated fellows are able to practice at the top of their license immediately and confidently. Third, the reduced vacancy and shortened onboarding periods have reduced costs to the division. Fourth, the fellowship has created additional teaching avenues for the faculty. The medicine units at JHBMC are comprised of hospitalist and internal medicine residency services. The hospitalists spend the majority of their clinical time in direct patient care; however, they rotate on the residency service for two weeks out of the year. The majority of physicians welcome the chance to teach more, and partnering with an APP fellow provides that opportunity.

As we have developed and grown this program, the one great challenge has been what to do with graduating fellows when we cannot hire them. Fortunately, the market for highly qualified, well trained APPs is strong, and every one of the fellows that we could not hire within our group has been able to find a position either within our facility or outside our institution. To facilitate this process, program directors and recruiters are invited to meet with the fellows toward the end of their fellowship to share employment opportunities with them.

Our study has limitations. First, had the $276,000 from the attrition of two physicians been used to hire nonfellow APPs under the old model, then the costs of the two models would have been similar, but this was simply not possible because the positions could not be filled. Second, this is a single-site experience, and our findings may not be generalizable, particularly those pertaining to remuneration. Third, our study was underpowered to detect small but important differences in characteristics of APPs, especially time from graduation to hire, before and after the implementation of our fellowship. Further research comparing various programs both in structure and outcomes—such as fellows’ readiness for practice, costs, duration of vacancies, and provider satisfaction—are an important next step.

We have developed a pool of applicants within our division to fill vacancies left by turnover from senior NPs and PAs. This program has reduced costs and improved the joy of practice for both doctors and APPs. As the need for highly qualified NPs and PAs in hospital medicine continues to grow, we may see more APP fellowships in hospital medicine in the United States.

Acknowledgments

The authors thank the advanced practice providers who have helped us grow and refine our fellowship.

Disclosures

The authors have nothing to disclose

There is an increasing utilization of advanced practice providers (APPs) in the delivery of healthcare in the United States.^1,2 As of 2016, there were 157, 025 nurse practitioners (NPs) and 102,084 physician assistants (PAs) with a projected growth rate of 6.8% and 4.3%, respectively, which exceeds the physician growth rate of 1.1%.² This increased growth rate has been attributed to the expectation that APPs can enhance the quality of physician care, relieve physician shortages, and reduce service costs, as APPs are less expensive to hire than physicians.^3,4 Hospital medicine is the fastest growing medical field in the United States, and approximately 83% of hospitalist groups around the country utilize APPs; however, the demand for hospitalists continues to exceed the supply, and this has led to increased utilization of APPs in hospital medicine.^5-10

APPs receive very limited inpatient training and there is wide variation in their clinical abilities after graduation.¹¹ This is an issue that has become exacerbated in recent years by a change in the training process for PAs. Before 2005, PA programs were typically two to three years long and required the same prerequisite courses as medical schools.¹¹ PA students completed more than 2,000 hours of clinical rotations and then had to pass the Physician Assistant National Certifying Exam before they could practice.¹² Traditionally, PA programs typically attracted students with prior healthcare experience.¹¹ In 2005, PA programs began transitioning from bachelor’s degrees to requiring a master’s level degree for completion of the programs. This has shifted the demographics of the students matriculating to younger students with little-to-no prior healthcare experience; moreover, these fresh graduates lack exposure to hospital medicine.¹¹

NPs usually gain clinical experience working as registered nurses (RNs) for two or more years prior to entry into the NP program. NP programs for baccalaureate-prepared RNs vary in length from two to three years.² There is an acute care focus for NPs in training; however, there is no standardized training or licensure to ensure that hospital medicine competencies are met.^13-15 Some studies have shown that a lack of structured support has been found to affect NP role transition negatively during the first year of practice,¹⁶ and graduating NPs have indicated that they needed more out of their clinical education in terms of content, clinical experience, and competency testing.¹⁷

Hiring new APP graduates as hospitalists requires a longer and more rigorous onboarding process. On‐the‐job training in hospital medicine for new APP graduates can take as long as six to 12 months in order for them to acquire the basic skill set necessary to adequately manage hospitalized patients.¹⁵ This extended onboarding is costly because the APPs are receiving a full hospitalist salary, yet they are not functioning at full capacity. Ideally, there should be an intermediary training step between graduation and employment as hospitalist APPs. Studies have shown that APPs are interested in formal postgraduate hospital medicine training, even if it means having a lower stipend during the first year after graduating from their NP or PA program.^9,15,18

The growing need for hospitalists, driven by residency work-hour reform, increased age and complexity of patients, and the need to improve the quality of inpatient care while simultaneously reducing waste, has contributed to the increasing utilization of and need for highly qualified APPs in hospital medicine.^11,19,20 We established a fellowship to train APPs. The goal of this study was to determine if an APP fellowship is a cost-effective pipeline for filling vacancies within a hospitalist program.

Design and Setting

Johns Hopkins Bayview Medical Center (JHBMC) is a 440 bed hospital in Baltimore Maryland. The hospitalist group was started in 1996 with one physician seeing approximately 500 discharges a year. Over the last 20 years, the group has grown and is now its own division with 57 providers, including 42 physicians, 11 APPs, and four APP fellows. The hospitalist division manages ~7,000 discharges a year, which corresponds to approximately 60% of admissions to general medicine. Hospitalist APPs help staff general medicine by working alongside doctors and admitting patients during the day and night. The APPs also staff the pulmonary step down unit with a pulmonary attending and the chemical dependency unit with an internal medicine addiction specialist.

The growth of the division of hospital medicine at JHBMC is a result of increasing volumes and reduced residency duty hours. The increasing full time equivalents (FTEs) resulted in a need for APPs; however, vacancies went unfilled for an average of 35 weeks due to the time it took to post open positions, interview applicants, and hire applicants through the credentialing process. Further, it took as long as 22 to 34 weeks for a new hire to work independently. The APP vacancies and onboarding resulted in increased costs to the division incurred by physician moonlighting to cover open shifts. The hourly physician moonlighting rate at JHBMC is $150. All costs were calculated on the basis of a 40-hour work week. We performed a pre- and postanalysis of outcomes of interest between January 2009 and June 2018. This study was exempt from institutional review board review.

Intervention

In 2014, a one year APP clinical fellowship in hospital medicine was started. The fellows evaluate and manage patients working one-on-one with an experienced hospitalist faculty member. The program consists of 80% clinical experience in the inpatient setting and 20% didactic instruction (Table 1). Up to four fellows are accepted each year and are eligible for hire after training if vacancies exist. The program is cost neutral and was financed by downsizing, through attrition, two physician FTEs. Four APP fellows’ salaries are the equivalent of two entry-level hospitalist physicians’ salaries at JHBMC. The annual salary for an APP fellow is $69,000.

Downsizing by two physician FTEs meant that one less doctor was scheduled every day. The patient load previously seen by that one doctor (10 patients) was absorbed by the MD–APP fellow dyads. Paired with a fellow, each physician sees a higher cap of 13 patients, and it takes six weeks for the fellows to ramp-up to this patient load. When the fellow first starts, the team sees 10 patients. Every two weeks, the pair’s census increases by one patient to the cap of 13. Collectively, the four APP fellow–MD dyads make it possible for four physicians to see an additional 12 patients. The two extra patients absorbed by the service per day results in a net increase in capacity of up to 730 patient encounters a year.

Outcomes and Analysis

Our main outcomes of interest were duration of onboarding and cost incurred by the division to (1) staff the service during a vacancy and (2) onboard new hires. Secondary outcomes included duration of vacancy and total time spent with the group. We collected basic demographic data on participants, including, age, gender, and race. Demographics and outcomes of interest were compared pre- (2009-2013) and post- (2014-2018) initiation of the APP clinical fellowship using the chi-square test, the t-test for normally distributed data, and the Wilcoxon rank-sum for nonnormally distributed data, as appropriate. The normality of the data distribution was tested using the Shapiro-Wilk W test. Two-tailed P values less than .05 were considered to be statistically significant. Results were analyzed using Stata/MP version 13.0 (StataCorp Inc, College Station, Texas).

Twelve fellows have been recruited, and of these, 10 have graduated. Two chose to leave the program prior to completion. Of the 10 fellows that have graduated, six have been hired into our group, one was hired within our facility, and three were hired as hospitalists at other institutions. The median time from APP school graduation to hire was also not different between the two groups (10.5 vs 3.9 months, P = .069). In addition, the total time that the new APP hires spent with the group was nonstatistically significantly different between the two periods (17.9 vs 18.3 months, P = .735). Both the mean duration of onboarding and the cost to the division were significantly reduced after implementation of the program (25.4 vs 11.0 weeks, P = .017 and $361,714 vs $66,000, P = .004; Table 2).

The yearly cost of an APP vacancy and onboarding is incurred by doctor moonlighting costs (at the rate of $150 per hour) to cover open shifts. The mean duration of vacancies and onboarding each year was 34.9 and 25.4 weeks, respectively, before the fellowship. The yearly cost of onboarding, after the establishment of the fellowship, is a maximum of $66,000, derived from physician moonlighting to cover the six-week ramp-up at the very beginning of the fellowship and the five weeks of orientation to the pulmonary and chemical dependency units after the fellowship (Table 3).

Our APP clinical fellowship in hospital medicine at JHBMC has produced several benefits. First, the fellowship has become a pipeline for filling APP vacancies within our division. We have been able to hire for four consecutive years from the fellowship. Second, the ready availability of high-functioning and efficient APP hospitalists has cut down on the onboarding time for our new APP hires. Many new APP graduates lack confidence in caring for complex hospitalized patients. Following our 12-month clinical fellowship, our matriculated fellows are able to practice at the top of their license immediately and confidently. Third, the reduced vacancy and shortened onboarding periods have reduced costs to the division. Fourth, the fellowship has created additional teaching avenues for the faculty. The medicine units at JHBMC are comprised of hospitalist and internal medicine residency services. The hospitalists spend the majority of their clinical time in direct patient care; however, they rotate on the residency service for two weeks out of the year. The majority of physicians welcome the chance to teach more, and partnering with an APP fellow provides that opportunity.

As we have developed and grown this program, the one great challenge has been what to do with graduating fellows when we cannot hire them. Fortunately, the market for highly qualified, well trained APPs is strong, and every one of the fellows that we could not hire within our group has been able to find a position either within our facility or outside our institution. To facilitate this process, program directors and recruiters are invited to meet with the fellows toward the end of their fellowship to share employment opportunities with them.

Our study has limitations. First, had the $276,000 from the attrition of two physicians been used to hire nonfellow APPs under the old model, then the costs of the two models would have been similar, but this was simply not possible because the positions could not be filled. Second, this is a single-site experience, and our findings may not be generalizable, particularly those pertaining to remuneration. Third, our study was underpowered to detect small but important differences in characteristics of APPs, especially time from graduation to hire, before and after the implementation of our fellowship. Further research comparing various programs both in structure and outcomes—such as fellows’ readiness for practice, costs, duration of vacancies, and provider satisfaction—are an important next step.

We have developed a pool of applicants within our division to fill vacancies left by turnover from senior NPs and PAs. This program has reduced costs and improved the joy of practice for both doctors and APPs. As the need for highly qualified NPs and PAs in hospital medicine continues to grow, we may see more APP fellowships in hospital medicine in the United States.

Acknowledgments

The authors thank the advanced practice providers who have helped us grow and refine our fellowship.

Disclosures

The authors have nothing to disclose

There is an increasing utilization of advanced practice providers (APPs) in the delivery of healthcare in the United States.^1,2 As of 2016, there were 157, 025 nurse practitioners (NPs) and 102,084 physician assistants (PAs) with a projected growth rate of 6.8% and 4.3%, respectively, which exceeds the physician growth rate of 1.1%.² This increased growth rate has been attributed to the expectation that APPs can enhance the quality of physician care, relieve physician shortages, and reduce service costs, as APPs are less expensive to hire than physicians.^3,4 Hospital medicine is the fastest growing medical field in the United States, and approximately 83% of hospitalist groups around the country utilize APPs; however, the demand for hospitalists continues to exceed the supply, and this has led to increased utilization of APPs in hospital medicine.^5-10

APPs receive very limited inpatient training and there is wide variation in their clinical abilities after graduation.¹¹ This is an issue that has become exacerbated in recent years by a change in the training process for PAs. Before 2005, PA programs were typically two to three years long and required the same prerequisite courses as medical schools.¹¹ PA students completed more than 2,000 hours of clinical rotations and then had to pass the Physician Assistant National Certifying Exam before they could practice.¹² Traditionally, PA programs typically attracted students with prior healthcare experience.¹¹ In 2005, PA programs began transitioning from bachelor’s degrees to requiring a master’s level degree for completion of the programs. This has shifted the demographics of the students matriculating to younger students with little-to-no prior healthcare experience; moreover, these fresh graduates lack exposure to hospital medicine.¹¹

NPs usually gain clinical experience working as registered nurses (RNs) for two or more years prior to entry into the NP program. NP programs for baccalaureate-prepared RNs vary in length from two to three years.² There is an acute care focus for NPs in training; however, there is no standardized training or licensure to ensure that hospital medicine competencies are met.^13-15 Some studies have shown that a lack of structured support has been found to affect NP role transition negatively during the first year of practice,¹⁶ and graduating NPs have indicated that they needed more out of their clinical education in terms of content, clinical experience, and competency testing.¹⁷

Hiring new APP graduates as hospitalists requires a longer and more rigorous onboarding process. On‐the‐job training in hospital medicine for new APP graduates can take as long as six to 12 months in order for them to acquire the basic skill set necessary to adequately manage hospitalized patients.¹⁵ This extended onboarding is costly because the APPs are receiving a full hospitalist salary, yet they are not functioning at full capacity. Ideally, there should be an intermediary training step between graduation and employment as hospitalist APPs. Studies have shown that APPs are interested in formal postgraduate hospital medicine training, even if it means having a lower stipend during the first year after graduating from their NP or PA program.^9,15,18

The growing need for hospitalists, driven by residency work-hour reform, increased age and complexity of patients, and the need to improve the quality of inpatient care while simultaneously reducing waste, has contributed to the increasing utilization of and need for highly qualified APPs in hospital medicine.^11,19,20 We established a fellowship to train APPs. The goal of this study was to determine if an APP fellowship is a cost-effective pipeline for filling vacancies within a hospitalist program.

Design and Setting

Johns Hopkins Bayview Medical Center (JHBMC) is a 440 bed hospital in Baltimore Maryland. The hospitalist group was started in 1996 with one physician seeing approximately 500 discharges a year. Over the last 20 years, the group has grown and is now its own division with 57 providers, including 42 physicians, 11 APPs, and four APP fellows. The hospitalist division manages ~7,000 discharges a year, which corresponds to approximately 60% of admissions to general medicine. Hospitalist APPs help staff general medicine by working alongside doctors and admitting patients during the day and night. The APPs also staff the pulmonary step down unit with a pulmonary attending and the chemical dependency unit with an internal medicine addiction specialist.

The growth of the division of hospital medicine at JHBMC is a result of increasing volumes and reduced residency duty hours. The increasing full time equivalents (FTEs) resulted in a need for APPs; however, vacancies went unfilled for an average of 35 weeks due to the time it took to post open positions, interview applicants, and hire applicants through the credentialing process. Further, it took as long as 22 to 34 weeks for a new hire to work independently. The APP vacancies and onboarding resulted in increased costs to the division incurred by physician moonlighting to cover open shifts. The hourly physician moonlighting rate at JHBMC is $150. All costs were calculated on the basis of a 40-hour work week. We performed a pre- and postanalysis of outcomes of interest between January 2009 and June 2018. This study was exempt from institutional review board review.

Intervention

In 2014, a one year APP clinical fellowship in hospital medicine was started. The fellows evaluate and manage patients working one-on-one with an experienced hospitalist faculty member. The program consists of 80% clinical experience in the inpatient setting and 20% didactic instruction (Table 1). Up to four fellows are accepted each year and are eligible for hire after training if vacancies exist. The program is cost neutral and was financed by downsizing, through attrition, two physician FTEs. Four APP fellows’ salaries are the equivalent of two entry-level hospitalist physicians’ salaries at JHBMC. The annual salary for an APP fellow is $69,000.

Downsizing by two physician FTEs meant that one less doctor was scheduled every day. The patient load previously seen by that one doctor (10 patients) was absorbed by the MD–APP fellow dyads. Paired with a fellow, each physician sees a higher cap of 13 patients, and it takes six weeks for the fellows to ramp-up to this patient load. When the fellow first starts, the team sees 10 patients. Every two weeks, the pair’s census increases by one patient to the cap of 13. Collectively, the four APP fellow–MD dyads make it possible for four physicians to see an additional 12 patients. The two extra patients absorbed by the service per day results in a net increase in capacity of up to 730 patient encounters a year.

Outcomes and Analysis

Our main outcomes of interest were duration of onboarding and cost incurred by the division to (1) staff the service during a vacancy and (2) onboard new hires. Secondary outcomes included duration of vacancy and total time spent with the group. We collected basic demographic data on participants, including, age, gender, and race. Demographics and outcomes of interest were compared pre- (2009-2013) and post- (2014-2018) initiation of the APP clinical fellowship using the chi-square test, the t-test for normally distributed data, and the Wilcoxon rank-sum for nonnormally distributed data, as appropriate. The normality of the data distribution was tested using the Shapiro-Wilk W test. Two-tailed P values less than .05 were considered to be statistically significant. Results were analyzed using Stata/MP version 13.0 (StataCorp Inc, College Station, Texas).

Twelve fellows have been recruited, and of these, 10 have graduated. Two chose to leave the program prior to completion. Of the 10 fellows that have graduated, six have been hired into our group, one was hired within our facility, and three were hired as hospitalists at other institutions. The median time from APP school graduation to hire was also not different between the two groups (10.5 vs 3.9 months, P = .069). In addition, the total time that the new APP hires spent with the group was nonstatistically significantly different between the two periods (17.9 vs 18.3 months, P = .735). Both the mean duration of onboarding and the cost to the division were significantly reduced after implementation of the program (25.4 vs 11.0 weeks, P = .017 and $361,714 vs $66,000, P = .004; Table 2).

The yearly cost of an APP vacancy and onboarding is incurred by doctor moonlighting costs (at the rate of $150 per hour) to cover open shifts. The mean duration of vacancies and onboarding each year was 34.9 and 25.4 weeks, respectively, before the fellowship. The yearly cost of onboarding, after the establishment of the fellowship, is a maximum of $66,000, derived from physician moonlighting to cover the six-week ramp-up at the very beginning of the fellowship and the five weeks of orientation to the pulmonary and chemical dependency units after the fellowship (Table 3).

Our APP clinical fellowship in hospital medicine at JHBMC has produced several benefits. First, the fellowship has become a pipeline for filling APP vacancies within our division. We have been able to hire for four consecutive years from the fellowship. Second, the ready availability of high-functioning and efficient APP hospitalists has cut down on the onboarding time for our new APP hires. Many new APP graduates lack confidence in caring for complex hospitalized patients. Following our 12-month clinical fellowship, our matriculated fellows are able to practice at the top of their license immediately and confidently. Third, the reduced vacancy and shortened onboarding periods have reduced costs to the division. Fourth, the fellowship has created additional teaching avenues for the faculty. The medicine units at JHBMC are comprised of hospitalist and internal medicine residency services. The hospitalists spend the majority of their clinical time in direct patient care; however, they rotate on the residency service for two weeks out of the year. The majority of physicians welcome the chance to teach more, and partnering with an APP fellow provides that opportunity.

As we have developed and grown this program, the one great challenge has been what to do with graduating fellows when we cannot hire them. Fortunately, the market for highly qualified, well trained APPs is strong, and every one of the fellows that we could not hire within our group has been able to find a position either within our facility or outside our institution. To facilitate this process, program directors and recruiters are invited to meet with the fellows toward the end of their fellowship to share employment opportunities with them.

Our study has limitations. First, had the $276,000 from the attrition of two physicians been used to hire nonfellow APPs under the old model, then the costs of the two models would have been similar, but this was simply not possible because the positions could not be filled. Second, this is a single-site experience, and our findings may not be generalizable, particularly those pertaining to remuneration. Third, our study was underpowered to detect small but important differences in characteristics of APPs, especially time from graduation to hire, before and after the implementation of our fellowship. Further research comparing various programs both in structure and outcomes—such as fellows’ readiness for practice, costs, duration of vacancies, and provider satisfaction—are an important next step.

We have developed a pool of applicants within our division to fill vacancies left by turnover from senior NPs and PAs. This program has reduced costs and improved the joy of practice for both doctors and APPs. As the need for highly qualified NPs and PAs in hospital medicine continues to grow, we may see more APP fellowships in hospital medicine in the United States.

Acknowledgments

The authors thank the advanced practice providers who have helped us grow and refine our fellowship.

Disclosures

The authors have nothing to disclose