Original Research

Provider attire has come under scrutiny in the more recent medical literature. Epidemiologic data have shown that lab coats, ties, and other articles of clothing are frequently contaminated with disease-causing pathogens including methicillin-resistant Staphylococcus aureus , vancomycin-resistant enterococci, Acinetobacter species, Enterobacteriaceae, Pseudomona s species, and Clostridium difficile.¹ Clothing may serve as a vector for spread of these bacteria and may contribute to hospital-acquired infections, increased cost of care, and patient morbidity. Prior to February 2015, the dermatology service line at Geisinger Medical Center in Danville, Pennsylvania, had followed a formal dress code that included white lab coats (white coats) along with long-sleeve shirts and ties/bowties for male providers and blouses, skirts, dress pants, and dresses for female providers. After a review of the recent literature on contamination rates of provider attire,² we transitioned away from formal attire to adopt fitted, embroidered, black or navy blue scrubs to be worn in the clinic (Figure). Fitted scrubs differ from traditional unisex operating room scrubs, conferring a more professional appearance.

Limited research has shown that dermatology patients may have a slight preference for formal provider attire.^2,3 In these studies, patients were shown photographs of providers in various dress (ie, professional attire, business attire, casual attire, scrubs). Patients preferred or had more confidence in the photograph of the provider in professional attire^2,3; however, it is unclear if dermatology provider attire has any measurable effect on overall patient satisfaction. Patient satisfaction relies on a myriad of factors, including both spoken and unspoken communication skills. Patient satisfaction has become an integral part of health care, and with an emphasis on value-based care, it will likely be one determining factor in how providers are reimbursed for their services.^4,5 In this study, we investigated if a change from formal attire to fitted scrubs influenced patient satisfaction using a common third-party patient satisfaction survey.

Methods

Patient Satisfaction Survey
We conducted a retrospective cohort study analyzing 10 questions from the care provider section of the Press Ganey third-party patient satisfaction survey regarding providers in our dermatology service line. Only providers with at least 12 months of survey data before (study period 1) and after (study period 2) the change in attire were included in the study. Mohs surgeons were excluded, as they already wore fitted scrubs in the clinic. Residents also were excluded, as they are rapidly developing their patient communication skills and may have a notable change in patient satisfaction over a 2-year period.

The survey data were collected, and provider names were removed and replaced with alphanumeric codes to protect anonymity while still allowing individual provider analysis. Aggregate patient comments from surveys before and after the change in attire were digitally searched using the terms scrub, coat, white, attire, and clothing for pertinent positive or negative comments.

Outcomes
We compared individual and aggregate satisfaction scores for our providers during the 12-month periods before and after the adoption of fitted scrubs. The primary outcome was statistically significant change in patient satisfaction scores before and after the institution of fitted scrubs. Secondary outcomes included summation of patient comments, both positive and negative, regarding provider attire, as recorded on satisfaction surveys.

Statistical Analysis
Overall survey scores and scores on individual survey items were summarized using mean (SD), median and interquartile range, or frequency counts and percentage, as appropriate. The overall satisfaction score and responses to individual survey items were compared using Mantel-Haenszel or Pearson χ² tests, as appropriate.

Assuming an equal number of surveys would be completed during study periods 1 and 2, an average (SD) satisfaction score of 95.4 (15), we calculated that as many as 2136 surveys would be needed to conclude satisfaction scores are the same for equivalence limits of −1.9 and 1.9 (a 1% difference). As few as 352 surveys would be needed to conclude satisfaction scores are the same for equivalence limits of −4.7 and 4.7 (a 5% difference). Sample size calculations assume 80% power and a significance level of 0.05. Comparison of responses for study periods 1 and 2 were made using the Mantel-Haenszel χ² test.

Because more than 80% of respondents selected very good for each question, the responses also were treated as dichotomous variables with a category for very good and a category for responses that were lower than very good (ie, good, fair, poor, very poor). Responses of very good versus less than very good were compared for the study periods 1 and 2 using the Pearson χ² test.

Two versions of an overall score were analyzed. The first version was for patients who responded to at least 1 of 10 survey items. If responses to all the items were very good, the patient was assigned to the category of all very good. If a patient answered any of the questions with a response less than very good, he/she was categorized as at least 1 less than very good. The second version was for patients who responded to all 10 survey items. If all 10 responses were very good, the patient was assigned to a category of all very good. If any of the 10 responses were less than very good, he/she was categorized as at least 1 less than very good. Differences between study periods for both score versions were tested using the Pearson χ² test.

 

Results

Data for 22 providers in the dermatology service line—13 staff dermatologists, 6 physician assistants, 1 nurse practitioner, and 2 podiatrists—were included in the study, with a total of 7702 patient satisfaction surveys completed between February 1, 2014, and January 31, 2016: 3511 were completed between February 1, 2014, and January 31, 2015 (study period 1), and 4191 were completed between February 1, 2015, and January 31, 2016 (study period 2).

Analysis of the overall distribution of possible responses for each survey item showed significant differences between study periods 1 and 2 for friendliness/courtesy of the care provider (P=.0307), explanations the care provider gave about the problem or condition (P=.0038), concern the care provider showed for questions or worries (P=.0087), care provider’s efforts to include the patient in decisions about treatment (P=.0377), and patient confidence in the care provider (P=.0156). These survey items trended toward more positive responses in study period 2. The full results are provided in eTable 1.

The analysis that looked at responses as binary (very good vs less than very good) showed a greater proportion of very good responses for friendliness/courtesy of the care provider (P=.0438), explanations the care provider gave about the problem or condition (P=.0115), concern the care provider showed for questions or worries (P=.0188), and patient confidence in the care provider (P=.0417). The full results are provided in eTable 2.

There were no significant differences in the overall satisfaction scores between the first and second study periods. The differences were statistically significant when the overall score was calculated if any questions were answered (P=.5177) and when the overall score was calculated if all 10 questions were answered (P=.9959). For patients who responded to all survey items, 75.3% selected all very good responses for both the first and second study periods.

Review of the surveys for comments from both study periods revealed only a single patient comment pertaining to attire. The comment, which was submitted during study period 2, was considered positive, referring to the fitted scrubs as neat and professional. No negative comments were found during either period.

Comment

In this study, we did not find that a change from formal attire to fitted scrubs had a measurable negative impact on patient satisfaction scores. Conversely, we found a small but statistically significant improvement on several survey items after the change to fitted scrubs. The data suggest that changing from formal attire to fitted scrubs in an outpatient dermatology clinic had little impact on overall patient satisfaction. Only 1 positive comment and no negative comments were received regarding providers wearing fitted scrubs.

A prior study in an outpatient gynecology/obstetrics clinic showed similar results.⁶ In that study, providers were randomly assigned to business attire, casual attire, or scrubs. A 10-question patient satisfaction survey was designed that specifically avoided asking about provider attire to reduce any bias. The study found that over a 3-month period, attire had no influence on patient satisfaction.⁶

Our data suggest that factors beyond provider attire have the greatest influence on patient satisfaction scores. Patient satisfaction is likely driven by other factors such as provider communication skills, concern for patient well-being, ability to empathize, and timeliness. Given the biologic plausibility of increased infection rate from contaminated provider attire, we feel that comfortable, washable, fitted scrubs provide a sanitary and acceptable alternative to more traditional formal provider attire in the office setting. Bearman et al¹ suggest consideration of a bare-below-the-elbows policy (with or without scrubs) for inpatient services and lab coats (if worn per facility policy), and other articles of clothing should be laundered frequently or if visibly soiled. We feel these policies also can be applied to outpatient dermatology clinics, as long as the rationale is well communicated to all parties.

Several items on the patient satisfaction survey were statistically improved during the second study period; however, it is impossible to determine if provider attire was an important factor in this change. Improvement in satisfaction scores could be attributed to ongoing departmental and institutional emphasis on patient care and servic

Anecdotally, most providers in our department were enthusiastic and supportive of the change to fitted scrubs. It is possible that provider happiness is reflected in improved patient satisfaction scores. Provider satisfaction has been shown to correlate with patient satisfaction.⁷

Limitations include possible other unmeasured variables that had a more substantial impact on patient satisfaction survey results. We also recognize that the survey used in this study contained no questions that directly asked patients about their satisfaction with provider attire; however, bias or any preconception patients may have had regarding attire may have been avoided in the process. We also were not able to separate patient surveys based on age or other demographics. Finally, our results may not be generalizable to other settings where patient perceptions may be different from those of central Pennsylvania.

Conclusion

Transitioning from formal provider attire to fitted scrubs did not have a strong impact on overall patient satisfaction scores in an outpatient dermatology clinic. Providers and institutions should consider this information when developing dress code policies.

Provider attire has come under scrutiny in the more recent medical literature. Epidemiologic data have shown that lab coats, ties, and other articles of clothing are frequently contaminated with disease-causing pathogens including methicillin-resistant Staphylococcus aureus , vancomycin-resistant enterococci, Acinetobacter species, Enterobacteriaceae, Pseudomona s species, and Clostridium difficile.¹ Clothing may serve as a vector for spread of these bacteria and may contribute to hospital-acquired infections, increased cost of care, and patient morbidity. Prior to February 2015, the dermatology service line at Geisinger Medical Center in Danville, Pennsylvania, had followed a formal dress code that included white lab coats (white coats) along with long-sleeve shirts and ties/bowties for male providers and blouses, skirts, dress pants, and dresses for female providers. After a review of the recent literature on contamination rates of provider attire,² we transitioned away from formal attire to adopt fitted, embroidered, black or navy blue scrubs to be worn in the clinic (Figure). Fitted scrubs differ from traditional unisex operating room scrubs, conferring a more professional appearance.

Limited research has shown that dermatology patients may have a slight preference for formal provider attire.^2,3 In these studies, patients were shown photographs of providers in various dress (ie, professional attire, business attire, casual attire, scrubs). Patients preferred or had more confidence in the photograph of the provider in professional attire^2,3; however, it is unclear if dermatology provider attire has any measurable effect on overall patient satisfaction. Patient satisfaction relies on a myriad of factors, including both spoken and unspoken communication skills. Patient satisfaction has become an integral part of health care, and with an emphasis on value-based care, it will likely be one determining factor in how providers are reimbursed for their services.^4,5 In this study, we investigated if a change from formal attire to fitted scrubs influenced patient satisfaction using a common third-party patient satisfaction survey.

Methods

Patient Satisfaction Survey
We conducted a retrospective cohort study analyzing 10 questions from the care provider section of the Press Ganey third-party patient satisfaction survey regarding providers in our dermatology service line. Only providers with at least 12 months of survey data before (study period 1) and after (study period 2) the change in attire were included in the study. Mohs surgeons were excluded, as they already wore fitted scrubs in the clinic. Residents also were excluded, as they are rapidly developing their patient communication skills and may have a notable change in patient satisfaction over a 2-year period.

The survey data were collected, and provider names were removed and replaced with alphanumeric codes to protect anonymity while still allowing individual provider analysis. Aggregate patient comments from surveys before and after the change in attire were digitally searched using the terms scrub, coat, white, attire, and clothing for pertinent positive or negative comments.

Outcomes
We compared individual and aggregate satisfaction scores for our providers during the 12-month periods before and after the adoption of fitted scrubs. The primary outcome was statistically significant change in patient satisfaction scores before and after the institution of fitted scrubs. Secondary outcomes included summation of patient comments, both positive and negative, regarding provider attire, as recorded on satisfaction surveys.

Statistical Analysis
Overall survey scores and scores on individual survey items were summarized using mean (SD), median and interquartile range, or frequency counts and percentage, as appropriate. The overall satisfaction score and responses to individual survey items were compared using Mantel-Haenszel or Pearson χ² tests, as appropriate.

Assuming an equal number of surveys would be completed during study periods 1 and 2, an average (SD) satisfaction score of 95.4 (15), we calculated that as many as 2136 surveys would be needed to conclude satisfaction scores are the same for equivalence limits of −1.9 and 1.9 (a 1% difference). As few as 352 surveys would be needed to conclude satisfaction scores are the same for equivalence limits of −4.7 and 4.7 (a 5% difference). Sample size calculations assume 80% power and a significance level of 0.05. Comparison of responses for study periods 1 and 2 were made using the Mantel-Haenszel χ² test.

Because more than 80% of respondents selected very good for each question, the responses also were treated as dichotomous variables with a category for very good and a category for responses that were lower than very good (ie, good, fair, poor, very poor). Responses of very good versus less than very good were compared for the study periods 1 and 2 using the Pearson χ² test.

Two versions of an overall score were analyzed. The first version was for patients who responded to at least 1 of 10 survey items. If responses to all the items were very good, the patient was assigned to the category of all very good. If a patient answered any of the questions with a response less than very good, he/she was categorized as at least 1 less than very good. The second version was for patients who responded to all 10 survey items. If all 10 responses were very good, the patient was assigned to a category of all very good. If any of the 10 responses were less than very good, he/she was categorized as at least 1 less than very good. Differences between study periods for both score versions were tested using the Pearson χ² test.

 

Results

Data for 22 providers in the dermatology service line—13 staff dermatologists, 6 physician assistants, 1 nurse practitioner, and 2 podiatrists—were included in the study, with a total of 7702 patient satisfaction surveys completed between February 1, 2014, and January 31, 2016: 3511 were completed between February 1, 2014, and January 31, 2015 (study period 1), and 4191 were completed between February 1, 2015, and January 31, 2016 (study period 2).

Analysis of the overall distribution of possible responses for each survey item showed significant differences between study periods 1 and 2 for friendliness/courtesy of the care provider (P=.0307), explanations the care provider gave about the problem or condition (P=.0038), concern the care provider showed for questions or worries (P=.0087), care provider’s efforts to include the patient in decisions about treatment (P=.0377), and patient confidence in the care provider (P=.0156). These survey items trended toward more positive responses in study period 2. The full results are provided in eTable 1.

The analysis that looked at responses as binary (very good vs less than very good) showed a greater proportion of very good responses for friendliness/courtesy of the care provider (P=.0438), explanations the care provider gave about the problem or condition (P=.0115), concern the care provider showed for questions or worries (P=.0188), and patient confidence in the care provider (P=.0417). The full results are provided in eTable 2.

There were no significant differences in the overall satisfaction scores between the first and second study periods. The differences were statistically significant when the overall score was calculated if any questions were answered (P=.5177) and when the overall score was calculated if all 10 questions were answered (P=.9959). For patients who responded to all survey items, 75.3% selected all very good responses for both the first and second study periods.

Review of the surveys for comments from both study periods revealed only a single patient comment pertaining to attire. The comment, which was submitted during study period 2, was considered positive, referring to the fitted scrubs as neat and professional. No negative comments were found during either period.

Comment

In this study, we did not find that a change from formal attire to fitted scrubs had a measurable negative impact on patient satisfaction scores. Conversely, we found a small but statistically significant improvement on several survey items after the change to fitted scrubs. The data suggest that changing from formal attire to fitted scrubs in an outpatient dermatology clinic had little impact on overall patient satisfaction. Only 1 positive comment and no negative comments were received regarding providers wearing fitted scrubs.

A prior study in an outpatient gynecology/obstetrics clinic showed similar results.⁶ In that study, providers were randomly assigned to business attire, casual attire, or scrubs. A 10-question patient satisfaction survey was designed that specifically avoided asking about provider attire to reduce any bias. The study found that over a 3-month period, attire had no influence on patient satisfaction.⁶

Our data suggest that factors beyond provider attire have the greatest influence on patient satisfaction scores. Patient satisfaction is likely driven by other factors such as provider communication skills, concern for patient well-being, ability to empathize, and timeliness. Given the biologic plausibility of increased infection rate from contaminated provider attire, we feel that comfortable, washable, fitted scrubs provide a sanitary and acceptable alternative to more traditional formal provider attire in the office setting. Bearman et al¹ suggest consideration of a bare-below-the-elbows policy (with or without scrubs) for inpatient services and lab coats (if worn per facility policy), and other articles of clothing should be laundered frequently or if visibly soiled. We feel these policies also can be applied to outpatient dermatology clinics, as long as the rationale is well communicated to all parties.

Several items on the patient satisfaction survey were statistically improved during the second study period; however, it is impossible to determine if provider attire was an important factor in this change. Improvement in satisfaction scores could be attributed to ongoing departmental and institutional emphasis on patient care and servic

Anecdotally, most providers in our department were enthusiastic and supportive of the change to fitted scrubs. It is possible that provider happiness is reflected in improved patient satisfaction scores. Provider satisfaction has been shown to correlate with patient satisfaction.⁷

Limitations include possible other unmeasured variables that had a more substantial impact on patient satisfaction survey results. We also recognize that the survey used in this study contained no questions that directly asked patients about their satisfaction with provider attire; however, bias or any preconception patients may have had regarding attire may have been avoided in the process. We also were not able to separate patient surveys based on age or other demographics. Finally, our results may not be generalizable to other settings where patient perceptions may be different from those of central Pennsylvania.

Conclusion

Transitioning from formal provider attire to fitted scrubs did not have a strong impact on overall patient satisfaction scores in an outpatient dermatology clinic. Providers and institutions should consider this information when developing dress code policies.

Provider attire has come under scrutiny in the more recent medical literature. Epidemiologic data have shown that lab coats, ties, and other articles of clothing are frequently contaminated with disease-causing pathogens including methicillin-resistant Staphylococcus aureus , vancomycin-resistant enterococci, Acinetobacter species, Enterobacteriaceae, Pseudomona s species, and Clostridium difficile.¹ Clothing may serve as a vector for spread of these bacteria and may contribute to hospital-acquired infections, increased cost of care, and patient morbidity. Prior to February 2015, the dermatology service line at Geisinger Medical Center in Danville, Pennsylvania, had followed a formal dress code that included white lab coats (white coats) along with long-sleeve shirts and ties/bowties for male providers and blouses, skirts, dress pants, and dresses for female providers. After a review of the recent literature on contamination rates of provider attire,² we transitioned away from formal attire to adopt fitted, embroidered, black or navy blue scrubs to be worn in the clinic (Figure). Fitted scrubs differ from traditional unisex operating room scrubs, conferring a more professional appearance.

Limited research has shown that dermatology patients may have a slight preference for formal provider attire.^2,3 In these studies, patients were shown photographs of providers in various dress (ie, professional attire, business attire, casual attire, scrubs). Patients preferred or had more confidence in the photograph of the provider in professional attire^2,3; however, it is unclear if dermatology provider attire has any measurable effect on overall patient satisfaction. Patient satisfaction relies on a myriad of factors, including both spoken and unspoken communication skills. Patient satisfaction has become an integral part of health care, and with an emphasis on value-based care, it will likely be one determining factor in how providers are reimbursed for their services.^4,5 In this study, we investigated if a change from formal attire to fitted scrubs influenced patient satisfaction using a common third-party patient satisfaction survey.

Methods

Patient Satisfaction Survey
We conducted a retrospective cohort study analyzing 10 questions from the care provider section of the Press Ganey third-party patient satisfaction survey regarding providers in our dermatology service line. Only providers with at least 12 months of survey data before (study period 1) and after (study period 2) the change in attire were included in the study. Mohs surgeons were excluded, as they already wore fitted scrubs in the clinic. Residents also were excluded, as they are rapidly developing their patient communication skills and may have a notable change in patient satisfaction over a 2-year period.

The survey data were collected, and provider names were removed and replaced with alphanumeric codes to protect anonymity while still allowing individual provider analysis. Aggregate patient comments from surveys before and after the change in attire were digitally searched using the terms scrub, coat, white, attire, and clothing for pertinent positive or negative comments.

Outcomes
We compared individual and aggregate satisfaction scores for our providers during the 12-month periods before and after the adoption of fitted scrubs. The primary outcome was statistically significant change in patient satisfaction scores before and after the institution of fitted scrubs. Secondary outcomes included summation of patient comments, both positive and negative, regarding provider attire, as recorded on satisfaction surveys.

Statistical Analysis
Overall survey scores and scores on individual survey items were summarized using mean (SD), median and interquartile range, or frequency counts and percentage, as appropriate. The overall satisfaction score and responses to individual survey items were compared using Mantel-Haenszel or Pearson χ² tests, as appropriate.

Assuming an equal number of surveys would be completed during study periods 1 and 2, an average (SD) satisfaction score of 95.4 (15), we calculated that as many as 2136 surveys would be needed to conclude satisfaction scores are the same for equivalence limits of −1.9 and 1.9 (a 1% difference). As few as 352 surveys would be needed to conclude satisfaction scores are the same for equivalence limits of −4.7 and 4.7 (a 5% difference). Sample size calculations assume 80% power and a significance level of 0.05. Comparison of responses for study periods 1 and 2 were made using the Mantel-Haenszel χ² test.

Because more than 80% of respondents selected very good for each question, the responses also were treated as dichotomous variables with a category for very good and a category for responses that were lower than very good (ie, good, fair, poor, very poor). Responses of very good versus less than very good were compared for the study periods 1 and 2 using the Pearson χ² test.

Two versions of an overall score were analyzed. The first version was for patients who responded to at least 1 of 10 survey items. If responses to all the items were very good, the patient was assigned to the category of all very good. If a patient answered any of the questions with a response less than very good, he/she was categorized as at least 1 less than very good. The second version was for patients who responded to all 10 survey items. If all 10 responses were very good, the patient was assigned to a category of all very good. If any of the 10 responses were less than very good, he/she was categorized as at least 1 less than very good. Differences between study periods for both score versions were tested using the Pearson χ² test.

 

Results

Data for 22 providers in the dermatology service line—13 staff dermatologists, 6 physician assistants, 1 nurse practitioner, and 2 podiatrists—were included in the study, with a total of 7702 patient satisfaction surveys completed between February 1, 2014, and January 31, 2016: 3511 were completed between February 1, 2014, and January 31, 2015 (study period 1), and 4191 were completed between February 1, 2015, and January 31, 2016 (study period 2).

Analysis of the overall distribution of possible responses for each survey item showed significant differences between study periods 1 and 2 for friendliness/courtesy of the care provider (P=.0307), explanations the care provider gave about the problem or condition (P=.0038), concern the care provider showed for questions or worries (P=.0087), care provider’s efforts to include the patient in decisions about treatment (P=.0377), and patient confidence in the care provider (P=.0156). These survey items trended toward more positive responses in study period 2. The full results are provided in eTable 1.

The analysis that looked at responses as binary (very good vs less than very good) showed a greater proportion of very good responses for friendliness/courtesy of the care provider (P=.0438), explanations the care provider gave about the problem or condition (P=.0115), concern the care provider showed for questions or worries (P=.0188), and patient confidence in the care provider (P=.0417). The full results are provided in eTable 2.

There were no significant differences in the overall satisfaction scores between the first and second study periods. The differences were statistically significant when the overall score was calculated if any questions were answered (P=.5177) and when the overall score was calculated if all 10 questions were answered (P=.9959). For patients who responded to all survey items, 75.3% selected all very good responses for both the first and second study periods.

Review of the surveys for comments from both study periods revealed only a single patient comment pertaining to attire. The comment, which was submitted during study period 2, was considered positive, referring to the fitted scrubs as neat and professional. No negative comments were found during either period.

Comment

In this study, we did not find that a change from formal attire to fitted scrubs had a measurable negative impact on patient satisfaction scores. Conversely, we found a small but statistically significant improvement on several survey items after the change to fitted scrubs. The data suggest that changing from formal attire to fitted scrubs in an outpatient dermatology clinic had little impact on overall patient satisfaction. Only 1 positive comment and no negative comments were received regarding providers wearing fitted scrubs.

A prior study in an outpatient gynecology/obstetrics clinic showed similar results.⁶ In that study, providers were randomly assigned to business attire, casual attire, or scrubs. A 10-question patient satisfaction survey was designed that specifically avoided asking about provider attire to reduce any bias. The study found that over a 3-month period, attire had no influence on patient satisfaction.⁶

Our data suggest that factors beyond provider attire have the greatest influence on patient satisfaction scores. Patient satisfaction is likely driven by other factors such as provider communication skills, concern for patient well-being, ability to empathize, and timeliness. Given the biologic plausibility of increased infection rate from contaminated provider attire, we feel that comfortable, washable, fitted scrubs provide a sanitary and acceptable alternative to more traditional formal provider attire in the office setting. Bearman et al¹ suggest consideration of a bare-below-the-elbows policy (with or without scrubs) for inpatient services and lab coats (if worn per facility policy), and other articles of clothing should be laundered frequently or if visibly soiled. We feel these policies also can be applied to outpatient dermatology clinics, as long as the rationale is well communicated to all parties.

Several items on the patient satisfaction survey were statistically improved during the second study period; however, it is impossible to determine if provider attire was an important factor in this change. Improvement in satisfaction scores could be attributed to ongoing departmental and institutional emphasis on patient care and servic

Anecdotally, most providers in our department were enthusiastic and supportive of the change to fitted scrubs. It is possible that provider happiness is reflected in improved patient satisfaction scores. Provider satisfaction has been shown to correlate with patient satisfaction.⁷

Limitations include possible other unmeasured variables that had a more substantial impact on patient satisfaction survey results. We also recognize that the survey used in this study contained no questions that directly asked patients about their satisfaction with provider attire; however, bias or any preconception patients may have had regarding attire may have been avoided in the process. We also were not able to separate patient surveys based on age or other demographics. Finally, our results may not be generalizable to other settings where patient perceptions may be different from those of central Pennsylvania.

Conclusion

Transitioning from formal provider attire to fitted scrubs did not have a strong impact on overall patient satisfaction scores in an outpatient dermatology clinic. Providers and institutions should consider this information when developing dress code policies.

Peripherally inserted central catheters (PICCs) have evolved since their inception in the early 1970s and are used with increasing frequency for pediatric inpatients and outpatients.^1-3 Emerging literature, including a meta-analysis of international observational studies,⁴ reports PICC failure (complications necessitating premature removal) occurs in up to 30% of PICCs, most commonly due to infection, thrombosis, occlusion, and fracture.^4-7 Raffini et al.⁷ report the increasing incidence of pediatric PICC-related thrombosis increases morbidity and mortality⁸ and negatively impacts future vessel health and preservation.⁹

PICCs have progressed from relatively simple, silicone-based catheters with an external clamp to chemically engineered polyurethane with pressure-activated valves placed at the proximal or distal catheter hub with the intent to reduce occlusion.¹⁰ Further modernization of PICC material occurred with the incorporation of antithrombogenic (AT) material (Endexo^®). These PICCs are designed to contain a nonstick polymer, which is designed to reduce the adherence of blood components (platelets and clotting factors) and inhibit thrombus formation (and hence prevent deep vein thrombosis andocclusion, as well as inhibit microbial biofilm attachment [and subsequent infection]).¹¹

In addition to new materials, other aspects of this PICC design have been the addition of a pressure-activated safety valve (PASV^®) built into the proximal hub. Pressure-activated valve technology promises to prevent catheter occlusion by reducing blood reflux into the PICC; the valve opens with pressure during infusion and aspiration and remains closed with normal venous pressure, circumventing the need for clinicians to manually clamp the PICC and reducing human error and the potential for thrombosis, occlusion, and fracture development.¹² Hoffer et al.¹³ reported half as many occlusions of valved PICCs (3.3%) compared with nonvalved or clamped PICCs (7.1%); although not statistically significant (P = .10), perhaps due to the small sample, overall complications, including occlusion and infection, were significantly lessened with the valved PICC (35% vs 79%; P = .02). Comparatively, Pittiruti et al.¹⁴ conducted a trial of 2 types of valved PICCs with an open-ended, nonvalved PICC and found no reduction in PICC occlusion or catheter malfunction.

Today, PICC use is common for patients who require short-to-medium intravenous therapy. PICCs are increasingly recognized for their significant complications, including thrombosis and infection.¹⁵ Novel PICC technology, including the incorporation of AT material such as Endexo^® and PASV^®, may reduce complications; however, the clinical efficacy, cost-effectiveness, and acceptability of these innovations have not been tested through randomized trials in pediatric patients. In accordance with Medical Research Council guidelines¹⁶ for developing interventions, we pilot tested the feasibility of the BioFlo^® PICC, including intervention acceptability, compliance, recruitment, and initial estimates of effect, in anticipation of a subsequent full-scale efficacy randomized controlled trial. Our secondary aim was to compare the effectiveness of the BioFlo^® PICC with Endexo^® and PASV^® technology in reducing PICC complications and failure.

METHODS

Design

We undertook a pilot randomized controlled trial comparing the standard polyurethane PICC (with external clamp) with the BioFlo^® PICC (with internal valve) in preventing catheter failure in pediatric patients. The study was prospectively registered with the Australian Clinical Trials Registry (ACTRN12615001290583), and the research protocol was published.¹⁷

Study Setting

The study commenced in March 2016 at the Lady Cilento Children’s Hospital in South Brisbane, Australia, a tertiary-level, specialist, pediatric teaching hospital in Queensland, Australia, providing full-spectrum health services to children and young people from birth to 18 years of age. Recruitment, including data collection, was completed in November 2016.

Sample

The target sample size was 110 participants, 50 participants per group plus 10% for potential attrition, as determined by standard pilot trial sample size recommendations.¹⁸ With ethics approval, the sample size was later increased to 150 participants in order to adequately pilot a microbiological substudy method (published separately).¹⁷ Participants were consecutively recruited if they met the inclusion criteria: PICC insertion, age <18 years, predicted hospital stay >24 hours, single-lumen PICC, and written informed consent by an English-speaking, legal parent or guardian. Patients were excluded if they had a current (<48 hours) blood stream infection (BSI), vessel size <2 mm, could not speak English without an interpreter, required a multilumen PICC, or were previously enrolled in the study.

 

Interventions

Participants were randomized to receive either of the following PICCs: (1) standard care: Cook™ polyurethane, turbo-ject, power-injectable PICC (Cook Medical, Bloomington, IN) or (2) comparison: BioFlo^® polyurethane with Endexo^® technology (AngioDynamics Inc, Queensbury, NY).

Outcomes

The primary outcome was feasibility of a full-efficacy trial established by composite analysis of the elements of eligibility (>70% of patients will be eligible), recruitment (>70% of patients will agree to enroll), retention and attrition (<15% of participants are lost to follow-up or withdraw from the study), protocol adherence (>80% of participants receive their allocated, randomly assigned study product), missing data (<10% of data are missed during data collection), parent and healthcare staff satisfaction, and PICC failure effect size estimates to allow sample size calculations.^18,19 PICC failure was defined as the following complications associated with PICC removal: (1) catheter-associated BSI,^8,20-22 (2) local site infection,²² (3) venous thrombosis,²³ (4) occlusion,^24,25 (5) PICC fracture, or (6) PICC dislodgement.^25,26 Parents (or caregivers) and healthcare staff were asked to rate their level of confidence with the study product and ease of PICC removal by using a 0 to 100 numeric rating scale (NRS) of increasing confidence and/or ease. These data were collected at the time of PICC removal. Operators were also asked to rate their levels of satisfaction with the insertion equipment and ease of PICC insertion immediately upon completion of the insertion procedure (both 0-100 NRS of increasing satisfaction and/or ease). Secondary outcomes included individual PICC complications (eg, occlusion) occurring at any time point during the PICC dwell (including at removal), adverse events, pain, redness at the insertion site, and overall PICC dwell.

Study Procedures

The research nurse (ReN) screened operating theater lists for patients, obtained written informed consent, and initiated the randomization. Randomization was computer generated, and web based via Griffith University (https://www151.griffith.edu.au/random) to ensure allocation concealment until study entry. Patients were randomly assigned in a 1:1 ratio with computer-generated and randomly varied block sizes of 2 and 4. Data were collected by the ReN on the day of insertion, at day 1 postinsertion, then every 2 to 3 days thereafter so that PICCs were checked at least twice per week until study completion. Participants were included in the trial until 12 weeks post-PICC insertion, study withdrawal or PICC removal (whichever came first), with an additional 48 hours follow-up for infection outcomes. Patient review was face to face during the inpatient stay, with discharged patients’ follow-up occurring via outpatient clinics, hospital-in-the-home service, or telephone.

Data collection was via Research Electronic Data Capture (http://project-redcap.org/). The ReN collected data on primary and secondary outcomes by using the predefined criteria. Demographic and clinical data were collected to assess the success of randomization, describe the participant group, and display characteristics known to increase the risk of PICC complication and thrombosis. A blinded radiologist and infectious disease specialist reviewed and diagnosed thrombosis of deep veins and catheter-associated BSI outcomes, respectively.

PICC Procedures

Extensive prestudy education for 2 months prior to trial commencement was provided to all clinicians involved with the insertion and care of PICCs, including the study products. PICCs were inserted in an operating theater environment by a qualified consultant pediatric anesthetist, a senior anesthetic registrar or fellow in an approved anesthetic training program, or pediatric vascular access nurse practitioner. Ultrasound guidance was used to assess a patient’s vasculature and puncture the vessel. The operator chose the PICC size on the basis of clinical judgment of vessel size and patient needs and then inserted the allocated PICC.²⁷ Preferred PICC tip location was the cavoatrial junction. All PICC tip positions were confirmed with a chest x-ray before use.

Postinsertion, PICCs were managed by local interdisciplinary clinicians in accordance with local practice guidelines.^27-31 PICC care and management includes the use of 2% chlorhexidine gluconate in 70% alcohol for site antisepsis and neutral displacement needleless connectors (TUTA Pulse; Medical Australia Limited, Lidcombe, New South Wales, Australia); normal saline was used to flush after medication administration, and if the device was not in use for 6 hours or longer, heparin is instilled with securement via bordered polyurethane dressing (Tegaderm 1616; 3M, St Paul, Minnesota) and a sutureless securement device (Statlock VPPCSP; Bard, Georgia).

Statistical Analyses

Data were exported to Stata 15³² for cleaning and analysis. Data cleaning of outlying figures and missing and implausible data was undertaken prior to analysis. Missing data were not imputed. The PICC was the unit of measurement, and all randomly assigned patients were analyzed on an intention-to-treat basis.³³ Descriptive statistics (frequencies and percentages) were used to ascertain the primary outcome of feasibility for the larger trial. Incidence rates (per 1,000 catheter days) and rate ratios, including 95% confidence intervals (CIs), were calculated. The comparability of groups at baseline was described across demographic, clinical, and device characteristics. Kaplan-Meier survival curves (with log-rank tests) were used to compare PICC failure between study groups over time. Associations between baseline characteristics and failure were described by calculating hazard ratios (HRs). Univariable Cox regression was performed only due to the relatively low number of outcomes. P values of <.05 were considered statistically significant.

 

Ethics

The Children’s Health Service District, Queensland (Human Research Ethics Committee/15/QRCH/164), and Griffith University (2016/077) Human Research Ethics Committees provided ethics and governance approval. Informed consent was obtained from parents or legal guardians, with children providing youth assent if they were 7 years or older, dependent upon cognitive ability.

RESULTS

Participant and PICC Characteristics

Participant and PICC characteristics are described in Table 1. The majority of participant and PICC characteristics were balanced between intervention groups. The mean patient age was 7.3 years (standard deviation 5.0; range 0-18). PICC insertion was most commonly for a respiratory diagnosis (n = 98; 65%). Most PICCs were placed in the basilica vein (n = 115; 79%), with insertion being successful on the first attempt (n = 125; 86%). There was some imbalance (>10% absolute difference between groups) in nurse practitioner and registrar insertions (standard care 35% and 23% vs BioFlo^® 51% and 8%, respectively) and patients with leucocytes <1000 µl (standard care 10% vs BioFlo^® 22%). Optimal PICC tip location at the cavoatrial junction was higher with BioFlo^® than standard care, although this difference was <10%.

Feasibility Outcomes

As shown in Figure 1, the majority of feasibility criteria were met, with 94% of 188 screened patients being eligible to participate and 97% of eligible patients consenting to enroll. Of 150 patients randomly assigned, 4 (1 in standard care and 3 in BioFlo^®) were unable to have a PICC inserted or the procedure was cancelled. Demographic data only were collected for these 4 patients. No participants were lost to follow-up, and no primary outcome data were missing. Staff satisfaction with insertion kit and ease of insertion, ease of removal of the PICC, and parental confidence in the PICC product were similar across both groups (Table 2).

PICC Failure and Complications

In total, 24 of 146 participants (16%) experienced PICC failure. There were 16 (22%) failures of standard care PICCs and 8 (11%) failures of BioFlo^® PICCs. This corresponded to incident rates of 12.6 and 7.3 per 1000 catheter days (incident rate ratio 0.58; 95% CI, 0.21-1.43; P = .172; Table 2). Failure was most commonly from thrombosis (n = 5; 7%) or occlusion (n = 5; 7%) in the standard care group, with lower incidences in the BioFlo^® group (n = 2 [3%] and n = 1 [1%], respectively). Figure 2 displays survival from PICC failure.

Considering the entire PICC dwell, of the 74 standard care patients, 49 (66%) had no complications, 9 (12%) had complications during the dwell but none at removal, 2 (3%) had no complications during the dwell but had a complication (ie, failure) at removal, and 14 (19%) had complications during the dwell and at removal. For the 72 BioFlo^® patients, 61 (85%) had no PICC complications, 3 (4%) had complications during the dwell but none at removal, 4 (5.5%) had no complications during the dwell but had a complication (ie, failure) at removal, and 4 (5.5%) had complications during the dwell and at removal.

More than twice as many standard care patients as BioFlo^® patients had a complication during the PICC dwell, and this difference was statistically significant (25 of 74, 34% vs 11 of 72, 15%; P = .009; Table 2). These results are consistent with the Kaplan-Meier curve, which shows longer complication-free survival with BioFlo^® (Figure 2A and 2B). The median BioFlo^® dwell was 1 day longer (13.8 vs 12.9 days), and the median time to first complication was one day later (4.0 BioFlo^® vs 3.0 standard care; Table 2).

As per supplementary Table 1, univariate Cox regression identified PICC failure as significantly associated with tip placement in the proximal superior vena cava (SVC) compared to the SVC–right atrium junction (HR 2.61; 95% CI, 1.17-5.82; P = .024). Reduced risk of PICC failure was significantly associated with any infusion during the dwell (continuous fluid infusion, P = .007; continuous antibiotic, P = .042; or intermittent infusion, P = .046) compared to no infusion. Other variables potentially influencing the risk of failure included PICC insertion by nurse specialist compared to consultant anesthetist (HR 2.61; 95% CI, 0.85-5.44) or registrar (HR 1.97; 95% CI, 0.57-6.77). These differences were not statistically significant; however, baseline imbalance between study groups for this variable and the feasibility design preclude absolute conclusions.

DISCUSSION

This is the first pilot feasibility trial of new PICC materials and valve design incorporated in the BioFlo^® PICC in the pediatric population. The trial incorporated best practice for randomized trials, including using a concurrent control group, centralized and concealed randomization, predetermined feasibility criteria, and a registered and published trial protocol.¹⁷ As in other studies,^15,24,34 PICC failure and complication prevalence was unacceptably high for this essential device. Standard care PICCs failed twice as often as the new BioFlo^® PICCs (22% vs 11%), which is a clinically important difference. As researchers in a pilot study, we did not expect to detect statistically significant differences; however, we found that overall complications during the dwell occurred significantly more with the standard care than BioFlo^® PICCs (P = .009).

 

BioFlo^® PICC material offers a major advancement in PICC material through the incorporation of AT technologies into catheter materials, such as PICCs. Endexo^® is a low molecular–weight, fluoro-oligomeric additive that self-locates to the top few nanometers of the material surface. When added to power-injectable polyurethane, the additive results in a strong but passive, nonstick, fluorinated surface in the base PICC material. This inhibits platelet adhesion, suppresses protein procoagulant conformation, and thereby reduces thrombus formation in medical devices. Additionally, Endexo^® is not a catheter coating; rather, it is incorporated within the polyurethane of the PICC, thereby ensuring these AT properties are present on the internal, external, and cut surfaces of the PICC. If this technology can reduce complication during treatment and reduce failure from infection, thrombosis, occlusion, fracture, and dislodgement, it will improve patient outcomes considerably and lower health system costs. Previous studies investigating valve technology in PICC design to reduce occlusion have been inconclusive.^12-14,35,36 Occlusion (both partial and complete) was less frequent in our study with the BioFlo^® group (n = 3; 4%) compared to the standard care group (n = 6; 8%). The results of this pilot study suggest that either the Endexo^® material or PASV^® technology has a positive association with occlusion reduction during PICC treatment.

Thrombosis was the primary failure type for the standard care PICCs, comprising one-third of failures. All but one patient with radiologically confirmed thrombosis required the removal of the PICC prior to completion of treatment. The decision to remove the PICC or retain and treat conservatively remained with the treating team. Raffini et al.⁷ found thrombosis to increase in patients with one or more coexisting chronic medical condition. Slightly more standard care than BioFlo^® patients were free of such comorbidities (25% vs 16%), yet standard care patients still had the higher number of thromboses (7% vs 3%). Morgenthaler and Rodriguez³⁷ reported vascular access-associated thrombosis in pediatrics to be less common than in adults but higher in medically complex children. Worryingly, Menendez et al.³⁸ reported pediatric thrombosis to be largely asymptomatic, so the true incidence in our study is likely higher because only radiologically confirmed thromboses were recorded.

Occlusion (partial or complete) was the predominant complication across the study, being associated with one-third of all failures. When occlusion complications during the dwell (some of which were resolved with treatment), in addition to those causing failure, were considered, this number was even greater. Occlusion complications are prevalent and costly. Smith et al.²⁴ reported that occlusion was the most common reason for PICC removal and the most likely complication to delay treatment. Both the BioFlo^® and standard care PICCs are pressure rated with good tensile strength; however, fracture occurred in 4% (n = 3) of standard care PICCs compared to no fractures in BioFlo^® PICCs. Although the numbers are small, it may suggest a superior tensile strength of the BioFlo^® material.

This study reinforces previously published results^24,38 that PICC tip position is important and can influence complications, such as occlusion and thrombosis. In addition, we found a significant association with failure when PICCs did not have a continuous infusion. These findings reinforce the need for optimal tip location at insertion and ongoing flushing and maintenance of PICCs not used for infusions.

Limitations of this study include the small sample size, which was not designed to detect statistical differences in the primary outcome between groups. Despite randomization, there were slight imbalances at baseline for inserter type and leukocyte count, although these were not significantly associated with PICC failure in the Cox regression (data not shown), and thus were unlikely to influence findings. Additionally, a difference of <10% was associated with PICC tip position, favoring the BioFlo^® group. PICC tip position outside the cavoatrial junction was positively associated with failure; therefore, the effect of tip positioning on outcomes is difficult to ascertain given the small sample size and feasibility nature of the study. Further study is warranted to further explore this effect. The population sampled was pediatric medical and surgical inpatients with a vessel size >2 mm attending the operating theater suite for PICC insertion, thereby limiting the study’s generalizability to adults and other populations, including neonates and those with PICCs inserted in the pediatric intensive care unit. The study could not be blinded because study products had to be visible to the clinical and research staff. However, it is unlikely that staff would intentionally sabotage PICCs to bias the study. Blinding was possible for the assessment of blood culture and ultrasound reports to diagnose infection and thrombosis. Strengths of this study included 100% protocol adherence, and no patients were lost to follow-up.

 

CONCLUSION

These results confirm that PICC failure is unacceptably high and suggest that the innovative BioFlo^® PICC material and design holds promise to improve PICC outcomes by reducing complications and overall PICC failure. Trials of this technology are feasible, safe, and acceptable to healthcare staff and parents. Further trials are required, including in other patient populations, to definitively identify clinical, cost-effective methods to prevent PICC failure and improve reliability during treatment.

Acknowledgments

The authors thank the children and parents of Lady Cilento Children’s Hospital for participating in this important research. A special thank you goes to the nurses within the Vascular Assessment and Management Service and to Karen Turner, Julieta Woosley, and Anna Dean for their efforts in data collecting and ensuring protocol adherence.

Disclosure

Griffith University has received unrestricted, investigator-initiated research or educational grants to support the research of T. K., A. J. U., and C. R. M. from product manufacturers 3M, Adhezion Inc, AngioDynamics, Bard Medical, Baxter, B. Braun Medical Inc, Becton Dickinson, CareFusion, Centurion Medical Products, Cook Medical, Entrotech, FloMedical, ICU Medical Inc, Medical Australia Limited, Medtronic, Smiths Medical, and Teleflex. Griffith University has received consultancy payments on behalf of C. R. M., A. J. U., and T. K. from manufacturers 3M, AngioDynamics, Bard Medical, B. Braun Medical Inc, Becton Dickinson, CareFusion, Mayo Healthcare Inc, ResQDevices, and Smiths Medical. AngioDynamics (the BioFlo^® PICC manufacturer) provided partial funds to undertake this research via an unrestricted donation to Griffith University (but not the study authors). Queensland Health provided in-kind support to fund the remainder of the trial. The funders had no role in the study design, collection, analysis, or interpretation of the data, writing of the report, or decision to submit the article for publication.

Peripherally inserted central catheters (PICCs) have evolved since their inception in the early 1970s and are used with increasing frequency for pediatric inpatients and outpatients.^1-3 Emerging literature, including a meta-analysis of international observational studies,⁴ reports PICC failure (complications necessitating premature removal) occurs in up to 30% of PICCs, most commonly due to infection, thrombosis, occlusion, and fracture.^4-7 Raffini et al.⁷ report the increasing incidence of pediatric PICC-related thrombosis increases morbidity and mortality⁸ and negatively impacts future vessel health and preservation.⁹

PICCs have progressed from relatively simple, silicone-based catheters with an external clamp to chemically engineered polyurethane with pressure-activated valves placed at the proximal or distal catheter hub with the intent to reduce occlusion.¹⁰ Further modernization of PICC material occurred with the incorporation of antithrombogenic (AT) material (Endexo^®). These PICCs are designed to contain a nonstick polymer, which is designed to reduce the adherence of blood components (platelets and clotting factors) and inhibit thrombus formation (and hence prevent deep vein thrombosis andocclusion, as well as inhibit microbial biofilm attachment [and subsequent infection]).¹¹

In addition to new materials, other aspects of this PICC design have been the addition of a pressure-activated safety valve (PASV^®) built into the proximal hub. Pressure-activated valve technology promises to prevent catheter occlusion by reducing blood reflux into the PICC; the valve opens with pressure during infusion and aspiration and remains closed with normal venous pressure, circumventing the need for clinicians to manually clamp the PICC and reducing human error and the potential for thrombosis, occlusion, and fracture development.¹² Hoffer et al.¹³ reported half as many occlusions of valved PICCs (3.3%) compared with nonvalved or clamped PICCs (7.1%); although not statistically significant (P = .10), perhaps due to the small sample, overall complications, including occlusion and infection, were significantly lessened with the valved PICC (35% vs 79%; P = .02). Comparatively, Pittiruti et al.¹⁴ conducted a trial of 2 types of valved PICCs with an open-ended, nonvalved PICC and found no reduction in PICC occlusion or catheter malfunction.

Today, PICC use is common for patients who require short-to-medium intravenous therapy. PICCs are increasingly recognized for their significant complications, including thrombosis and infection.¹⁵ Novel PICC technology, including the incorporation of AT material such as Endexo^® and PASV^®, may reduce complications; however, the clinical efficacy, cost-effectiveness, and acceptability of these innovations have not been tested through randomized trials in pediatric patients. In accordance with Medical Research Council guidelines¹⁶ for developing interventions, we pilot tested the feasibility of the BioFlo^® PICC, including intervention acceptability, compliance, recruitment, and initial estimates of effect, in anticipation of a subsequent full-scale efficacy randomized controlled trial. Our secondary aim was to compare the effectiveness of the BioFlo^® PICC with Endexo^® and PASV^® technology in reducing PICC complications and failure.

METHODS

Design

We undertook a pilot randomized controlled trial comparing the standard polyurethane PICC (with external clamp) with the BioFlo^® PICC (with internal valve) in preventing catheter failure in pediatric patients. The study was prospectively registered with the Australian Clinical Trials Registry (ACTRN12615001290583), and the research protocol was published.¹⁷

Study Setting

The study commenced in March 2016 at the Lady Cilento Children’s Hospital in South Brisbane, Australia, a tertiary-level, specialist, pediatric teaching hospital in Queensland, Australia, providing full-spectrum health services to children and young people from birth to 18 years of age. Recruitment, including data collection, was completed in November 2016.

Sample

The target sample size was 110 participants, 50 participants per group plus 10% for potential attrition, as determined by standard pilot trial sample size recommendations.¹⁸ With ethics approval, the sample size was later increased to 150 participants in order to adequately pilot a microbiological substudy method (published separately).¹⁷ Participants were consecutively recruited if they met the inclusion criteria: PICC insertion, age <18 years, predicted hospital stay >24 hours, single-lumen PICC, and written informed consent by an English-speaking, legal parent or guardian. Patients were excluded if they had a current (<48 hours) blood stream infection (BSI), vessel size <2 mm, could not speak English without an interpreter, required a multilumen PICC, or were previously enrolled in the study.

 

Interventions

Participants were randomized to receive either of the following PICCs: (1) standard care: Cook™ polyurethane, turbo-ject, power-injectable PICC (Cook Medical, Bloomington, IN) or (2) comparison: BioFlo^® polyurethane with Endexo^® technology (AngioDynamics Inc, Queensbury, NY).

Outcomes

The primary outcome was feasibility of a full-efficacy trial established by composite analysis of the elements of eligibility (>70% of patients will be eligible), recruitment (>70% of patients will agree to enroll), retention and attrition (<15% of participants are lost to follow-up or withdraw from the study), protocol adherence (>80% of participants receive their allocated, randomly assigned study product), missing data (<10% of data are missed during data collection), parent and healthcare staff satisfaction, and PICC failure effect size estimates to allow sample size calculations.^18,19 PICC failure was defined as the following complications associated with PICC removal: (1) catheter-associated BSI,^8,20-22 (2) local site infection,²² (3) venous thrombosis,²³ (4) occlusion,^24,25 (5) PICC fracture, or (6) PICC dislodgement.^25,26 Parents (or caregivers) and healthcare staff were asked to rate their level of confidence with the study product and ease of PICC removal by using a 0 to 100 numeric rating scale (NRS) of increasing confidence and/or ease. These data were collected at the time of PICC removal. Operators were also asked to rate their levels of satisfaction with the insertion equipment and ease of PICC insertion immediately upon completion of the insertion procedure (both 0-100 NRS of increasing satisfaction and/or ease). Secondary outcomes included individual PICC complications (eg, occlusion) occurring at any time point during the PICC dwell (including at removal), adverse events, pain, redness at the insertion site, and overall PICC dwell.

Study Procedures

The research nurse (ReN) screened operating theater lists for patients, obtained written informed consent, and initiated the randomization. Randomization was computer generated, and web based via Griffith University (https://www151.griffith.edu.au/random) to ensure allocation concealment until study entry. Patients were randomly assigned in a 1:1 ratio with computer-generated and randomly varied block sizes of 2 and 4. Data were collected by the ReN on the day of insertion, at day 1 postinsertion, then every 2 to 3 days thereafter so that PICCs were checked at least twice per week until study completion. Participants were included in the trial until 12 weeks post-PICC insertion, study withdrawal or PICC removal (whichever came first), with an additional 48 hours follow-up for infection outcomes. Patient review was face to face during the inpatient stay, with discharged patients’ follow-up occurring via outpatient clinics, hospital-in-the-home service, or telephone.

Data collection was via Research Electronic Data Capture (http://project-redcap.org/). The ReN collected data on primary and secondary outcomes by using the predefined criteria. Demographic and clinical data were collected to assess the success of randomization, describe the participant group, and display characteristics known to increase the risk of PICC complication and thrombosis. A blinded radiologist and infectious disease specialist reviewed and diagnosed thrombosis of deep veins and catheter-associated BSI outcomes, respectively.

PICC Procedures

Extensive prestudy education for 2 months prior to trial commencement was provided to all clinicians involved with the insertion and care of PICCs, including the study products. PICCs were inserted in an operating theater environment by a qualified consultant pediatric anesthetist, a senior anesthetic registrar or fellow in an approved anesthetic training program, or pediatric vascular access nurse practitioner. Ultrasound guidance was used to assess a patient’s vasculature and puncture the vessel. The operator chose the PICC size on the basis of clinical judgment of vessel size and patient needs and then inserted the allocated PICC.²⁷ Preferred PICC tip location was the cavoatrial junction. All PICC tip positions were confirmed with a chest x-ray before use.

Postinsertion, PICCs were managed by local interdisciplinary clinicians in accordance with local practice guidelines.^27-31 PICC care and management includes the use of 2% chlorhexidine gluconate in 70% alcohol for site antisepsis and neutral displacement needleless connectors (TUTA Pulse; Medical Australia Limited, Lidcombe, New South Wales, Australia); normal saline was used to flush after medication administration, and if the device was not in use for 6 hours or longer, heparin is instilled with securement via bordered polyurethane dressing (Tegaderm 1616; 3M, St Paul, Minnesota) and a sutureless securement device (Statlock VPPCSP; Bard, Georgia).

Statistical Analyses

Data were exported to Stata 15³² for cleaning and analysis. Data cleaning of outlying figures and missing and implausible data was undertaken prior to analysis. Missing data were not imputed. The PICC was the unit of measurement, and all randomly assigned patients were analyzed on an intention-to-treat basis.³³ Descriptive statistics (frequencies and percentages) were used to ascertain the primary outcome of feasibility for the larger trial. Incidence rates (per 1,000 catheter days) and rate ratios, including 95% confidence intervals (CIs), were calculated. The comparability of groups at baseline was described across demographic, clinical, and device characteristics. Kaplan-Meier survival curves (with log-rank tests) were used to compare PICC failure between study groups over time. Associations between baseline characteristics and failure were described by calculating hazard ratios (HRs). Univariable Cox regression was performed only due to the relatively low number of outcomes. P values of <.05 were considered statistically significant.

 

Ethics

The Children’s Health Service District, Queensland (Human Research Ethics Committee/15/QRCH/164), and Griffith University (2016/077) Human Research Ethics Committees provided ethics and governance approval. Informed consent was obtained from parents or legal guardians, with children providing youth assent if they were 7 years or older, dependent upon cognitive ability.

RESULTS

Participant and PICC Characteristics

Participant and PICC characteristics are described in Table 1. The majority of participant and PICC characteristics were balanced between intervention groups. The mean patient age was 7.3 years (standard deviation 5.0; range 0-18). PICC insertion was most commonly for a respiratory diagnosis (n = 98; 65%). Most PICCs were placed in the basilica vein (n = 115; 79%), with insertion being successful on the first attempt (n = 125; 86%). There was some imbalance (>10% absolute difference between groups) in nurse practitioner and registrar insertions (standard care 35% and 23% vs BioFlo^® 51% and 8%, respectively) and patients with leucocytes <1000 µl (standard care 10% vs BioFlo^® 22%). Optimal PICC tip location at the cavoatrial junction was higher with BioFlo^® than standard care, although this difference was <10%.

Feasibility Outcomes

As shown in Figure 1, the majority of feasibility criteria were met, with 94% of 188 screened patients being eligible to participate and 97% of eligible patients consenting to enroll. Of 150 patients randomly assigned, 4 (1 in standard care and 3 in BioFlo^®) were unable to have a PICC inserted or the procedure was cancelled. Demographic data only were collected for these 4 patients. No participants were lost to follow-up, and no primary outcome data were missing. Staff satisfaction with insertion kit and ease of insertion, ease of removal of the PICC, and parental confidence in the PICC product were similar across both groups (Table 2).

PICC Failure and Complications

In total, 24 of 146 participants (16%) experienced PICC failure. There were 16 (22%) failures of standard care PICCs and 8 (11%) failures of BioFlo^® PICCs. This corresponded to incident rates of 12.6 and 7.3 per 1000 catheter days (incident rate ratio 0.58; 95% CI, 0.21-1.43; P = .172; Table 2). Failure was most commonly from thrombosis (n = 5; 7%) or occlusion (n = 5; 7%) in the standard care group, with lower incidences in the BioFlo^® group (n = 2 [3%] and n = 1 [1%], respectively). Figure 2 displays survival from PICC failure.

Considering the entire PICC dwell, of the 74 standard care patients, 49 (66%) had no complications, 9 (12%) had complications during the dwell but none at removal, 2 (3%) had no complications during the dwell but had a complication (ie, failure) at removal, and 14 (19%) had complications during the dwell and at removal. For the 72 BioFlo^® patients, 61 (85%) had no PICC complications, 3 (4%) had complications during the dwell but none at removal, 4 (5.5%) had no complications during the dwell but had a complication (ie, failure) at removal, and 4 (5.5%) had complications during the dwell and at removal.

More than twice as many standard care patients as BioFlo^® patients had a complication during the PICC dwell, and this difference was statistically significant (25 of 74, 34% vs 11 of 72, 15%; P = .009; Table 2). These results are consistent with the Kaplan-Meier curve, which shows longer complication-free survival with BioFlo^® (Figure 2A and 2B). The median BioFlo^® dwell was 1 day longer (13.8 vs 12.9 days), and the median time to first complication was one day later (4.0 BioFlo^® vs 3.0 standard care; Table 2).

As per supplementary Table 1, univariate Cox regression identified PICC failure as significantly associated with tip placement in the proximal superior vena cava (SVC) compared to the SVC–right atrium junction (HR 2.61; 95% CI, 1.17-5.82; P = .024). Reduced risk of PICC failure was significantly associated with any infusion during the dwell (continuous fluid infusion, P = .007; continuous antibiotic, P = .042; or intermittent infusion, P = .046) compared to no infusion. Other variables potentially influencing the risk of failure included PICC insertion by nurse specialist compared to consultant anesthetist (HR 2.61; 95% CI, 0.85-5.44) or registrar (HR 1.97; 95% CI, 0.57-6.77). These differences were not statistically significant; however, baseline imbalance between study groups for this variable and the feasibility design preclude absolute conclusions.

DISCUSSION

This is the first pilot feasibility trial of new PICC materials and valve design incorporated in the BioFlo^® PICC in the pediatric population. The trial incorporated best practice for randomized trials, including using a concurrent control group, centralized and concealed randomization, predetermined feasibility criteria, and a registered and published trial protocol.¹⁷ As in other studies,^15,24,34 PICC failure and complication prevalence was unacceptably high for this essential device. Standard care PICCs failed twice as often as the new BioFlo^® PICCs (22% vs 11%), which is a clinically important difference. As researchers in a pilot study, we did not expect to detect statistically significant differences; however, we found that overall complications during the dwell occurred significantly more with the standard care than BioFlo^® PICCs (P = .009).

 

BioFlo^® PICC material offers a major advancement in PICC material through the incorporation of AT technologies into catheter materials, such as PICCs. Endexo^® is a low molecular–weight, fluoro-oligomeric additive that self-locates to the top few nanometers of the material surface. When added to power-injectable polyurethane, the additive results in a strong but passive, nonstick, fluorinated surface in the base PICC material. This inhibits platelet adhesion, suppresses protein procoagulant conformation, and thereby reduces thrombus formation in medical devices. Additionally, Endexo^® is not a catheter coating; rather, it is incorporated within the polyurethane of the PICC, thereby ensuring these AT properties are present on the internal, external, and cut surfaces of the PICC. If this technology can reduce complication during treatment and reduce failure from infection, thrombosis, occlusion, fracture, and dislodgement, it will improve patient outcomes considerably and lower health system costs. Previous studies investigating valve technology in PICC design to reduce occlusion have been inconclusive.^12-14,35,36 Occlusion (both partial and complete) was less frequent in our study with the BioFlo^® group (n = 3; 4%) compared to the standard care group (n = 6; 8%). The results of this pilot study suggest that either the Endexo^® material or PASV^® technology has a positive association with occlusion reduction during PICC treatment.

Thrombosis was the primary failure type for the standard care PICCs, comprising one-third of failures. All but one patient with radiologically confirmed thrombosis required the removal of the PICC prior to completion of treatment. The decision to remove the PICC or retain and treat conservatively remained with the treating team. Raffini et al.⁷ found thrombosis to increase in patients with one or more coexisting chronic medical condition. Slightly more standard care than BioFlo^® patients were free of such comorbidities (25% vs 16%), yet standard care patients still had the higher number of thromboses (7% vs 3%). Morgenthaler and Rodriguez³⁷ reported vascular access-associated thrombosis in pediatrics to be less common than in adults but higher in medically complex children. Worryingly, Menendez et al.³⁸ reported pediatric thrombosis to be largely asymptomatic, so the true incidence in our study is likely higher because only radiologically confirmed thromboses were recorded.

Occlusion (partial or complete) was the predominant complication across the study, being associated with one-third of all failures. When occlusion complications during the dwell (some of which were resolved with treatment), in addition to those causing failure, were considered, this number was even greater. Occlusion complications are prevalent and costly. Smith et al.²⁴ reported that occlusion was the most common reason for PICC removal and the most likely complication to delay treatment. Both the BioFlo^® and standard care PICCs are pressure rated with good tensile strength; however, fracture occurred in 4% (n = 3) of standard care PICCs compared to no fractures in BioFlo^® PICCs. Although the numbers are small, it may suggest a superior tensile strength of the BioFlo^® material.

This study reinforces previously published results^24,38 that PICC tip position is important and can influence complications, such as occlusion and thrombosis. In addition, we found a significant association with failure when PICCs did not have a continuous infusion. These findings reinforce the need for optimal tip location at insertion and ongoing flushing and maintenance of PICCs not used for infusions.

Limitations of this study include the small sample size, which was not designed to detect statistical differences in the primary outcome between groups. Despite randomization, there were slight imbalances at baseline for inserter type and leukocyte count, although these were not significantly associated with PICC failure in the Cox regression (data not shown), and thus were unlikely to influence findings. Additionally, a difference of <10% was associated with PICC tip position, favoring the BioFlo^® group. PICC tip position outside the cavoatrial junction was positively associated with failure; therefore, the effect of tip positioning on outcomes is difficult to ascertain given the small sample size and feasibility nature of the study. Further study is warranted to further explore this effect. The population sampled was pediatric medical and surgical inpatients with a vessel size >2 mm attending the operating theater suite for PICC insertion, thereby limiting the study’s generalizability to adults and other populations, including neonates and those with PICCs inserted in the pediatric intensive care unit. The study could not be blinded because study products had to be visible to the clinical and research staff. However, it is unlikely that staff would intentionally sabotage PICCs to bias the study. Blinding was possible for the assessment of blood culture and ultrasound reports to diagnose infection and thrombosis. Strengths of this study included 100% protocol adherence, and no patients were lost to follow-up.

 

CONCLUSION

These results confirm that PICC failure is unacceptably high and suggest that the innovative BioFlo^® PICC material and design holds promise to improve PICC outcomes by reducing complications and overall PICC failure. Trials of this technology are feasible, safe, and acceptable to healthcare staff and parents. Further trials are required, including in other patient populations, to definitively identify clinical, cost-effective methods to prevent PICC failure and improve reliability during treatment.

Acknowledgments

The authors thank the children and parents of Lady Cilento Children’s Hospital for participating in this important research. A special thank you goes to the nurses within the Vascular Assessment and Management Service and to Karen Turner, Julieta Woosley, and Anna Dean for their efforts in data collecting and ensuring protocol adherence.

Disclosure

Griffith University has received unrestricted, investigator-initiated research or educational grants to support the research of T. K., A. J. U., and C. R. M. from product manufacturers 3M, Adhezion Inc, AngioDynamics, Bard Medical, Baxter, B. Braun Medical Inc, Becton Dickinson, CareFusion, Centurion Medical Products, Cook Medical, Entrotech, FloMedical, ICU Medical Inc, Medical Australia Limited, Medtronic, Smiths Medical, and Teleflex. Griffith University has received consultancy payments on behalf of C. R. M., A. J. U., and T. K. from manufacturers 3M, AngioDynamics, Bard Medical, B. Braun Medical Inc, Becton Dickinson, CareFusion, Mayo Healthcare Inc, ResQDevices, and Smiths Medical. AngioDynamics (the BioFlo^® PICC manufacturer) provided partial funds to undertake this research via an unrestricted donation to Griffith University (but not the study authors). Queensland Health provided in-kind support to fund the remainder of the trial. The funders had no role in the study design, collection, analysis, or interpretation of the data, writing of the report, or decision to submit the article for publication.

Peripherally inserted central catheters (PICCs) have evolved since their inception in the early 1970s and are used with increasing frequency for pediatric inpatients and outpatients.^1-3 Emerging literature, including a meta-analysis of international observational studies,⁴ reports PICC failure (complications necessitating premature removal) occurs in up to 30% of PICCs, most commonly due to infection, thrombosis, occlusion, and fracture.^4-7 Raffini et al.⁷ report the increasing incidence of pediatric PICC-related thrombosis increases morbidity and mortality⁸ and negatively impacts future vessel health and preservation.⁹

PICCs have progressed from relatively simple, silicone-based catheters with an external clamp to chemically engineered polyurethane with pressure-activated valves placed at the proximal or distal catheter hub with the intent to reduce occlusion.¹⁰ Further modernization of PICC material occurred with the incorporation of antithrombogenic (AT) material (Endexo^®). These PICCs are designed to contain a nonstick polymer, which is designed to reduce the adherence of blood components (platelets and clotting factors) and inhibit thrombus formation (and hence prevent deep vein thrombosis andocclusion, as well as inhibit microbial biofilm attachment [and subsequent infection]).¹¹

In addition to new materials, other aspects of this PICC design have been the addition of a pressure-activated safety valve (PASV^®) built into the proximal hub. Pressure-activated valve technology promises to prevent catheter occlusion by reducing blood reflux into the PICC; the valve opens with pressure during infusion and aspiration and remains closed with normal venous pressure, circumventing the need for clinicians to manually clamp the PICC and reducing human error and the potential for thrombosis, occlusion, and fracture development.¹² Hoffer et al.¹³ reported half as many occlusions of valved PICCs (3.3%) compared with nonvalved or clamped PICCs (7.1%); although not statistically significant (P = .10), perhaps due to the small sample, overall complications, including occlusion and infection, were significantly lessened with the valved PICC (35% vs 79%; P = .02). Comparatively, Pittiruti et al.¹⁴ conducted a trial of 2 types of valved PICCs with an open-ended, nonvalved PICC and found no reduction in PICC occlusion or catheter malfunction.

Today, PICC use is common for patients who require short-to-medium intravenous therapy. PICCs are increasingly recognized for their significant complications, including thrombosis and infection.¹⁵ Novel PICC technology, including the incorporation of AT material such as Endexo^® and PASV^®, may reduce complications; however, the clinical efficacy, cost-effectiveness, and acceptability of these innovations have not been tested through randomized trials in pediatric patients. In accordance with Medical Research Council guidelines¹⁶ for developing interventions, we pilot tested the feasibility of the BioFlo^® PICC, including intervention acceptability, compliance, recruitment, and initial estimates of effect, in anticipation of a subsequent full-scale efficacy randomized controlled trial. Our secondary aim was to compare the effectiveness of the BioFlo^® PICC with Endexo^® and PASV^® technology in reducing PICC complications and failure.

METHODS

Design

We undertook a pilot randomized controlled trial comparing the standard polyurethane PICC (with external clamp) with the BioFlo^® PICC (with internal valve) in preventing catheter failure in pediatric patients. The study was prospectively registered with the Australian Clinical Trials Registry (ACTRN12615001290583), and the research protocol was published.¹⁷

Study Setting

The study commenced in March 2016 at the Lady Cilento Children’s Hospital in South Brisbane, Australia, a tertiary-level, specialist, pediatric teaching hospital in Queensland, Australia, providing full-spectrum health services to children and young people from birth to 18 years of age. Recruitment, including data collection, was completed in November 2016.

Sample

The target sample size was 110 participants, 50 participants per group plus 10% for potential attrition, as determined by standard pilot trial sample size recommendations.¹⁸ With ethics approval, the sample size was later increased to 150 participants in order to adequately pilot a microbiological substudy method (published separately).¹⁷ Participants were consecutively recruited if they met the inclusion criteria: PICC insertion, age <18 years, predicted hospital stay >24 hours, single-lumen PICC, and written informed consent by an English-speaking, legal parent or guardian. Patients were excluded if they had a current (<48 hours) blood stream infection (BSI), vessel size <2 mm, could not speak English without an interpreter, required a multilumen PICC, or were previously enrolled in the study.

 

Interventions

Participants were randomized to receive either of the following PICCs: (1) standard care: Cook™ polyurethane, turbo-ject, power-injectable PICC (Cook Medical, Bloomington, IN) or (2) comparison: BioFlo^® polyurethane with Endexo^® technology (AngioDynamics Inc, Queensbury, NY).

Outcomes

The primary outcome was feasibility of a full-efficacy trial established by composite analysis of the elements of eligibility (>70% of patients will be eligible), recruitment (>70% of patients will agree to enroll), retention and attrition (<15% of participants are lost to follow-up or withdraw from the study), protocol adherence (>80% of participants receive their allocated, randomly assigned study product), missing data (<10% of data are missed during data collection), parent and healthcare staff satisfaction, and PICC failure effect size estimates to allow sample size calculations.^18,19 PICC failure was defined as the following complications associated with PICC removal: (1) catheter-associated BSI,^8,20-22 (2) local site infection,²² (3) venous thrombosis,²³ (4) occlusion,^24,25 (5) PICC fracture, or (6) PICC dislodgement.^25,26 Parents (or caregivers) and healthcare staff were asked to rate their level of confidence with the study product and ease of PICC removal by using a 0 to 100 numeric rating scale (NRS) of increasing confidence and/or ease. These data were collected at the time of PICC removal. Operators were also asked to rate their levels of satisfaction with the insertion equipment and ease of PICC insertion immediately upon completion of the insertion procedure (both 0-100 NRS of increasing satisfaction and/or ease). Secondary outcomes included individual PICC complications (eg, occlusion) occurring at any time point during the PICC dwell (including at removal), adverse events, pain, redness at the insertion site, and overall PICC dwell.

Study Procedures

The research nurse (ReN) screened operating theater lists for patients, obtained written informed consent, and initiated the randomization. Randomization was computer generated, and web based via Griffith University (https://www151.griffith.edu.au/random) to ensure allocation concealment until study entry. Patients were randomly assigned in a 1:1 ratio with computer-generated and randomly varied block sizes of 2 and 4. Data were collected by the ReN on the day of insertion, at day 1 postinsertion, then every 2 to 3 days thereafter so that PICCs were checked at least twice per week until study completion. Participants were included in the trial until 12 weeks post-PICC insertion, study withdrawal or PICC removal (whichever came first), with an additional 48 hours follow-up for infection outcomes. Patient review was face to face during the inpatient stay, with discharged patients’ follow-up occurring via outpatient clinics, hospital-in-the-home service, or telephone.

Data collection was via Research Electronic Data Capture (http://project-redcap.org/). The ReN collected data on primary and secondary outcomes by using the predefined criteria. Demographic and clinical data were collected to assess the success of randomization, describe the participant group, and display characteristics known to increase the risk of PICC complication and thrombosis. A blinded radiologist and infectious disease specialist reviewed and diagnosed thrombosis of deep veins and catheter-associated BSI outcomes, respectively.

PICC Procedures

Extensive prestudy education for 2 months prior to trial commencement was provided to all clinicians involved with the insertion and care of PICCs, including the study products. PICCs were inserted in an operating theater environment by a qualified consultant pediatric anesthetist, a senior anesthetic registrar or fellow in an approved anesthetic training program, or pediatric vascular access nurse practitioner. Ultrasound guidance was used to assess a patient’s vasculature and puncture the vessel. The operator chose the PICC size on the basis of clinical judgment of vessel size and patient needs and then inserted the allocated PICC.²⁷ Preferred PICC tip location was the cavoatrial junction. All PICC tip positions were confirmed with a chest x-ray before use.

Postinsertion, PICCs were managed by local interdisciplinary clinicians in accordance with local practice guidelines.^27-31 PICC care and management includes the use of 2% chlorhexidine gluconate in 70% alcohol for site antisepsis and neutral displacement needleless connectors (TUTA Pulse; Medical Australia Limited, Lidcombe, New South Wales, Australia); normal saline was used to flush after medication administration, and if the device was not in use for 6 hours or longer, heparin is instilled with securement via bordered polyurethane dressing (Tegaderm 1616; 3M, St Paul, Minnesota) and a sutureless securement device (Statlock VPPCSP; Bard, Georgia).

Statistical Analyses

Data were exported to Stata 15³² for cleaning and analysis. Data cleaning of outlying figures and missing and implausible data was undertaken prior to analysis. Missing data were not imputed. The PICC was the unit of measurement, and all randomly assigned patients were analyzed on an intention-to-treat basis.³³ Descriptive statistics (frequencies and percentages) were used to ascertain the primary outcome of feasibility for the larger trial. Incidence rates (per 1,000 catheter days) and rate ratios, including 95% confidence intervals (CIs), were calculated. The comparability of groups at baseline was described across demographic, clinical, and device characteristics. Kaplan-Meier survival curves (with log-rank tests) were used to compare PICC failure between study groups over time. Associations between baseline characteristics and failure were described by calculating hazard ratios (HRs). Univariable Cox regression was performed only due to the relatively low number of outcomes. P values of <.05 were considered statistically significant.

 

Ethics

The Children’s Health Service District, Queensland (Human Research Ethics Committee/15/QRCH/164), and Griffith University (2016/077) Human Research Ethics Committees provided ethics and governance approval. Informed consent was obtained from parents or legal guardians, with children providing youth assent if they were 7 years or older, dependent upon cognitive ability.

RESULTS

Participant and PICC Characteristics

Participant and PICC characteristics are described in Table 1. The majority of participant and PICC characteristics were balanced between intervention groups. The mean patient age was 7.3 years (standard deviation 5.0; range 0-18). PICC insertion was most commonly for a respiratory diagnosis (n = 98; 65%). Most PICCs were placed in the basilica vein (n = 115; 79%), with insertion being successful on the first attempt (n = 125; 86%). There was some imbalance (>10% absolute difference between groups) in nurse practitioner and registrar insertions (standard care 35% and 23% vs BioFlo^® 51% and 8%, respectively) and patients with leucocytes <1000 µl (standard care 10% vs BioFlo^® 22%). Optimal PICC tip location at the cavoatrial junction was higher with BioFlo^® than standard care, although this difference was <10%.

Feasibility Outcomes

As shown in Figure 1, the majority of feasibility criteria were met, with 94% of 188 screened patients being eligible to participate and 97% of eligible patients consenting to enroll. Of 150 patients randomly assigned, 4 (1 in standard care and 3 in BioFlo^®) were unable to have a PICC inserted or the procedure was cancelled. Demographic data only were collected for these 4 patients. No participants were lost to follow-up, and no primary outcome data were missing. Staff satisfaction with insertion kit and ease of insertion, ease of removal of the PICC, and parental confidence in the PICC product were similar across both groups (Table 2).

PICC Failure and Complications

In total, 24 of 146 participants (16%) experienced PICC failure. There were 16 (22%) failures of standard care PICCs and 8 (11%) failures of BioFlo^® PICCs. This corresponded to incident rates of 12.6 and 7.3 per 1000 catheter days (incident rate ratio 0.58; 95% CI, 0.21-1.43; P = .172; Table 2). Failure was most commonly from thrombosis (n = 5; 7%) or occlusion (n = 5; 7%) in the standard care group, with lower incidences in the BioFlo^® group (n = 2 [3%] and n = 1 [1%], respectively). Figure 2 displays survival from PICC failure.

Considering the entire PICC dwell, of the 74 standard care patients, 49 (66%) had no complications, 9 (12%) had complications during the dwell but none at removal, 2 (3%) had no complications during the dwell but had a complication (ie, failure) at removal, and 14 (19%) had complications during the dwell and at removal. For the 72 BioFlo^® patients, 61 (85%) had no PICC complications, 3 (4%) had complications during the dwell but none at removal, 4 (5.5%) had no complications during the dwell but had a complication (ie, failure) at removal, and 4 (5.5%) had complications during the dwell and at removal.

More than twice as many standard care patients as BioFlo^® patients had a complication during the PICC dwell, and this difference was statistically significant (25 of 74, 34% vs 11 of 72, 15%; P = .009; Table 2). These results are consistent with the Kaplan-Meier curve, which shows longer complication-free survival with BioFlo^® (Figure 2A and 2B). The median BioFlo^® dwell was 1 day longer (13.8 vs 12.9 days), and the median time to first complication was one day later (4.0 BioFlo^® vs 3.0 standard care; Table 2).

As per supplementary Table 1, univariate Cox regression identified PICC failure as significantly associated with tip placement in the proximal superior vena cava (SVC) compared to the SVC–right atrium junction (HR 2.61; 95% CI, 1.17-5.82; P = .024). Reduced risk of PICC failure was significantly associated with any infusion during the dwell (continuous fluid infusion, P = .007; continuous antibiotic, P = .042; or intermittent infusion, P = .046) compared to no infusion. Other variables potentially influencing the risk of failure included PICC insertion by nurse specialist compared to consultant anesthetist (HR 2.61; 95% CI, 0.85-5.44) or registrar (HR 1.97; 95% CI, 0.57-6.77). These differences were not statistically significant; however, baseline imbalance between study groups for this variable and the feasibility design preclude absolute conclusions.

DISCUSSION

This is the first pilot feasibility trial of new PICC materials and valve design incorporated in the BioFlo^® PICC in the pediatric population. The trial incorporated best practice for randomized trials, including using a concurrent control group, centralized and concealed randomization, predetermined feasibility criteria, and a registered and published trial protocol.¹⁷ As in other studies,^15,24,34 PICC failure and complication prevalence was unacceptably high for this essential device. Standard care PICCs failed twice as often as the new BioFlo^® PICCs (22% vs 11%), which is a clinically important difference. As researchers in a pilot study, we did not expect to detect statistically significant differences; however, we found that overall complications during the dwell occurred significantly more with the standard care than BioFlo^® PICCs (P = .009).

 

BioFlo^® PICC material offers a major advancement in PICC material through the incorporation of AT technologies into catheter materials, such as PICCs. Endexo^® is a low molecular–weight, fluoro-oligomeric additive that self-locates to the top few nanometers of the material surface. When added to power-injectable polyurethane, the additive results in a strong but passive, nonstick, fluorinated surface in the base PICC material. This inhibits platelet adhesion, suppresses protein procoagulant conformation, and thereby reduces thrombus formation in medical devices. Additionally, Endexo^® is not a catheter coating; rather, it is incorporated within the polyurethane of the PICC, thereby ensuring these AT properties are present on the internal, external, and cut surfaces of the PICC. If this technology can reduce complication during treatment and reduce failure from infection, thrombosis, occlusion, fracture, and dislodgement, it will improve patient outcomes considerably and lower health system costs. Previous studies investigating valve technology in PICC design to reduce occlusion have been inconclusive.^12-14,35,36 Occlusion (both partial and complete) was less frequent in our study with the BioFlo^® group (n = 3; 4%) compared to the standard care group (n = 6; 8%). The results of this pilot study suggest that either the Endexo^® material or PASV^® technology has a positive association with occlusion reduction during PICC treatment.

Thrombosis was the primary failure type for the standard care PICCs, comprising one-third of failures. All but one patient with radiologically confirmed thrombosis required the removal of the PICC prior to completion of treatment. The decision to remove the PICC or retain and treat conservatively remained with the treating team. Raffini et al.⁷ found thrombosis to increase in patients with one or more coexisting chronic medical condition. Slightly more standard care than BioFlo^® patients were free of such comorbidities (25% vs 16%), yet standard care patients still had the higher number of thromboses (7% vs 3%). Morgenthaler and Rodriguez³⁷ reported vascular access-associated thrombosis in pediatrics to be less common than in adults but higher in medically complex children. Worryingly, Menendez et al.³⁸ reported pediatric thrombosis to be largely asymptomatic, so the true incidence in our study is likely higher because only radiologically confirmed thromboses were recorded.

Occlusion (partial or complete) was the predominant complication across the study, being associated with one-third of all failures. When occlusion complications during the dwell (some of which were resolved with treatment), in addition to those causing failure, were considered, this number was even greater. Occlusion complications are prevalent and costly. Smith et al.²⁴ reported that occlusion was the most common reason for PICC removal and the most likely complication to delay treatment. Both the BioFlo^® and standard care PICCs are pressure rated with good tensile strength; however, fracture occurred in 4% (n = 3) of standard care PICCs compared to no fractures in BioFlo^® PICCs. Although the numbers are small, it may suggest a superior tensile strength of the BioFlo^® material.

This study reinforces previously published results^24,38 that PICC tip position is important and can influence complications, such as occlusion and thrombosis. In addition, we found a significant association with failure when PICCs did not have a continuous infusion. These findings reinforce the need for optimal tip location at insertion and ongoing flushing and maintenance of PICCs not used for infusions.

Limitations of this study include the small sample size, which was not designed to detect statistical differences in the primary outcome between groups. Despite randomization, there were slight imbalances at baseline for inserter type and leukocyte count, although these were not significantly associated with PICC failure in the Cox regression (data not shown), and thus were unlikely to influence findings. Additionally, a difference of <10% was associated with PICC tip position, favoring the BioFlo^® group. PICC tip position outside the cavoatrial junction was positively associated with failure; therefore, the effect of tip positioning on outcomes is difficult to ascertain given the small sample size and feasibility nature of the study. Further study is warranted to further explore this effect. The population sampled was pediatric medical and surgical inpatients with a vessel size >2 mm attending the operating theater suite for PICC insertion, thereby limiting the study’s generalizability to adults and other populations, including neonates and those with PICCs inserted in the pediatric intensive care unit. The study could not be blinded because study products had to be visible to the clinical and research staff. However, it is unlikely that staff would intentionally sabotage PICCs to bias the study. Blinding was possible for the assessment of blood culture and ultrasound reports to diagnose infection and thrombosis. Strengths of this study included 100% protocol adherence, and no patients were lost to follow-up.

 

CONCLUSION

These results confirm that PICC failure is unacceptably high and suggest that the innovative BioFlo^® PICC material and design holds promise to improve PICC outcomes by reducing complications and overall PICC failure. Trials of this technology are feasible, safe, and acceptable to healthcare staff and parents. Further trials are required, including in other patient populations, to definitively identify clinical, cost-effective methods to prevent PICC failure and improve reliability during treatment.

Acknowledgments

The authors thank the children and parents of Lady Cilento Children’s Hospital for participating in this important research. A special thank you goes to the nurses within the Vascular Assessment and Management Service and to Karen Turner, Julieta Woosley, and Anna Dean for their efforts in data collecting and ensuring protocol adherence.

Disclosure

Griffith University has received unrestricted, investigator-initiated research or educational grants to support the research of T. K., A. J. U., and C. R. M. from product manufacturers 3M, Adhezion Inc, AngioDynamics, Bard Medical, Baxter, B. Braun Medical Inc, Becton Dickinson, CareFusion, Centurion Medical Products, Cook Medical, Entrotech, FloMedical, ICU Medical Inc, Medical Australia Limited, Medtronic, Smiths Medical, and Teleflex. Griffith University has received consultancy payments on behalf of C. R. M., A. J. U., and T. K. from manufacturers 3M, AngioDynamics, Bard Medical, B. Braun Medical Inc, Becton Dickinson, CareFusion, Mayo Healthcare Inc, ResQDevices, and Smiths Medical. AngioDynamics (the BioFlo^® PICC manufacturer) provided partial funds to undertake this research via an unrestricted donation to Griffith University (but not the study authors). Queensland Health provided in-kind support to fund the remainder of the trial. The funders had no role in the study design, collection, analysis, or interpretation of the data, writing of the report, or decision to submit the article for publication.

Testing for adrenal insufficiency (AI) is common in the hospital setting. The gold standard remains the insulin tolerance test (ITT), in which cortisol concentration is measured after the induction of hypoglycemia to <35 mg/dL.¹ Alternatively, metyrapone testing works by blocking cortisol synthesis. If pretest adrenocorticotropic hormone (ACTH) concentrations are low and ACTH concentrations do not rise after the administration of metyrapone, the patient is given a diagnosis of AI. Both assays pose some risk to patients with AI and are typically only performed as confirmatory tests. Morning random cortisol concentrations can be used to suggest AI if concentrations are <3 mcg/dL, but they often provide indeterminate results if concentrations are between 3 and 15 mcg/dL.² Thus, morning cortisol concentrations in isolation are not diagnostic of AI. For these reasons, most experts recommend a dynamic, high-dose cosyntropin stimulation testing (CST) with 250 mcg of intravenous cosyntropin to screen for AI. The test can be done any time of day.³ Historically, an incremental response to cosyntropin, or “delta,” was also required to indicate a normal response to stimulation.⁴ However, the baseline cortisol concentration is dependent on circadian rhythm and level of stress. For this reason, a delta, whether large or small, has been abandoned as a requisite for the diagnosis of AI.^5-7 A normal CST is widely accepted to be identified by any cortisol concentration >18 mcg/dL during the test (basal or poststimulation).⁸

The seminal studies by Lindholm, Kehlet, and coauthors^9-11 validated the CST against the gold standard ITT and utilized only 0- and 30-minute cortisol concentrations. A later study in patients with pituitary disease demonstrated that 30-minute concentrations had a stronger correlation with the ITT than 60-minute concentrations (false-negative rate: 10% vs 27%).¹² However, in that study, a higher threshold was used for the 60-minute concentration than for what was obtained at 30 minutes (25.4 vs 21.8 mcg/dL, respectively). Multiple studies have shown that the 60-minute concentration is higher than the 30-minute concentration after cosyntropin stimulation.^4,5,13 Subsequent, small studies of patients who were known to have AI have shown that 60-minute concentrations are as useful as 30-minute concentrations.^5,14,15 Because 30-minute cortisol concentrations are often lower than 60-minute concentrations, a single 30-minute result may lead to a falsely abnormal test.^16,17 As such, the use of a single 60-minute test may be more appropriate. Indeed, some authors have suggested that measuring only 30-minute concentrations may lead to overdiagnosis of AI by missing an appropriate response, serum cortisol >18 mcg/dL, at 60 minutes.^17-19 Peak cortisol concentrations after low-dose cosyntropin stimulation (1 mcg) are seen at 60 minutes, and low-dose stimulation has been shown to be more variable than in the high-dose test (250 mg).^19,20

There is a lack of consensus to guide clinicians as to when cortisol concentrations should be measured after stimulation, and standard references lack uniformity. Commonly accessed medical resources—such as UpToDate and Jameson’s Endocrinology—recommend basal, 30-minute, and 60-minute cortisol concentrations, while Williams Textbook of Endocrinology recommends basal and 30-minute concentrations, and the Washington Manual recommends only a single 30-minute concentration.^7,21,22 Goldman-Cecil Medicine⁸ recommends checking a cortisol concentration between 30 and 60 minutes and recommends the same 18 mcg/dL cutoff for any test obtained in this time period. As a result of these variable recommendations, all 3 time points are often obtained. Prominent review articles continue to recommend checking all 3 concentrations while presenting evidence of peak cortisol response at 60 minutes poststimulation.¹³

In this study, we retrospectively examined CSTs in hospitalized, adult patients both in the intensive care unit (ICU) and hospital ward and/or floor settings to evaluate for significant differences in 30- and 60-minute cortisol concentrations and compare the concordance of screening at each time point alone with traditional CST at all 3 time points. By using these results, we discuss the utility of obtaining 3 cortisol samples.

METHODS

After receiving approval from the institutional review board, we retrospectively reviewed all standard, high-dose CSTs performed on adult inpatients at the Barnes-Jewish Hospital laboratory from January 1, 2012, to August 31, 2013. All patients received the same standard dose (250 mcg cosyntropin, a synthetic ACTH, at a concentration of 1 mcg/mL administered over 2 minutes) regardless of age or weight. We collected patient gender; age; time of baseline cortisol measurement; cortisol results at baseline, 30, and 60 minutes; and patient location (inpatient floor vs ICU status). Tests were included if results from all 3 time points (0, 30, 60 minute) were available.

 

Cortisol concentrations were assessed by the laboratory according to the manufacturer’s instructions by using the ADVIA Centaur Cortisol assay (Siemens Healthcare Diagnostics Inc, Tarrytown, NY), a competitive chemiluminescent immunoassay. For the traditional CST, a cortisol concentration ≥18 mcg/dL at any time point during the test was used to define normal (negative). Patients with a positive (no results >18 mcg/mL) CST were defined as “screen positives” for the purposes of this analysis. Patient location data were available that allowed for an ICU vs non-ICU comparison.

Statistical analyses were performed in SAS version 9.4 (SAS Institute Inc, Cary, North Carolina). Continuous variables were compared by using a 2-tailed Student t test. Percentiles and proportions were compared by using χ² tests or Fisher’s exact tests when appropriate. The concordance of screening at each time point compared with the traditional CST was calculated. Positive percent agreement (PPA) with the traditional CST in each subgroup (ICU and floor) and combined was also evaluated. A P value of .05 was used to determine significance.

RESULTS

A total of 702 complete cosyntropin tests on separate patients were included in the analysis. This included 198 ICU patients and 504 non-ICU (floor) patients. Fifty-one percent of patients were male in both the floor and ICU subgroups. The average age of ICU patients was 60.2 ± 13.2 years compared to 57.3 ± 17.3 years for patients on a general medicine floor (P = .02).

Cortisol concentrations obtained at 30 minutes were significantly higher than baseline cortisol concentrations (baseline: 12.8 mcg/dL; 30 minutes: 23.9 mcg/dL; P < .001) for all patients. The average cortisol concentrations obtained at 60 minutes (27.4 mcg/dL) were significantly higher than those at baseline and 30 minutes (P < .001). This trend was seen in each subgroup of patients in the ICU and on the floor (Figure). The average baseline cortisol concentration was higher for ICU patients compared to floor patients (17.6 mcg/dL vs 10.9 mcg/dL, respectively).

By using the traditional CST, there were 26 (13.1%) positive tests for AI in ICU patients and 84 (16.7%) positive tests in floor patients (Table).

The Table shows the number of patients who screened positive at each time point and compares the concordance of these results with the results of the overall CST in each subgroup (ICU and floor). The 60-minute concentration demonstrated higher concordance with the traditional CST than the 30-minute concentration overall (99.7% vs 88.0%, respectively), in ICU patients (100% vs 91.9%, respectively), and in floor patients (99.6% vs 86.9%, respectively). In the ICU subgroup, 60-minute concentrations were 100% concordant with the traditional CSTs. The PPA of a 60-minute–only screening compared to a traditional CST was better than a 30-minute–only screening overall (98% vs 57%, respectively), in ICU patients (100% vs 62%, respectively), and in floor patients (98% vs 56%, respectively). A 60-minute concentration was required to prevent false-positive screening in 11.7% of all screening tests, but the 30-minute concentration only prevented false-positive screening in 0.3% of screening tests. Of all 30-minute concentrations screening positive for AI alone, 42.7% were negative for AI at 60 minutes. Conversely, only 1.8% of all 60-minute concentrations screening positive for AI alone were negative for AI at 30 minutes. The likelihood of a false-positive screening test at 30 minutes was higher in floor patients (13.1%) than in ICU patients (8.1%). The difference between the false-positive screening rate of a single 30-minute cortisol concentration and a single 60-minute concentration was significant (P < .0001) for both floor and ICU patients. There were no instances of basal cortisol concentrations >18 mcg/dL that were subsequently <18 mcg/dL at 30 and 60 minutes after cosyntropin stimulation.

Only 13% of CSTs were started in the recommended 3-hour window from 6:00 am to 8:59 am. The remaining tests were begun outside this window.

DISCUSSION

Our investigation of 702 CSTs, the largest retrospective analysis to date, finds that the 60-minute cortisol concentration is significantly higher than the 30-minute concentration in a standard, high-dose CST. Sixty-minute cortisol concentrations are more concordant with traditional CST results than the 30-minute concentrations in both critically ill ICU and noncritically ill floor patients. This suggests that a single 60-minute measurement is sufficient for AI screening. The use of only 30-minute concentrations would lead to a significant increase in false-positive screening tests and significantly lower PPA (98% vs 57%). With peak cortisol concentrations occurring at 60-minutes poststimulation, measuring both 30- and 60-minute poststimulation concentrations does not appear to be of significant clinical benefit. The cost-saving from reduced phlebotomy and laboratory expenses would be significant, especially in locations with limited staff or financial resources. Our findings are similar to other recent results by Chitale et al.,¹⁷ Mansoor et al.,¹⁶ and Zueger et al.¹⁸

 

Zueger et al.¹⁸ evaluated the results of high-dose CST in 73 patients and found 13.7% of patients with inadequate cortisol response (<18 mcg/dL) at 30 minutes had normal concentrations at 60 minutes (>18 mcg/dL). Their study did not identify a single case of normal cortisol concentration at 30 minutes that would have inappropriately screened positive for AI if cortisol concentrations were only checked at 60 minutes. Similarly, they suggested that the 30-minute test did not add any additional diagnostic value; however, no confirmatory testing was performed.

Higher cortisol concentrations at 60 minutes poststimulation may result in improved specificity for AI without reducing sensitivity, but it may also indicate that the cutoff value may need to be raised from 18 mcg/dL at 60 minutes to maintain an appropriate clinical sensitivity. Continued research should resolve this clinical question with gold-standard confirmatory testing. Furthermore, there is debate about an appropriate screening cortisol concentration threshold for critically ill patients. Researchers have compared concentrations of 25 mcg/dL to the traditional 18 mcg/dL to improve sensitivity for AI, but these studies do not involve comparisons to confirmatory testing and often result in reduced specificity.^23,24

In our study, only a small fraction of testing was performed in the early-morning hours, when basal cortisol results are of value. There may be indications to perform traditional CSTs with a basal concentration, such as for suspected secondary AI, but testing must be performed in the early morning for interpretable results per current recommendations. However, poststimulation cortisol concentrations may be interpreted regardless of the time of day at which the test was initiated.³

Our study is limited by its scope because it is a retrospective analysis. It is also limited by a lack of gold-standard, clinical confirmatory testing or analysis of other clinical data. Our method of testing and interpretation is considered the screening standard and is often used to plan treatment for AI without confirmatory testing, as ITT is not routinely available for hospitalized patients. The validation of the traditional CST to the ITT has been performed extensively, but a randomized trial comparing a single 60-minute concentration to the ITT may be useful. The exact timing of blood draws may have introduced error in the concentration measurements, and this is critical to screening accuracy. Total serum cortisol is 10% bound to albumin,²⁵ and medications such as steroids or opioids and medical conditions such as obesity or liver disease can affect cortisol concentrations.²⁶ Albumin and free cortisol concentrations that may be used to adjust for these variables were not available.

CONCLUSION

We recommend changes to the standard CST to exclude a basal cortisol concentration unless it is indicated for the evaluation of secondary AI or obtained at the appropriate early-morning hour. A single 60-minute poststimulation cortisol concentration may be an appropriate screening test for AI and demonstrates high concordance with the traditional CST. The use of a 30-minute poststimulation concentration alone may lead to a significantly higher number of false-positive results. Alternatively, the stimulated cortisol threshold used to define a normal test may need to be higher at 60 minutes to maintain the appropriate sensitivity. Further study and comparison with confirmatory testing are needed.

Disclosure

The authors have no relevant conflicts of interest to disclose.

Testing for adrenal insufficiency (AI) is common in the hospital setting. The gold standard remains the insulin tolerance test (ITT), in which cortisol concentration is measured after the induction of hypoglycemia to <35 mg/dL.¹ Alternatively, metyrapone testing works by blocking cortisol synthesis. If pretest adrenocorticotropic hormone (ACTH) concentrations are low and ACTH concentrations do not rise after the administration of metyrapone, the patient is given a diagnosis of AI. Both assays pose some risk to patients with AI and are typically only performed as confirmatory tests. Morning random cortisol concentrations can be used to suggest AI if concentrations are <3 mcg/dL, but they often provide indeterminate results if concentrations are between 3 and 15 mcg/dL.² Thus, morning cortisol concentrations in isolation are not diagnostic of AI. For these reasons, most experts recommend a dynamic, high-dose cosyntropin stimulation testing (CST) with 250 mcg of intravenous cosyntropin to screen for AI. The test can be done any time of day.³ Historically, an incremental response to cosyntropin, or “delta,” was also required to indicate a normal response to stimulation.⁴ However, the baseline cortisol concentration is dependent on circadian rhythm and level of stress. For this reason, a delta, whether large or small, has been abandoned as a requisite for the diagnosis of AI.^5-7 A normal CST is widely accepted to be identified by any cortisol concentration >18 mcg/dL during the test (basal or poststimulation).⁸

The seminal studies by Lindholm, Kehlet, and coauthors^9-11 validated the CST against the gold standard ITT and utilized only 0- and 30-minute cortisol concentrations. A later study in patients with pituitary disease demonstrated that 30-minute concentrations had a stronger correlation with the ITT than 60-minute concentrations (false-negative rate: 10% vs 27%).¹² However, in that study, a higher threshold was used for the 60-minute concentration than for what was obtained at 30 minutes (25.4 vs 21.8 mcg/dL, respectively). Multiple studies have shown that the 60-minute concentration is higher than the 30-minute concentration after cosyntropin stimulation.^4,5,13 Subsequent, small studies of patients who were known to have AI have shown that 60-minute concentrations are as useful as 30-minute concentrations.^5,14,15 Because 30-minute cortisol concentrations are often lower than 60-minute concentrations, a single 30-minute result may lead to a falsely abnormal test.^16,17 As such, the use of a single 60-minute test may be more appropriate. Indeed, some authors have suggested that measuring only 30-minute concentrations may lead to overdiagnosis of AI by missing an appropriate response, serum cortisol >18 mcg/dL, at 60 minutes.^17-19 Peak cortisol concentrations after low-dose cosyntropin stimulation (1 mcg) are seen at 60 minutes, and low-dose stimulation has been shown to be more variable than in the high-dose test (250 mg).^19,20

There is a lack of consensus to guide clinicians as to when cortisol concentrations should be measured after stimulation, and standard references lack uniformity. Commonly accessed medical resources—such as UpToDate and Jameson’s Endocrinology—recommend basal, 30-minute, and 60-minute cortisol concentrations, while Williams Textbook of Endocrinology recommends basal and 30-minute concentrations, and the Washington Manual recommends only a single 30-minute concentration.^7,21,22 Goldman-Cecil Medicine⁸ recommends checking a cortisol concentration between 30 and 60 minutes and recommends the same 18 mcg/dL cutoff for any test obtained in this time period. As a result of these variable recommendations, all 3 time points are often obtained. Prominent review articles continue to recommend checking all 3 concentrations while presenting evidence of peak cortisol response at 60 minutes poststimulation.¹³

In this study, we retrospectively examined CSTs in hospitalized, adult patients both in the intensive care unit (ICU) and hospital ward and/or floor settings to evaluate for significant differences in 30- and 60-minute cortisol concentrations and compare the concordance of screening at each time point alone with traditional CST at all 3 time points. By using these results, we discuss the utility of obtaining 3 cortisol samples.

METHODS

After receiving approval from the institutional review board, we retrospectively reviewed all standard, high-dose CSTs performed on adult inpatients at the Barnes-Jewish Hospital laboratory from January 1, 2012, to August 31, 2013. All patients received the same standard dose (250 mcg cosyntropin, a synthetic ACTH, at a concentration of 1 mcg/mL administered over 2 minutes) regardless of age or weight. We collected patient gender; age; time of baseline cortisol measurement; cortisol results at baseline, 30, and 60 minutes; and patient location (inpatient floor vs ICU status). Tests were included if results from all 3 time points (0, 30, 60 minute) were available.

 

Cortisol concentrations were assessed by the laboratory according to the manufacturer’s instructions by using the ADVIA Centaur Cortisol assay (Siemens Healthcare Diagnostics Inc, Tarrytown, NY), a competitive chemiluminescent immunoassay. For the traditional CST, a cortisol concentration ≥18 mcg/dL at any time point during the test was used to define normal (negative). Patients with a positive (no results >18 mcg/mL) CST were defined as “screen positives” for the purposes of this analysis. Patient location data were available that allowed for an ICU vs non-ICU comparison.

Statistical analyses were performed in SAS version 9.4 (SAS Institute Inc, Cary, North Carolina). Continuous variables were compared by using a 2-tailed Student t test. Percentiles and proportions were compared by using χ² tests or Fisher’s exact tests when appropriate. The concordance of screening at each time point compared with the traditional CST was calculated. Positive percent agreement (PPA) with the traditional CST in each subgroup (ICU and floor) and combined was also evaluated. A P value of .05 was used to determine significance.

RESULTS

A total of 702 complete cosyntropin tests on separate patients were included in the analysis. This included 198 ICU patients and 504 non-ICU (floor) patients. Fifty-one percent of patients were male in both the floor and ICU subgroups. The average age of ICU patients was 60.2 ± 13.2 years compared to 57.3 ± 17.3 years for patients on a general medicine floor (P = .02).

Cortisol concentrations obtained at 30 minutes were significantly higher than baseline cortisol concentrations (baseline: 12.8 mcg/dL; 30 minutes: 23.9 mcg/dL; P < .001) for all patients. The average cortisol concentrations obtained at 60 minutes (27.4 mcg/dL) were significantly higher than those at baseline and 30 minutes (P < .001). This trend was seen in each subgroup of patients in the ICU and on the floor (Figure). The average baseline cortisol concentration was higher for ICU patients compared to floor patients (17.6 mcg/dL vs 10.9 mcg/dL, respectively).

By using the traditional CST, there were 26 (13.1%) positive tests for AI in ICU patients and 84 (16.7%) positive tests in floor patients (Table).

The Table shows the number of patients who screened positive at each time point and compares the concordance of these results with the results of the overall CST in each subgroup (ICU and floor). The 60-minute concentration demonstrated higher concordance with the traditional CST than the 30-minute concentration overall (99.7% vs 88.0%, respectively), in ICU patients (100% vs 91.9%, respectively), and in floor patients (99.6% vs 86.9%, respectively). In the ICU subgroup, 60-minute concentrations were 100% concordant with the traditional CSTs. The PPA of a 60-minute–only screening compared to a traditional CST was better than a 30-minute–only screening overall (98% vs 57%, respectively), in ICU patients (100% vs 62%, respectively), and in floor patients (98% vs 56%, respectively). A 60-minute concentration was required to prevent false-positive screening in 11.7% of all screening tests, but the 30-minute concentration only prevented false-positive screening in 0.3% of screening tests. Of all 30-minute concentrations screening positive for AI alone, 42.7% were negative for AI at 60 minutes. Conversely, only 1.8% of all 60-minute concentrations screening positive for AI alone were negative for AI at 30 minutes. The likelihood of a false-positive screening test at 30 minutes was higher in floor patients (13.1%) than in ICU patients (8.1%). The difference between the false-positive screening rate of a single 30-minute cortisol concentration and a single 60-minute concentration was significant (P < .0001) for both floor and ICU patients. There were no instances of basal cortisol concentrations >18 mcg/dL that were subsequently <18 mcg/dL at 30 and 60 minutes after cosyntropin stimulation.

Only 13% of CSTs were started in the recommended 3-hour window from 6:00 am to 8:59 am. The remaining tests were begun outside this window.

DISCUSSION

Our investigation of 702 CSTs, the largest retrospective analysis to date, finds that the 60-minute cortisol concentration is significantly higher than the 30-minute concentration in a standard, high-dose CST. Sixty-minute cortisol concentrations are more concordant with traditional CST results than the 30-minute concentrations in both critically ill ICU and noncritically ill floor patients. This suggests that a single 60-minute measurement is sufficient for AI screening. The use of only 30-minute concentrations would lead to a significant increase in false-positive screening tests and significantly lower PPA (98% vs 57%). With peak cortisol concentrations occurring at 60-minutes poststimulation, measuring both 30- and 60-minute poststimulation concentrations does not appear to be of significant clinical benefit. The cost-saving from reduced phlebotomy and laboratory expenses would be significant, especially in locations with limited staff or financial resources. Our findings are similar to other recent results by Chitale et al.,¹⁷ Mansoor et al.,¹⁶ and Zueger et al.¹⁸

 

Zueger et al.¹⁸ evaluated the results of high-dose CST in 73 patients and found 13.7% of patients with inadequate cortisol response (<18 mcg/dL) at 30 minutes had normal concentrations at 60 minutes (>18 mcg/dL). Their study did not identify a single case of normal cortisol concentration at 30 minutes that would have inappropriately screened positive for AI if cortisol concentrations were only checked at 60 minutes. Similarly, they suggested that the 30-minute test did not add any additional diagnostic value; however, no confirmatory testing was performed.

Higher cortisol concentrations at 60 minutes poststimulation may result in improved specificity for AI without reducing sensitivity, but it may also indicate that the cutoff value may need to be raised from 18 mcg/dL at 60 minutes to maintain an appropriate clinical sensitivity. Continued research should resolve this clinical question with gold-standard confirmatory testing. Furthermore, there is debate about an appropriate screening cortisol concentration threshold for critically ill patients. Researchers have compared concentrations of 25 mcg/dL to the traditional 18 mcg/dL to improve sensitivity for AI, but these studies do not involve comparisons to confirmatory testing and often result in reduced specificity.^23,24

In our study, only a small fraction of testing was performed in the early-morning hours, when basal cortisol results are of value. There may be indications to perform traditional CSTs with a basal concentration, such as for suspected secondary AI, but testing must be performed in the early morning for interpretable results per current recommendations. However, poststimulation cortisol concentrations may be interpreted regardless of the time of day at which the test was initiated.³

Our study is limited by its scope because it is a retrospective analysis. It is also limited by a lack of gold-standard, clinical confirmatory testing or analysis of other clinical data. Our method of testing and interpretation is considered the screening standard and is often used to plan treatment for AI without confirmatory testing, as ITT is not routinely available for hospitalized patients. The validation of the traditional CST to the ITT has been performed extensively, but a randomized trial comparing a single 60-minute concentration to the ITT may be useful. The exact timing of blood draws may have introduced error in the concentration measurements, and this is critical to screening accuracy. Total serum cortisol is 10% bound to albumin,²⁵ and medications such as steroids or opioids and medical conditions such as obesity or liver disease can affect cortisol concentrations.²⁶ Albumin and free cortisol concentrations that may be used to adjust for these variables were not available.

CONCLUSION

We recommend changes to the standard CST to exclude a basal cortisol concentration unless it is indicated for the evaluation of secondary AI or obtained at the appropriate early-morning hour. A single 60-minute poststimulation cortisol concentration may be an appropriate screening test for AI and demonstrates high concordance with the traditional CST. The use of a 30-minute poststimulation concentration alone may lead to a significantly higher number of false-positive results. Alternatively, the stimulated cortisol threshold used to define a normal test may need to be higher at 60 minutes to maintain the appropriate sensitivity. Further study and comparison with confirmatory testing are needed.

Disclosure

The authors have no relevant conflicts of interest to disclose.

Testing for adrenal insufficiency (AI) is common in the hospital setting. The gold standard remains the insulin tolerance test (ITT), in which cortisol concentration is measured after the induction of hypoglycemia to <35 mg/dL.¹ Alternatively, metyrapone testing works by blocking cortisol synthesis. If pretest adrenocorticotropic hormone (ACTH) concentrations are low and ACTH concentrations do not rise after the administration of metyrapone, the patient is given a diagnosis of AI. Both assays pose some risk to patients with AI and are typically only performed as confirmatory tests. Morning random cortisol concentrations can be used to suggest AI if concentrations are <3 mcg/dL, but they often provide indeterminate results if concentrations are between 3 and 15 mcg/dL.² Thus, morning cortisol concentrations in isolation are not diagnostic of AI. For these reasons, most experts recommend a dynamic, high-dose cosyntropin stimulation testing (CST) with 250 mcg of intravenous cosyntropin to screen for AI. The test can be done any time of day.³ Historically, an incremental response to cosyntropin, or “delta,” was also required to indicate a normal response to stimulation.⁴ However, the baseline cortisol concentration is dependent on circadian rhythm and level of stress. For this reason, a delta, whether large or small, has been abandoned as a requisite for the diagnosis of AI.^5-7 A normal CST is widely accepted to be identified by any cortisol concentration >18 mcg/dL during the test (basal or poststimulation).⁸

The seminal studies by Lindholm, Kehlet, and coauthors^9-11 validated the CST against the gold standard ITT and utilized only 0- and 30-minute cortisol concentrations. A later study in patients with pituitary disease demonstrated that 30-minute concentrations had a stronger correlation with the ITT than 60-minute concentrations (false-negative rate: 10% vs 27%).¹² However, in that study, a higher threshold was used for the 60-minute concentration than for what was obtained at 30 minutes (25.4 vs 21.8 mcg/dL, respectively). Multiple studies have shown that the 60-minute concentration is higher than the 30-minute concentration after cosyntropin stimulation.^4,5,13 Subsequent, small studies of patients who were known to have AI have shown that 60-minute concentrations are as useful as 30-minute concentrations.^5,14,15 Because 30-minute cortisol concentrations are often lower than 60-minute concentrations, a single 30-minute result may lead to a falsely abnormal test.^16,17 As such, the use of a single 60-minute test may be more appropriate. Indeed, some authors have suggested that measuring only 30-minute concentrations may lead to overdiagnosis of AI by missing an appropriate response, serum cortisol >18 mcg/dL, at 60 minutes.^17-19 Peak cortisol concentrations after low-dose cosyntropin stimulation (1 mcg) are seen at 60 minutes, and low-dose stimulation has been shown to be more variable than in the high-dose test (250 mg).^19,20

There is a lack of consensus to guide clinicians as to when cortisol concentrations should be measured after stimulation, and standard references lack uniformity. Commonly accessed medical resources—such as UpToDate and Jameson’s Endocrinology—recommend basal, 30-minute, and 60-minute cortisol concentrations, while Williams Textbook of Endocrinology recommends basal and 30-minute concentrations, and the Washington Manual recommends only a single 30-minute concentration.^7,21,22 Goldman-Cecil Medicine⁸ recommends checking a cortisol concentration between 30 and 60 minutes and recommends the same 18 mcg/dL cutoff for any test obtained in this time period. As a result of these variable recommendations, all 3 time points are often obtained. Prominent review articles continue to recommend checking all 3 concentrations while presenting evidence of peak cortisol response at 60 minutes poststimulation.¹³

In this study, we retrospectively examined CSTs in hospitalized, adult patients both in the intensive care unit (ICU) and hospital ward and/or floor settings to evaluate for significant differences in 30- and 60-minute cortisol concentrations and compare the concordance of screening at each time point alone with traditional CST at all 3 time points. By using these results, we discuss the utility of obtaining 3 cortisol samples.

METHODS

After receiving approval from the institutional review board, we retrospectively reviewed all standard, high-dose CSTs performed on adult inpatients at the Barnes-Jewish Hospital laboratory from January 1, 2012, to August 31, 2013. All patients received the same standard dose (250 mcg cosyntropin, a synthetic ACTH, at a concentration of 1 mcg/mL administered over 2 minutes) regardless of age or weight. We collected patient gender; age; time of baseline cortisol measurement; cortisol results at baseline, 30, and 60 minutes; and patient location (inpatient floor vs ICU status). Tests were included if results from all 3 time points (0, 30, 60 minute) were available.

 

Cortisol concentrations were assessed by the laboratory according to the manufacturer’s instructions by using the ADVIA Centaur Cortisol assay (Siemens Healthcare Diagnostics Inc, Tarrytown, NY), a competitive chemiluminescent immunoassay. For the traditional CST, a cortisol concentration ≥18 mcg/dL at any time point during the test was used to define normal (negative). Patients with a positive (no results >18 mcg/mL) CST were defined as “screen positives” for the purposes of this analysis. Patient location data were available that allowed for an ICU vs non-ICU comparison.

Statistical analyses were performed in SAS version 9.4 (SAS Institute Inc, Cary, North Carolina). Continuous variables were compared by using a 2-tailed Student t test. Percentiles and proportions were compared by using χ² tests or Fisher’s exact tests when appropriate. The concordance of screening at each time point compared with the traditional CST was calculated. Positive percent agreement (PPA) with the traditional CST in each subgroup (ICU and floor) and combined was also evaluated. A P value of .05 was used to determine significance.

RESULTS

A total of 702 complete cosyntropin tests on separate patients were included in the analysis. This included 198 ICU patients and 504 non-ICU (floor) patients. Fifty-one percent of patients were male in both the floor and ICU subgroups. The average age of ICU patients was 60.2 ± 13.2 years compared to 57.3 ± 17.3 years for patients on a general medicine floor (P = .02).

Cortisol concentrations obtained at 30 minutes were significantly higher than baseline cortisol concentrations (baseline: 12.8 mcg/dL; 30 minutes: 23.9 mcg/dL; P < .001) for all patients. The average cortisol concentrations obtained at 60 minutes (27.4 mcg/dL) were significantly higher than those at baseline and 30 minutes (P < .001). This trend was seen in each subgroup of patients in the ICU and on the floor (Figure). The average baseline cortisol concentration was higher for ICU patients compared to floor patients (17.6 mcg/dL vs 10.9 mcg/dL, respectively).

By using the traditional CST, there were 26 (13.1%) positive tests for AI in ICU patients and 84 (16.7%) positive tests in floor patients (Table).

The Table shows the number of patients who screened positive at each time point and compares the concordance of these results with the results of the overall CST in each subgroup (ICU and floor). The 60-minute concentration demonstrated higher concordance with the traditional CST than the 30-minute concentration overall (99.7% vs 88.0%, respectively), in ICU patients (100% vs 91.9%, respectively), and in floor patients (99.6% vs 86.9%, respectively). In the ICU subgroup, 60-minute concentrations were 100% concordant with the traditional CSTs. The PPA of a 60-minute–only screening compared to a traditional CST was better than a 30-minute–only screening overall (98% vs 57%, respectively), in ICU patients (100% vs 62%, respectively), and in floor patients (98% vs 56%, respectively). A 60-minute concentration was required to prevent false-positive screening in 11.7% of all screening tests, but the 30-minute concentration only prevented false-positive screening in 0.3% of screening tests. Of all 30-minute concentrations screening positive for AI alone, 42.7% were negative for AI at 60 minutes. Conversely, only 1.8% of all 60-minute concentrations screening positive for AI alone were negative for AI at 30 minutes. The likelihood of a false-positive screening test at 30 minutes was higher in floor patients (13.1%) than in ICU patients (8.1%). The difference between the false-positive screening rate of a single 30-minute cortisol concentration and a single 60-minute concentration was significant (P < .0001) for both floor and ICU patients. There were no instances of basal cortisol concentrations >18 mcg/dL that were subsequently <18 mcg/dL at 30 and 60 minutes after cosyntropin stimulation.

Only 13% of CSTs were started in the recommended 3-hour window from 6:00 am to 8:59 am. The remaining tests were begun outside this window.

DISCUSSION

Our investigation of 702 CSTs, the largest retrospective analysis to date, finds that the 60-minute cortisol concentration is significantly higher than the 30-minute concentration in a standard, high-dose CST. Sixty-minute cortisol concentrations are more concordant with traditional CST results than the 30-minute concentrations in both critically ill ICU and noncritically ill floor patients. This suggests that a single 60-minute measurement is sufficient for AI screening. The use of only 30-minute concentrations would lead to a significant increase in false-positive screening tests and significantly lower PPA (98% vs 57%). With peak cortisol concentrations occurring at 60-minutes poststimulation, measuring both 30- and 60-minute poststimulation concentrations does not appear to be of significant clinical benefit. The cost-saving from reduced phlebotomy and laboratory expenses would be significant, especially in locations with limited staff or financial resources. Our findings are similar to other recent results by Chitale et al.,¹⁷ Mansoor et al.,¹⁶ and Zueger et al.¹⁸

 

Zueger et al.¹⁸ evaluated the results of high-dose CST in 73 patients and found 13.7% of patients with inadequate cortisol response (<18 mcg/dL) at 30 minutes had normal concentrations at 60 minutes (>18 mcg/dL). Their study did not identify a single case of normal cortisol concentration at 30 minutes that would have inappropriately screened positive for AI if cortisol concentrations were only checked at 60 minutes. Similarly, they suggested that the 30-minute test did not add any additional diagnostic value; however, no confirmatory testing was performed.

Higher cortisol concentrations at 60 minutes poststimulation may result in improved specificity for AI without reducing sensitivity, but it may also indicate that the cutoff value may need to be raised from 18 mcg/dL at 60 minutes to maintain an appropriate clinical sensitivity. Continued research should resolve this clinical question with gold-standard confirmatory testing. Furthermore, there is debate about an appropriate screening cortisol concentration threshold for critically ill patients. Researchers have compared concentrations of 25 mcg/dL to the traditional 18 mcg/dL to improve sensitivity for AI, but these studies do not involve comparisons to confirmatory testing and often result in reduced specificity.^23,24

In our study, only a small fraction of testing was performed in the early-morning hours, when basal cortisol results are of value. There may be indications to perform traditional CSTs with a basal concentration, such as for suspected secondary AI, but testing must be performed in the early morning for interpretable results per current recommendations. However, poststimulation cortisol concentrations may be interpreted regardless of the time of day at which the test was initiated.³

Our study is limited by its scope because it is a retrospective analysis. It is also limited by a lack of gold-standard, clinical confirmatory testing or analysis of other clinical data. Our method of testing and interpretation is considered the screening standard and is often used to plan treatment for AI without confirmatory testing, as ITT is not routinely available for hospitalized patients. The validation of the traditional CST to the ITT has been performed extensively, but a randomized trial comparing a single 60-minute concentration to the ITT may be useful. The exact timing of blood draws may have introduced error in the concentration measurements, and this is critical to screening accuracy. Total serum cortisol is 10% bound to albumin,²⁵ and medications such as steroids or opioids and medical conditions such as obesity or liver disease can affect cortisol concentrations.²⁶ Albumin and free cortisol concentrations that may be used to adjust for these variables were not available.

CONCLUSION

We recommend changes to the standard CST to exclude a basal cortisol concentration unless it is indicated for the evaluation of secondary AI or obtained at the appropriate early-morning hour. A single 60-minute poststimulation cortisol concentration may be an appropriate screening test for AI and demonstrates high concordance with the traditional CST. The use of a 30-minute poststimulation concentration alone may lead to a significantly higher number of false-positive results. Alternatively, the stimulated cortisol threshold used to define a normal test may need to be higher at 60 minutes to maintain the appropriate sensitivity. Further study and comparison with confirmatory testing are needed.

Disclosure

The authors have no relevant conflicts of interest to disclose.

Wasteful care may account for between 21% and 34% of the United States’ $3.2 trillion in annual healthcare expenditures, making it a prime target for cost-saving initiatives.^1,2 Telemetry is a target for value improvement strategies because telemetry is overutilized, rarely leads to a change in management, and has associated guidelines on appropriate use.^3-10 Telemetry use has been a focus of the Joint Commission’s National Patient Safety Goals since 2014, and it is also a focus of the Society of Hospital Medicine’s Choosing Wisely^® campaign.^11-13

Previous initiatives have evaluated how changes to telemetry orders or education and feedback affect telemetry use. Few studies have compared a system-wide electronic health record (EHR) approach to a multifaceted intervention. In seeking to address this gap, we adapted published guidelines from the American Heart Association (AHA) and incorporated them into our EHR ordering process.³ Simultaneously, we implemented a multifaceted quality improvement initiative and compared this combined program’s effectiveness to that of the EHR approach alone.

METHODS

Study Design, Setting, and Population

We performed a 2-group observational pre- to postintervention study at University of Utah Health. Hospital encounters of patients 18 years and older who had at least 1 inpatient acute care, nonintensive care unit (ICU) room charge and an admission date between January 1, 2014, and July 31, 2016, were included. Patient encounters with missing encounter-level covariates, such as case mix index (CMI) or attending provider identification, were excluded. The Institutional Review Board classified this project as quality improvement and did not require review and oversight.

Intervention

On July 6, 2015, our Epic (Epic Systems Corporation, Madison, WI) EHR telemetry order was modified to discourage unnecessary telemetry monitoring. The new order required providers ordering telemetry to choose a clinical indication and select a duration for monitoring, after which the order would expire and require physician renewal or discontinuation. These were the only changes that occurred for nonhospitalist providers. The nonhospitalist group included all admitting providers who were not hospitalists. This group included neurology (6.98%); cardiology (8.13%); other medical specialties such as pulmonology, hematology, and oncology (21.30%); cardiothoracic surgery (3.72%); orthopedic surgery (14.84%); general surgery (11.11%); neurosurgery (11.07%); and other surgical specialties, including urology, transplant, vascular surgery, and plastics (16.68%).

Between January 2015 and June 2015, we implemented a multicomponent program among our hospitalist service. The hospitalist service is composed of 4 teams with internal medicine residents and 2 teams with advanced practice providers, all staffed by academic hospitalists. Our program was composed of 5 elements, all of which were made before the hospital-wide changes to electronic telemetry orders and maintained throughout the study period, as follows: (1) a single provider education session reviewing available evidence (eg, AHA guidelines, Choosing Wisely^® campaign), (2) removal of the telemetry order from hospitalist admission order set on March 23, 2015, (3) inclusion of telemetry discussion in the hospitalist group’s daily “Rounding Checklist,”¹⁴ (4) monthly feedback provided as part of hospitalist group meetings, and (5) a financial incentive, awarded to the division (no individual provider payment) if performance targets were met. See supplementary Appendix (“Implementation Manual”) for further details.

Data Source

We obtained data on patient age, gender, Medicare Severity-Diagnosis Related Group, Charlson comorbidity index (CCI), CMI, admitting unit, attending physician, admission and discharge dates, length of stay (LOS), 30-day readmission, bed charge (telemetry or nontelemetry), ICU stay, and inpatient mortality from the enterprise data warehouse. Telemetry days were determined through room billing charges, which are assigned based on the presence or absence of an active telemetry order at midnight. Code events came from a log kept by the hospital telephone operator, who is responsible for sending out all calls to the code team. Code event data were available starting July 19, 2014.

 

Measures

Our primary outcome was the percentage of hospital days that had telemetry charges for individual patients. All billed telemetry days on acute care floors were included regardless of admission status (inpatient vs observation), service, indication, or ordering provider. Secondary outcomes were inpatient mortality, escalation of care, code event rates, and appropriate telemetry utilization rates. Escalation of care was defined as transfer to an ICU after initially being admitted to an acute care floor. The code event rate was defined as the ratio of the number of code team activations to the number of patient days. Appropriate telemetry utilization rates were determined via chart review, as detailed below.

In order to evaluate changes in appropriateness of telemetry monitoring, 4 of the authors who are internal medicine physicians (KE, CC, JC, DG) performed chart reviews of 25 randomly selected patients in each group (hospitalist and nonhospitalist) before and after the intervention who received at least 1 day of telemetry monitoring. Each reviewer was provided a key based on AHA guidelines for monitoring indications and associated maximum allowable durations.³ Chart reviews were performed to determine the indication (if any) for monitoring, as well as the number of days that were indicated. The number of indicated days was compared to the number of telemetry days the patient received to determine the overall proportion of days that were indicated (“Telemetry appropriateness per visit”). Three reviewers (KE, AR, CC) also evaluated 100 patients on the hospitalist service after the intervention who did not receive any telemetry monitoring to evaluate whether patients with indications for telemetry monitoring were not receiving it after the intervention. For patients who had a possible indication, the indication was classified as Class I (“Cardiac monitoring is indicated in most, if not all, patients in this group”) or Class II (“Cardiac monitoring may be of benefit in some patients but is not considered essential for all patients”).³

Adjustment Variables

To account for differences in patient characteristics between hospitalist and nonhospitalist groups, we included age, gender, CMI, and CCI in statistical models. CCI was calculated according to the algorithm specified by Quan et al.¹⁵ using all patient diagnoses from previous visits and the index visit identified from the facility billing system.

Statistical Analysis

The period between January 1, 2014, and December 31, 2014, was considered preintervention, and August 1, 2015, to July 31, 2016, was considered postintervention. January 1, 2015, to July 31, 2015, was considered a “run-in” period because it was the interval during which the interventions on the hospitalist service were being rolled out. Data from this period were not included in the pre- or postintervention analyses but are shown in Figure 1.

We computed descriptive statistics for study outcomes and visit characteristics for hospitalist and nonhospitalist visits for pre- and postintervention periods. Descriptive statistics were expressed as n (%) for categorical patient characteristics and outcome variables. For continuous patient characteristics, we expressed the variability of individual observations as the mean ± the standard deviation. For continuous outcomes, we expressed the precision of the mean estimates using standard error. Telemetry utilization per visit was weighted by the number of total acute care days per visit. Telemetry appropriateness per visit was weighted by the number of telemetry days per visit. Patients who did not receive any telemetry monitoring were included in the analysis and noted to have 0 telemetry days. All patients had at least 1 acute care day. Categorical variables were compared using χ² tests, and continuous variables were compared using t tests. Code event rates were compared using the binomial probability mid-p exact test for person-time data.¹⁶

We fitted generalized linear regression models using generalized estimating equations to evaluate the relative change in outcomes of interest in the postintervention period compared with the preintervention period after adjusting for study covariates. The models included study group (hospitalist and nonhospitalist), time period (pre- and postintervention), an interaction term between study group and time period, and study covariates (age, gender, CMI, and CCI). The models were defined using a binomial distributional assumption and logit link function for mortality, escalation of care, and whether patients had at least 1 telemetry day. A gamma distributional assumption and log link function were used for LOS, telemetry acute care days per visit, and total acute care days per visit. A negative binomial distributional assumption and log link function were used for telemetry utilization and telemetry appropriateness. We used the log of the acute care days as an offset for telemetry utilization and the log of the telemetry days per visit as an offset for telemetry appropriateness. An exchangeable working correlation matrix was used to account for physician-level clustering for all outcomes. Intervention effects, representing the difference in odds for categorical variables and in amount for continuous variables, were calculated as exponentiation of the beta parameters for the covariate minus 1.

P values <.05 were considered significant. We used SAS version 9.4 statistical software (SAS Institute Inc., Cary, NC) for data analysis.

 

RESULTS

There were 46,215 visits originally included in the study. Ninety-two visits (0.2%) were excluded due to missing or invalid data. A total of 10,344 visits occurred during the “run-in” period between January 1, 2015, and July 31, 2015, leaving 35,871 patient visits during the pre- and postintervention periods. In the hospitalist group, there were 3442 visits before the intervention and 3700 after. There were 13,470 visits in the nonhospitalist group before the intervention and 15,259 after.

The percent of patients who had any telemetry charges decreased from 36.2% to 15.9% (P < .001) in the hospitalist group and from 31.8% to 28.0% in the nonhospitalist group (P < .001; Table 1). Rates of code events did not change over time (P = .9).

Estimates from adjusted and unadjusted linear models are shown in Table 2. In adjusted models, telemetry utilization in the postintervention period was reduced by 69% (95% confidence interval [CI], −72% to −64%; P < .001) in the hospitalist group and by 22% (95% CI, −27% to −16%; P <.001) in the nonhospitalist group. Compared with nonhospitalists, hospitalists had a 60% greater reduction in telemetry rates (95% CI, −65% to −54%; P < .001).

In the randomly selected sample of patients pre- and postintervention who received telemetry monitoring, there was an increase in telemetry appropriateness on the hospitalist service (46% to 72%, P = .025; Table 1). In the nonhospitalist group, appropriate telemetry utilization did not change significantly. Of the 100 randomly selected patients in the hospitalist group after the intervention who did not receive telemetry, no patient had an AHA Class I indication, and only 4 patients had a Class II indication.^3,17

DISCUSSION

In this study, implementing a change in the EHR telemetry order produced reductions in telemetry days. However, when combined with a multicomponent program including education, audit and feedback, financial incentives, and changes to remove telemetry orders from admission orders sets, an even more marked improvement was seen. Neither intervention reduced LOS, increased code event rates, or increased rates of escalation of care.

Prior studies have evaluated interventions to reduce unnecessary telemetry monitoring with varying degrees of success. The most successful EHR intervention to date, from Dressler et al.,¹⁸ achieved a 70% reduction in overall telemetry use by integrating the AHA guidelines into their EHR and incorporating nursing discontinuation guidelines to ensure that telemetry discontinuation was both safe and timely. Other studies using stewardship approaches and standardized protocols have been less successful.^19,20 One study utilizing a multidisciplinary approach but not including an EHR component showed modest improvements in telemetry.²¹

Although we are unable to differentiate the exact effect of each component of the intervention, we did note an immediate decrease in telemetry orders after removing the telemetry order from our admission order set, a trend that was magnified after the addition of broader EHR changes (Figure 1). Important additional contributors to our success seem to have been the standardization of rounds to include daily discussion of telemetry and the provision of routine feedback. We cannot discern whether other components of our program (such as the financial incentives) contributed more or less to our program, though the sum of these interventions produced an overall program that required substantial buy in and sustained focus from the hospitalist group. The importance of the hospitalist program is highlighted by the relatively large differences in improvement compared with the nonhospitalist group.

Our study has several limitations. First, the study was conducted at a single center, which may limit its generalizability. Second, the intervention was multifaceted, diminishing our ability to discern which aspects beyond the system-wide change in the telemetry order were most responsible for the observed effect among hospitalists. Third, we are unable to fully account for baseline differences in telemetry utilization between hospitalist and nonhospitalist groups. It is likely that different services utilize telemetry monitoring in different ways, and the hospitalist group may have been more aware of the existing guidelines for monitoring prior to the intervention. Furthermore, we had a limited sample size for the chart audits, which reduced the available statistical power for determining changes in the appropriateness of telemetry utilization. Additionally, because internal medicine residents rotate through various services, it is possible that the education they received on their hospitalist rotation as part of our intervention had a spillover effect in the nonhospitalist group. However, any effect should have decreased the difference between the groups. Lastly, although our postintervention time period was 1 year, we do not have data beyond that to monitor for sustainability of the results.

 

CONCLUSION

In this single-site study, combining EHR orders prompting physicians to choose a clinical indication and duration for monitoring with a broader program—including upstream changes in ordering as well as education, audit, and feedback—produced reductions in telemetry usage. Whether this reduction improves the appropriateness of telemetry utilization or reduces other effects of telemetry (eg, alert fatigue, calls for benign arrhythmias) cannot be discerned from our study. However, our results support the idea that multipronged approaches to telemetry use are most likely to produce improvements.

Acknowledgments

The authors thank Dr. Frank Thomas for his assistance with process engineering and Mr. Andrew Wood for his routine provision of data. The statistical analysis was supported by the University of Utah Study Design and Biostatistics Center, with funding in part from the National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health, through Grant 5UL1TR001067-05 (formerly 8UL1TR000105 and UL1RR025764).

Disclosure

The authors have no conflicts of interest to report.

Wasteful care may account for between 21% and 34% of the United States’ $3.2 trillion in annual healthcare expenditures, making it a prime target for cost-saving initiatives.^1,2 Telemetry is a target for value improvement strategies because telemetry is overutilized, rarely leads to a change in management, and has associated guidelines on appropriate use.^3-10 Telemetry use has been a focus of the Joint Commission’s National Patient Safety Goals since 2014, and it is also a focus of the Society of Hospital Medicine’s Choosing Wisely^® campaign.^11-13

Previous initiatives have evaluated how changes to telemetry orders or education and feedback affect telemetry use. Few studies have compared a system-wide electronic health record (EHR) approach to a multifaceted intervention. In seeking to address this gap, we adapted published guidelines from the American Heart Association (AHA) and incorporated them into our EHR ordering process.³ Simultaneously, we implemented a multifaceted quality improvement initiative and compared this combined program’s effectiveness to that of the EHR approach alone.

METHODS

Study Design, Setting, and Population

We performed a 2-group observational pre- to postintervention study at University of Utah Health. Hospital encounters of patients 18 years and older who had at least 1 inpatient acute care, nonintensive care unit (ICU) room charge and an admission date between January 1, 2014, and July 31, 2016, were included. Patient encounters with missing encounter-level covariates, such as case mix index (CMI) or attending provider identification, were excluded. The Institutional Review Board classified this project as quality improvement and did not require review and oversight.

Intervention

On July 6, 2015, our Epic (Epic Systems Corporation, Madison, WI) EHR telemetry order was modified to discourage unnecessary telemetry monitoring. The new order required providers ordering telemetry to choose a clinical indication and select a duration for monitoring, after which the order would expire and require physician renewal or discontinuation. These were the only changes that occurred for nonhospitalist providers. The nonhospitalist group included all admitting providers who were not hospitalists. This group included neurology (6.98%); cardiology (8.13%); other medical specialties such as pulmonology, hematology, and oncology (21.30%); cardiothoracic surgery (3.72%); orthopedic surgery (14.84%); general surgery (11.11%); neurosurgery (11.07%); and other surgical specialties, including urology, transplant, vascular surgery, and plastics (16.68%).

Between January 2015 and June 2015, we implemented a multicomponent program among our hospitalist service. The hospitalist service is composed of 4 teams with internal medicine residents and 2 teams with advanced practice providers, all staffed by academic hospitalists. Our program was composed of 5 elements, all of which were made before the hospital-wide changes to electronic telemetry orders and maintained throughout the study period, as follows: (1) a single provider education session reviewing available evidence (eg, AHA guidelines, Choosing Wisely^® campaign), (2) removal of the telemetry order from hospitalist admission order set on March 23, 2015, (3) inclusion of telemetry discussion in the hospitalist group’s daily “Rounding Checklist,”¹⁴ (4) monthly feedback provided as part of hospitalist group meetings, and (5) a financial incentive, awarded to the division (no individual provider payment) if performance targets were met. See supplementary Appendix (“Implementation Manual”) for further details.

Data Source

We obtained data on patient age, gender, Medicare Severity-Diagnosis Related Group, Charlson comorbidity index (CCI), CMI, admitting unit, attending physician, admission and discharge dates, length of stay (LOS), 30-day readmission, bed charge (telemetry or nontelemetry), ICU stay, and inpatient mortality from the enterprise data warehouse. Telemetry days were determined through room billing charges, which are assigned based on the presence or absence of an active telemetry order at midnight. Code events came from a log kept by the hospital telephone operator, who is responsible for sending out all calls to the code team. Code event data were available starting July 19, 2014.

 

Measures

Our primary outcome was the percentage of hospital days that had telemetry charges for individual patients. All billed telemetry days on acute care floors were included regardless of admission status (inpatient vs observation), service, indication, or ordering provider. Secondary outcomes were inpatient mortality, escalation of care, code event rates, and appropriate telemetry utilization rates. Escalation of care was defined as transfer to an ICU after initially being admitted to an acute care floor. The code event rate was defined as the ratio of the number of code team activations to the number of patient days. Appropriate telemetry utilization rates were determined via chart review, as detailed below.

In order to evaluate changes in appropriateness of telemetry monitoring, 4 of the authors who are internal medicine physicians (KE, CC, JC, DG) performed chart reviews of 25 randomly selected patients in each group (hospitalist and nonhospitalist) before and after the intervention who received at least 1 day of telemetry monitoring. Each reviewer was provided a key based on AHA guidelines for monitoring indications and associated maximum allowable durations.³ Chart reviews were performed to determine the indication (if any) for monitoring, as well as the number of days that were indicated. The number of indicated days was compared to the number of telemetry days the patient received to determine the overall proportion of days that were indicated (“Telemetry appropriateness per visit”). Three reviewers (KE, AR, CC) also evaluated 100 patients on the hospitalist service after the intervention who did not receive any telemetry monitoring to evaluate whether patients with indications for telemetry monitoring were not receiving it after the intervention. For patients who had a possible indication, the indication was classified as Class I (“Cardiac monitoring is indicated in most, if not all, patients in this group”) or Class II (“Cardiac monitoring may be of benefit in some patients but is not considered essential for all patients”).³

Adjustment Variables

To account for differences in patient characteristics between hospitalist and nonhospitalist groups, we included age, gender, CMI, and CCI in statistical models. CCI was calculated according to the algorithm specified by Quan et al.¹⁵ using all patient diagnoses from previous visits and the index visit identified from the facility billing system.

Statistical Analysis

The period between January 1, 2014, and December 31, 2014, was considered preintervention, and August 1, 2015, to July 31, 2016, was considered postintervention. January 1, 2015, to July 31, 2015, was considered a “run-in” period because it was the interval during which the interventions on the hospitalist service were being rolled out. Data from this period were not included in the pre- or postintervention analyses but are shown in Figure 1.

We computed descriptive statistics for study outcomes and visit characteristics for hospitalist and nonhospitalist visits for pre- and postintervention periods. Descriptive statistics were expressed as n (%) for categorical patient characteristics and outcome variables. For continuous patient characteristics, we expressed the variability of individual observations as the mean ± the standard deviation. For continuous outcomes, we expressed the precision of the mean estimates using standard error. Telemetry utilization per visit was weighted by the number of total acute care days per visit. Telemetry appropriateness per visit was weighted by the number of telemetry days per visit. Patients who did not receive any telemetry monitoring were included in the analysis and noted to have 0 telemetry days. All patients had at least 1 acute care day. Categorical variables were compared using χ² tests, and continuous variables were compared using t tests. Code event rates were compared using the binomial probability mid-p exact test for person-time data.¹⁶

We fitted generalized linear regression models using generalized estimating equations to evaluate the relative change in outcomes of interest in the postintervention period compared with the preintervention period after adjusting for study covariates. The models included study group (hospitalist and nonhospitalist), time period (pre- and postintervention), an interaction term between study group and time period, and study covariates (age, gender, CMI, and CCI). The models were defined using a binomial distributional assumption and logit link function for mortality, escalation of care, and whether patients had at least 1 telemetry day. A gamma distributional assumption and log link function were used for LOS, telemetry acute care days per visit, and total acute care days per visit. A negative binomial distributional assumption and log link function were used for telemetry utilization and telemetry appropriateness. We used the log of the acute care days as an offset for telemetry utilization and the log of the telemetry days per visit as an offset for telemetry appropriateness. An exchangeable working correlation matrix was used to account for physician-level clustering for all outcomes. Intervention effects, representing the difference in odds for categorical variables and in amount for continuous variables, were calculated as exponentiation of the beta parameters for the covariate minus 1.

P values <.05 were considered significant. We used SAS version 9.4 statistical software (SAS Institute Inc., Cary, NC) for data analysis.

 

RESULTS

There were 46,215 visits originally included in the study. Ninety-two visits (0.2%) were excluded due to missing or invalid data. A total of 10,344 visits occurred during the “run-in” period between January 1, 2015, and July 31, 2015, leaving 35,871 patient visits during the pre- and postintervention periods. In the hospitalist group, there were 3442 visits before the intervention and 3700 after. There were 13,470 visits in the nonhospitalist group before the intervention and 15,259 after.

The percent of patients who had any telemetry charges decreased from 36.2% to 15.9% (P < .001) in the hospitalist group and from 31.8% to 28.0% in the nonhospitalist group (P < .001; Table 1). Rates of code events did not change over time (P = .9).

Estimates from adjusted and unadjusted linear models are shown in Table 2. In adjusted models, telemetry utilization in the postintervention period was reduced by 69% (95% confidence interval [CI], −72% to −64%; P < .001) in the hospitalist group and by 22% (95% CI, −27% to −16%; P <.001) in the nonhospitalist group. Compared with nonhospitalists, hospitalists had a 60% greater reduction in telemetry rates (95% CI, −65% to −54%; P < .001).

In the randomly selected sample of patients pre- and postintervention who received telemetry monitoring, there was an increase in telemetry appropriateness on the hospitalist service (46% to 72%, P = .025; Table 1). In the nonhospitalist group, appropriate telemetry utilization did not change significantly. Of the 100 randomly selected patients in the hospitalist group after the intervention who did not receive telemetry, no patient had an AHA Class I indication, and only 4 patients had a Class II indication.^3,17

DISCUSSION

In this study, implementing a change in the EHR telemetry order produced reductions in telemetry days. However, when combined with a multicomponent program including education, audit and feedback, financial incentives, and changes to remove telemetry orders from admission orders sets, an even more marked improvement was seen. Neither intervention reduced LOS, increased code event rates, or increased rates of escalation of care.

Prior studies have evaluated interventions to reduce unnecessary telemetry monitoring with varying degrees of success. The most successful EHR intervention to date, from Dressler et al.,¹⁸ achieved a 70% reduction in overall telemetry use by integrating the AHA guidelines into their EHR and incorporating nursing discontinuation guidelines to ensure that telemetry discontinuation was both safe and timely. Other studies using stewardship approaches and standardized protocols have been less successful.^19,20 One study utilizing a multidisciplinary approach but not including an EHR component showed modest improvements in telemetry.²¹

Although we are unable to differentiate the exact effect of each component of the intervention, we did note an immediate decrease in telemetry orders after removing the telemetry order from our admission order set, a trend that was magnified after the addition of broader EHR changes (Figure 1). Important additional contributors to our success seem to have been the standardization of rounds to include daily discussion of telemetry and the provision of routine feedback. We cannot discern whether other components of our program (such as the financial incentives) contributed more or less to our program, though the sum of these interventions produced an overall program that required substantial buy in and sustained focus from the hospitalist group. The importance of the hospitalist program is highlighted by the relatively large differences in improvement compared with the nonhospitalist group.

Our study has several limitations. First, the study was conducted at a single center, which may limit its generalizability. Second, the intervention was multifaceted, diminishing our ability to discern which aspects beyond the system-wide change in the telemetry order were most responsible for the observed effect among hospitalists. Third, we are unable to fully account for baseline differences in telemetry utilization between hospitalist and nonhospitalist groups. It is likely that different services utilize telemetry monitoring in different ways, and the hospitalist group may have been more aware of the existing guidelines for monitoring prior to the intervention. Furthermore, we had a limited sample size for the chart audits, which reduced the available statistical power for determining changes in the appropriateness of telemetry utilization. Additionally, because internal medicine residents rotate through various services, it is possible that the education they received on their hospitalist rotation as part of our intervention had a spillover effect in the nonhospitalist group. However, any effect should have decreased the difference between the groups. Lastly, although our postintervention time period was 1 year, we do not have data beyond that to monitor for sustainability of the results.

 

CONCLUSION

In this single-site study, combining EHR orders prompting physicians to choose a clinical indication and duration for monitoring with a broader program—including upstream changes in ordering as well as education, audit, and feedback—produced reductions in telemetry usage. Whether this reduction improves the appropriateness of telemetry utilization or reduces other effects of telemetry (eg, alert fatigue, calls for benign arrhythmias) cannot be discerned from our study. However, our results support the idea that multipronged approaches to telemetry use are most likely to produce improvements.

Acknowledgments

The authors thank Dr. Frank Thomas for his assistance with process engineering and Mr. Andrew Wood for his routine provision of data. The statistical analysis was supported by the University of Utah Study Design and Biostatistics Center, with funding in part from the National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health, through Grant 5UL1TR001067-05 (formerly 8UL1TR000105 and UL1RR025764).

Disclosure

The authors have no conflicts of interest to report.

Wasteful care may account for between 21% and 34% of the United States’ $3.2 trillion in annual healthcare expenditures, making it a prime target for cost-saving initiatives.^1,2 Telemetry is a target for value improvement strategies because telemetry is overutilized, rarely leads to a change in management, and has associated guidelines on appropriate use.^3-10 Telemetry use has been a focus of the Joint Commission’s National Patient Safety Goals since 2014, and it is also a focus of the Society of Hospital Medicine’s Choosing Wisely^® campaign.^11-13

Previous initiatives have evaluated how changes to telemetry orders or education and feedback affect telemetry use. Few studies have compared a system-wide electronic health record (EHR) approach to a multifaceted intervention. In seeking to address this gap, we adapted published guidelines from the American Heart Association (AHA) and incorporated them into our EHR ordering process.³ Simultaneously, we implemented a multifaceted quality improvement initiative and compared this combined program’s effectiveness to that of the EHR approach alone.

METHODS

Study Design, Setting, and Population

We performed a 2-group observational pre- to postintervention study at University of Utah Health. Hospital encounters of patients 18 years and older who had at least 1 inpatient acute care, nonintensive care unit (ICU) room charge and an admission date between January 1, 2014, and July 31, 2016, were included. Patient encounters with missing encounter-level covariates, such as case mix index (CMI) or attending provider identification, were excluded. The Institutional Review Board classified this project as quality improvement and did not require review and oversight.

Intervention

On July 6, 2015, our Epic (Epic Systems Corporation, Madison, WI) EHR telemetry order was modified to discourage unnecessary telemetry monitoring. The new order required providers ordering telemetry to choose a clinical indication and select a duration for monitoring, after which the order would expire and require physician renewal or discontinuation. These were the only changes that occurred for nonhospitalist providers. The nonhospitalist group included all admitting providers who were not hospitalists. This group included neurology (6.98%); cardiology (8.13%); other medical specialties such as pulmonology, hematology, and oncology (21.30%); cardiothoracic surgery (3.72%); orthopedic surgery (14.84%); general surgery (11.11%); neurosurgery (11.07%); and other surgical specialties, including urology, transplant, vascular surgery, and plastics (16.68%).

Between January 2015 and June 2015, we implemented a multicomponent program among our hospitalist service. The hospitalist service is composed of 4 teams with internal medicine residents and 2 teams with advanced practice providers, all staffed by academic hospitalists. Our program was composed of 5 elements, all of which were made before the hospital-wide changes to electronic telemetry orders and maintained throughout the study period, as follows: (1) a single provider education session reviewing available evidence (eg, AHA guidelines, Choosing Wisely^® campaign), (2) removal of the telemetry order from hospitalist admission order set on March 23, 2015, (3) inclusion of telemetry discussion in the hospitalist group’s daily “Rounding Checklist,”¹⁴ (4) monthly feedback provided as part of hospitalist group meetings, and (5) a financial incentive, awarded to the division (no individual provider payment) if performance targets were met. See supplementary Appendix (“Implementation Manual”) for further details.

Data Source

We obtained data on patient age, gender, Medicare Severity-Diagnosis Related Group, Charlson comorbidity index (CCI), CMI, admitting unit, attending physician, admission and discharge dates, length of stay (LOS), 30-day readmission, bed charge (telemetry or nontelemetry), ICU stay, and inpatient mortality from the enterprise data warehouse. Telemetry days were determined through room billing charges, which are assigned based on the presence or absence of an active telemetry order at midnight. Code events came from a log kept by the hospital telephone operator, who is responsible for sending out all calls to the code team. Code event data were available starting July 19, 2014.

 

Measures

Our primary outcome was the percentage of hospital days that had telemetry charges for individual patients. All billed telemetry days on acute care floors were included regardless of admission status (inpatient vs observation), service, indication, or ordering provider. Secondary outcomes were inpatient mortality, escalation of care, code event rates, and appropriate telemetry utilization rates. Escalation of care was defined as transfer to an ICU after initially being admitted to an acute care floor. The code event rate was defined as the ratio of the number of code team activations to the number of patient days. Appropriate telemetry utilization rates were determined via chart review, as detailed below.

In order to evaluate changes in appropriateness of telemetry monitoring, 4 of the authors who are internal medicine physicians (KE, CC, JC, DG) performed chart reviews of 25 randomly selected patients in each group (hospitalist and nonhospitalist) before and after the intervention who received at least 1 day of telemetry monitoring. Each reviewer was provided a key based on AHA guidelines for monitoring indications and associated maximum allowable durations.³ Chart reviews were performed to determine the indication (if any) for monitoring, as well as the number of days that were indicated. The number of indicated days was compared to the number of telemetry days the patient received to determine the overall proportion of days that were indicated (“Telemetry appropriateness per visit”). Three reviewers (KE, AR, CC) also evaluated 100 patients on the hospitalist service after the intervention who did not receive any telemetry monitoring to evaluate whether patients with indications for telemetry monitoring were not receiving it after the intervention. For patients who had a possible indication, the indication was classified as Class I (“Cardiac monitoring is indicated in most, if not all, patients in this group”) or Class II (“Cardiac monitoring may be of benefit in some patients but is not considered essential for all patients”).³

Adjustment Variables

To account for differences in patient characteristics between hospitalist and nonhospitalist groups, we included age, gender, CMI, and CCI in statistical models. CCI was calculated according to the algorithm specified by Quan et al.¹⁵ using all patient diagnoses from previous visits and the index visit identified from the facility billing system.

Statistical Analysis

The period between January 1, 2014, and December 31, 2014, was considered preintervention, and August 1, 2015, to July 31, 2016, was considered postintervention. January 1, 2015, to July 31, 2015, was considered a “run-in” period because it was the interval during which the interventions on the hospitalist service were being rolled out. Data from this period were not included in the pre- or postintervention analyses but are shown in Figure 1.

We computed descriptive statistics for study outcomes and visit characteristics for hospitalist and nonhospitalist visits for pre- and postintervention periods. Descriptive statistics were expressed as n (%) for categorical patient characteristics and outcome variables. For continuous patient characteristics, we expressed the variability of individual observations as the mean ± the standard deviation. For continuous outcomes, we expressed the precision of the mean estimates using standard error. Telemetry utilization per visit was weighted by the number of total acute care days per visit. Telemetry appropriateness per visit was weighted by the number of telemetry days per visit. Patients who did not receive any telemetry monitoring were included in the analysis and noted to have 0 telemetry days. All patients had at least 1 acute care day. Categorical variables were compared using χ² tests, and continuous variables were compared using t tests. Code event rates were compared using the binomial probability mid-p exact test for person-time data.¹⁶

We fitted generalized linear regression models using generalized estimating equations to evaluate the relative change in outcomes of interest in the postintervention period compared with the preintervention period after adjusting for study covariates. The models included study group (hospitalist and nonhospitalist), time period (pre- and postintervention), an interaction term between study group and time period, and study covariates (age, gender, CMI, and CCI). The models were defined using a binomial distributional assumption and logit link function for mortality, escalation of care, and whether patients had at least 1 telemetry day. A gamma distributional assumption and log link function were used for LOS, telemetry acute care days per visit, and total acute care days per visit. A negative binomial distributional assumption and log link function were used for telemetry utilization and telemetry appropriateness. We used the log of the acute care days as an offset for telemetry utilization and the log of the telemetry days per visit as an offset for telemetry appropriateness. An exchangeable working correlation matrix was used to account for physician-level clustering for all outcomes. Intervention effects, representing the difference in odds for categorical variables and in amount for continuous variables, were calculated as exponentiation of the beta parameters for the covariate minus 1.

P values <.05 were considered significant. We used SAS version 9.4 statistical software (SAS Institute Inc., Cary, NC) for data analysis.

 

RESULTS

There were 46,215 visits originally included in the study. Ninety-two visits (0.2%) were excluded due to missing or invalid data. A total of 10,344 visits occurred during the “run-in” period between January 1, 2015, and July 31, 2015, leaving 35,871 patient visits during the pre- and postintervention periods. In the hospitalist group, there were 3442 visits before the intervention and 3700 after. There were 13,470 visits in the nonhospitalist group before the intervention and 15,259 after.

The percent of patients who had any telemetry charges decreased from 36.2% to 15.9% (P < .001) in the hospitalist group and from 31.8% to 28.0% in the nonhospitalist group (P < .001; Table 1). Rates of code events did not change over time (P = .9).

Estimates from adjusted and unadjusted linear models are shown in Table 2. In adjusted models, telemetry utilization in the postintervention period was reduced by 69% (95% confidence interval [CI], −72% to −64%; P < .001) in the hospitalist group and by 22% (95% CI, −27% to −16%; P <.001) in the nonhospitalist group. Compared with nonhospitalists, hospitalists had a 60% greater reduction in telemetry rates (95% CI, −65% to −54%; P < .001).

In the randomly selected sample of patients pre- and postintervention who received telemetry monitoring, there was an increase in telemetry appropriateness on the hospitalist service (46% to 72%, P = .025; Table 1). In the nonhospitalist group, appropriate telemetry utilization did not change significantly. Of the 100 randomly selected patients in the hospitalist group after the intervention who did not receive telemetry, no patient had an AHA Class I indication, and only 4 patients had a Class II indication.^3,17

DISCUSSION

In this study, implementing a change in the EHR telemetry order produced reductions in telemetry days. However, when combined with a multicomponent program including education, audit and feedback, financial incentives, and changes to remove telemetry orders from admission orders sets, an even more marked improvement was seen. Neither intervention reduced LOS, increased code event rates, or increased rates of escalation of care.

Prior studies have evaluated interventions to reduce unnecessary telemetry monitoring with varying degrees of success. The most successful EHR intervention to date, from Dressler et al.,¹⁸ achieved a 70% reduction in overall telemetry use by integrating the AHA guidelines into their EHR and incorporating nursing discontinuation guidelines to ensure that telemetry discontinuation was both safe and timely. Other studies using stewardship approaches and standardized protocols have been less successful.^19,20 One study utilizing a multidisciplinary approach but not including an EHR component showed modest improvements in telemetry.²¹

Although we are unable to differentiate the exact effect of each component of the intervention, we did note an immediate decrease in telemetry orders after removing the telemetry order from our admission order set, a trend that was magnified after the addition of broader EHR changes (Figure 1). Important additional contributors to our success seem to have been the standardization of rounds to include daily discussion of telemetry and the provision of routine feedback. We cannot discern whether other components of our program (such as the financial incentives) contributed more or less to our program, though the sum of these interventions produced an overall program that required substantial buy in and sustained focus from the hospitalist group. The importance of the hospitalist program is highlighted by the relatively large differences in improvement compared with the nonhospitalist group.

Our study has several limitations. First, the study was conducted at a single center, which may limit its generalizability. Second, the intervention was multifaceted, diminishing our ability to discern which aspects beyond the system-wide change in the telemetry order were most responsible for the observed effect among hospitalists. Third, we are unable to fully account for baseline differences in telemetry utilization between hospitalist and nonhospitalist groups. It is likely that different services utilize telemetry monitoring in different ways, and the hospitalist group may have been more aware of the existing guidelines for monitoring prior to the intervention. Furthermore, we had a limited sample size for the chart audits, which reduced the available statistical power for determining changes in the appropriateness of telemetry utilization. Additionally, because internal medicine residents rotate through various services, it is possible that the education they received on their hospitalist rotation as part of our intervention had a spillover effect in the nonhospitalist group. However, any effect should have decreased the difference between the groups. Lastly, although our postintervention time period was 1 year, we do not have data beyond that to monitor for sustainability of the results.

 

CONCLUSION

In this single-site study, combining EHR orders prompting physicians to choose a clinical indication and duration for monitoring with a broader program—including upstream changes in ordering as well as education, audit, and feedback—produced reductions in telemetry usage. Whether this reduction improves the appropriateness of telemetry utilization or reduces other effects of telemetry (eg, alert fatigue, calls for benign arrhythmias) cannot be discerned from our study. However, our results support the idea that multipronged approaches to telemetry use are most likely to produce improvements.

Acknowledgments

The authors thank Dr. Frank Thomas for his assistance with process engineering and Mr. Andrew Wood for his routine provision of data. The statistical analysis was supported by the University of Utah Study Design and Biostatistics Center, with funding in part from the National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health, through Grant 5UL1TR001067-05 (formerly 8UL1TR000105 and UL1RR025764).

Disclosure

The authors have no conflicts of interest to report.

Given the high cost of readmissions to the healthcare system, there has been a substantial push to reduce readmissions by policymakers.¹ Among these is the Hospital Readmissions Reduction Program (HRRP), in which hospitals with higher than expected readmission rates receive reduced payments from Medicare.² Recent evidence has suggested the success of such policy changes, with multiple reports demonstrating a decrease in 30-day readmission rates in the Medicare population starting in 2010.^3-8

Initiatives to reduce readmissions can also have an effect on total number of admissions.^9,10 Indeed, along with the recent reduction in readmission, there has been a reduction in all admissions among Medicare beneficiaries.^11,12 Some studies have found that as admissions have decreased, the burden of comorbidity has increased among hospitalized patients,^3,11 suggesting that hospitals may be increasingly filled with patients at high risk of readmission. However, whether readmission risk among hospitalized patients has changed remains unknown, and understanding changes in risk profile could help inform which patients to target with future interventions to reduce readmissions.

Hospital efforts to reduce readmissions may have differential effects on types of patients by risk. For instance, low-intensity, system-wide interventions such as standardized discharge instructions or medicine reconciliation may have a stronger effect on patients at relatively low risk of readmission who may have a few important drivers of readmission that are easily overcome. Alternatively, the impact of intensive care transitions management might be greatest for high-risk patients, who have the most need for postdischarge medications, follow-up, and self-care.

The purpose of this study was therefore twofold: (1) to observe changes in average monthly risk of readmission among hospitalized Medicare patients and (2) to examine changes in readmission rates for Medicare patients at various risk of readmission. We hypothesized that readmission risk in the Medicare population would increase in recent years, as overall number of admissions and readmissions have fallen.^7,11 Additionally, we hypothesized that standardized readmission rates would decline less in highest risk patients as compared with the lowest risk patients because transitional care interventions may not be able to mitigate the large burden of comorbidity and social issues present in many high-risk patients.^13,14

METHODS

We performed a retrospective cohort study of hospitalizations to US nonfederal short-term acute care facilities by Medicare beneficiaries between January 2009 and June 2015. The design involved 4 steps. First, we estimated a predictive model for unplanned readmissions within 30 days of discharge. Second, we assigned each hospitalization a predicted risk of readmission based on the model. Third, we studied trends in mean predicted risk of readmission during the study period. Fourth, we examined trends in observed to expected (O/E) readmission for hospitalizations in the lowest, middle, and highest categories of predicted risk of readmission to determine whether reductions in readmissions were more substantial in certain risk groups than in others.

Data were obtained from the Centers for Medicare and Medicaid Services (CMS) Inpatient Standard Analytic File and the Medicare Enrollment Data Base. We included hospitalizations of fee-for-service Medicare beneficiaries age ≥65 with continuous enrollment in Part A Medicare fee-for-service for at least 1 year prior and 30 days after the hospitalization.¹⁵ Hospitalizations with a discharge disposition of death, transfer to another acute hospital, and left against medical advice (AMA) were excluded. We also excluded patients with enrollment in hospice care prior to hospitalization. We excluded hospitalizations in June 2012 because of an irregularity in data availability for that month.

Hospitalizations were categorized into 5 specialty cohorts according to service line. The 5 cohorts were those used for the CMS hospital-wide readmission measure and included surgery/gynecology, medicine, cardiovascular, cardiorespiratory, and neurology.¹⁵ Among the 3 clinical conditions tracked as part of HRRP, heart failure and pneumonia were a subset of the cardiorespiratory cohort, while acute myocardial infarction was a subset of the cardiovascular cohort. Our use of cohorts was threefold: first, the average risk of readmission differs substantially across these cohorts, so pooling them produces heterogeneous risk strata; second, risk variables perform differently in different cohorts, so one single model may not be as accurate for calculating risk; and, third, the use of disease cohorts makes our results comparable to the CMS model and similar to other readmission studies in Medicare.^7,8,15

For development of the risk model, the outcome was 30-day unplanned hospital readmission. Planned readmissions were excluded; these were defined by the CMS algorithm as readmissions in which a typically planned procedure occurred in a hospitalization with a nonacute principal diagnosis.¹⁶ Independent variables included age and comorbidities in the final hospital-wide readmission models for each of the 5 specialty cohorts.¹⁵ In order to produce the best possible individual risk prediction for each patient, we added additional independent variables that CMS avoids for hospital quality measurement purposes but that contribute to risk of readmission: sex, race, dual eligibility status, number of prior AMA discharges, intensive care unit stay during current hospitalization, coronary care unit stay during current hospitalization, and hospitalization in the prior 30, 90, and 180 days. We also included an indicator variable for hospitalizations with more than 9 discharge diagnosis codes on or after January 2011, the time at which Medicare allowed an increase of the number of International Classification of Diseases, 9th Revision-Clinical Modification diagnosis billing codes from 9 to 25.¹⁷ This indicator adjusts for the increased availability of comorbidity codes, which might otherwise inflate the predicted risk relative to hospitalizations prior to that date.

Based on the risk models, each hospitalization was assigned a predicted risk of readmission. For each specialty cohort, we pooled all hospitalizations across all study years and divided them into risk quintiles. We categorized hospitalizations as high risk if in the highest quintile, medium risk if in the middle 3 quintiles, and low risk if in the lowest quintile of predicted risk for all study hospitalizations in a given specialty cohort.

For our time trend analyses, we studied 2 outcomes: monthly mean predicted risk and monthly ratio of observed readmissions to expected readmissions for patients in the lowest, middle, and highest categories of predicted risk of readmission. We studied monthly predicted risk to determine whether the average readmission risk of patients was changing over time as admission and readmission rates were declining. We studied the ratio of O/E readmissions to determine whether the decline in overall readmissions was more substantial in particular risk strata; we used the ratio of O/E readmissions, which measures number of readmissions divided by number of readmissions predicted by the model, rather than crude observed readmissions, as O/E readmissions account for any changes in risk profiles over time within each risk stratum. Independent variables in our trend analyses were year—entered as a continuous variable—and indicators for postintroduction of the Affordable Care Act (ACA, March 2010) and for postintroduction of HRRP (October 2012); these time indicators were included because of prior studies demonstrating that the introduction of ACA was associated with a decrease from baseline in readmission rates, which leveled off after introduction of HRRP.⁷ We also included an indicator for calendar quarter to account for seasonal effects.

 

Statistical Analysis

We developed generalized estimating equation models to predict 30-day unplanned readmission for each of the 5 specialty cohorts. The 5 models were fit using all patients in each cohort for the included time period and were adjusted for clustering by hospital. We assessed discrimination by calculating area under the receiver operating characteristic curve (AUC) for the 5 models; the AUCs measured the models’ ability to distinguish patients who were readmitted versus those who were not.¹⁸ We also calculated AUCs for each year to examine model performance over time.

Using these models, we calculated predicted risk for each hospitalization and averaged these to obtain mean predicted risk for each specialty cohort for each month. To test for trends in mean risk, we estimated 5 time series models, one for each cohort, with the dependent variable of monthly mean predicted risk. For each cohort, we first estimated a series of 12 empty autoregressive models, each with a different autoregressive term (1, 2...12). For each model, we calculated χ² for the test that the autocorrelation was 0; based on a comparison of chi-squared values, we specified an autocorrelation of 1 month for all models. Accordingly, a 1-month lag was used to estimate one final model for each cohort. Independent variables included year and indicators for post-ACA and post-HRRP; these variables captured the effect of trends over time and the introduction of these policy changes, respectively.¹⁹

To determine whether changes in risk over time were a result of changes in particular risk groups, we categorized hospitalizations into risk strata based on quintiles of predicted risk for each specialty cohort for the entire study period. For each individual year, we calculated the proportion of hospitalizations in the highest, middle, and lowest readmission risk strata for each cohort.

We calculated the monthly ratio of O/E readmission for hospitalizations in the lowest 20%, middle 60%, and highest 20% of readmission risk by month; O/E reflects the excess or deficit observed events relative to the number predicted by the model. Using this monthly O/E as the dependent variable, we developed autoregressive time series models as above, again with a 1-month lag, for each of these 3 risk strata in each cohort. As before, independent variables were year as a continuous variable, indicator variables for post-ACA and post-HRRP, and a categorical variable for calendar quarter.

All analyses were done in SAS version 9.3 (SAS Institute Inc., Cary, NC) and Stata version 14.2 (StataCorp LLC, College Station, TX).

RESULTS

We included 47,288,961 hospitalizations in the study, of which 11,231,242 (23.8%) were in the surgery/gynecology cohort, 19,548,711 (41.3%) were in the medicine cohort, 5,433,125 (11.5%) were in the cardiovascular cohort, 8,179,691 (17.3%) were in the cardiorespiratory cohort, and 2,896,192 (6.1%) were in the neurology cohort. The readmission rate was 16.2% (n = 7,642,161) overall, with the highest rates observed in the cardiorespiratory (20.5%) and medicine (17.6%) cohorts and the lowest rates observed in the surgery/gynecology (11.8%) and neurology (13.8%) cohorts.

The final predictive models for each cohort ranged in number of parameters from 56 for the cardiorespiratory cohort to 264 for the surgery/gynecology cohort. The models had AUCs of 0.70, 0.65, 0.67, 0.65, and 0.63 for the surgery/gynecology, medicine, cardiovascular, cardiorespiratory, and neurology cohorts, respectively; AUCs remained fairly stable over time for all disease cohorts (Appendix Table 1).

We observed an increase in the mean predicted readmission risk for hospitalizations in the surgery/gynecology and cardiovascular hospitalizations in early 2011 (Figure 1), a period between the introduction of ACA in March 2010 and the introduction of HRRP in October 2012. In time series models, the surgery/gynecology, cardiovascular, and neurology cohorts had increased predictive risks of readmission of 0.24%, 0.32%, and 0.13% per year, respectively, although this difference did not reach statistical significance for the cardiovascular cohort (Table 1). We found no association between introduction of ACA or HRRP and predicted risk for these cohorts (Table 1). There were no trends or differences in predicted readmission risk for hospitalizations in the medicine cohort. We observed a seasonal variation in predicted readmission risk for the cardiorespiratory cohort but no notable change in predicted risk over time (Figure 1); in the time series model, there was a slight decrease in risk following introduction of HRRP (Table 1).

After categorizing hospitalizations by predicted readmission risk, trends in the percent of hospitalizations in low, middle, and high risk strata differed by cohort. In the surgery/gynecology cohort, the proportion of hospitalizations in the lowest risk stratum increased only slightly, from 20.1% in 2009 to 21.1% of all surgery/gynecology hospitalizations in 2015 (Appendix Table 2). The proportion of surgery/gynecology hospitalizations in the high risk stratum (top quintile of risk) increased from 16.1% to 21.6% between 2009 and 2011 and remained at 21.8% in 2015, and the proportion of surgery/gynecology hospitalizations in the middle risk stratum (middle three quintiles of risk) decreased from 63.7% in 2009 to 59.4% in 2011 to 57.1% in 2015. Low-risk hospitalizations in the medicine cohort decreased from 21.7% in 2009 to 19.0% in 2015, while high-risk hospitalizations increased from 18.2% to 20.7% during the period. Hospitalizations in the lowest stratum of risk steadily declined in both the cardiovascular and neurology cohorts, from 24.9% to 14.8% and 22.6% to 17.3% of hospitalizations during the period, respectively; this was accompanied by an increase in the proportion of high-risk hospitalizations in each of these cohorts from 16.0% to 23.4% and 17.8% to 21.6%, respectively. The proportion of hospitalizations in each of the 3 risk strata remained relatively stable in the cardiorespiratory cohort (Appendix Table 2).

In each of the 5 cohorts, O/E readmissions steadily declined from 2009 to 2015 for hospitalizations with the lowest, middle, and highest predicted readmission risk (Figure 2). Each risk stratum had similar rates of decline during the study period for all cohorts (Table 2). Among surgery/gynecology hospitalizations, the monthly O/E readmission declined by 0.030 per year from an initial ratio of 0.936 for the lowest risk hospitalizations, by 0.037 per year for the middle risk hospitalizations, and by 0.036 per year for the highest risk hospitalizations (Table 2). Similarly, for hospitalizations in the lowest versus highest risk of readmission, annual decreases in O/E readmission rates were 0.018 versus 0.015, 0.034 versus 0.033, 0.020 versus 0.015, and 0.038 versus 0.029 for the medicine, cardiovascular, cardiorespiratory, and neurology cohorts, respectively. For all cohorts and in all risk strata, we found no significant change in O/E readmission risk with introduction of ACA or HRRP (Table 2).

 

DISCUSSION

In this 6-year, national study of Medicare hospitalizations, we found that readmission risk increased over time for surgical and neurological patients but did not increase in medicine or cardiorespiratory hospitalizations, even though those cohorts are known to have had substantial decreases in admissions and readmissions over the same time period.^7,8 Moreover, we found that O/E readmissions decreased similarly for all hospitalized Medicare patients, whether of low, moderate, or high risk of readmission. These findings suggest that hospital efforts have resulted in improved outcomes across the risk spectrum.

A number of mechanisms may account for the across-the-board improvements in readmission reduction. Many hospitals have instituted system-wide interventions, including patient education, medicine reconciliation, and early postdischarge follow-up,²⁰ which may have reduced readmissions across all patient risk strata. Alternatively, hospitals may have implemented interventions that disproportionally benefited low-risk patients while simultaneously utilizing interventions that only benefited high-risk patients. For instance, increasing threshold for admission⁷ may have the greatest effect on low-risk patients who could be most easily managed at home, while many intensive transitional care interventions have been developed to target only high-risk patients.^21,22

With the introduction of HRRP, there have been a number of concerns about the readmission measure used to penalize hospitals for high readmission rates. One major concern has been that the readmission metric may be flawed in its ability to capture continued improvement related to readmission.²³ Some have suggested that with better population health management, admissions will decrease, patient risk of the remaining patients will increase, and hospitals will be increasingly filled with patients who have high likelihood of readmission. This potential for increased risk with HRRP was suggested by a recent study that found that comorbidities increased in hospitalized Medicare beneficiaries between 2010 and 2013.¹¹ Our results were mixed in supporting this potential phenomenon because we examined global risk of readmission and found that some of the cohorts had increased risk over time while others did not. Others have expressed concern that readmission measure does not account for socioeconomic status, which has been associated with readmission rates.^24-27 Although we did not directly examine socioeconomic status in our study, we found that hospitals have been able to reduce readmission across all levels of risk, which includes markers of socioeconomic status, including race and Medicaid eligibility status.

Although we hypothesized that readmission risk would increase as number of hospitalizations decreased over time, we found no increase in readmission risk among the cohorts with HRRP diagnoses that had the largest decrease in readmission rates.^7,8 Conversely, readmission risk did increase—with a concurrent increase in the proportion of high-risk hospitalizations—in the surgery/gynecology and neurology cohorts that were not subject to HRRP penalties. Nonetheless, rehospitalizations were reduced for all risk categories in these 2 cohorts. Notably, surgery/gynecology and neurology had the lowest readmission rates overall. These findings suggest that initiatives to prevent initial hospitalizations, such as increasing the threshold for postoperative admission, may have had a greater effect on low- versus high-risk patients in low-risk hospitalizations. However, once a patient is hospitalized, multidisciplinary strategies appear to be effective at reducing readmissions for all risk classes in these cohorts.

For the 3 cohorts in which we observed an increase in readmission risk among hospitalized patients, the risk appeared to increase in early 2011. This time was about 10 months after passage of ACA, the timing of which was previously associated with a drop in readmission rates,^7,8 but well before HRRP went into effect in October 2012. The increase in readmission risk coincided with an increase in the number of diagnostic codes that could be included on a hospital claim to Medicare.¹⁷ This increase in allowable codes allowed us to capture more diagnoses for some patients, potentially resulting in an increase in apparent predicted risk of readmissions. While we adjusted for this in our predictive models, we may not have fully accounted for differences in risk related to coding change. As a result, some of the observed differences in risk in our study may be attributable to coding differences. More broadly, studies demonstrating the success of HRRP have typically examined risk-adjusted rates of readmission.^3,7 It is possible that a small portion of the observed reduction in risk-adjusted readmission rates may be related to the increase in predicted risk of readmission observed in our study. Future assessment of trends in readmission during this period should consider accounting for change in the number of allowed billing codes.

Other limitations should be considered in the interpretation of this study. First, like many predictive models for readmission,¹⁴ ours had imperfect discrimination, which could affect our results. Second, our study was based on older Medicare patients, so findings may not be applicable to younger patients. Third, while we accounted for surrogates for socioeconomic status, including dual eligibility and race, our models lacked other socioeconomic and community factors that can influence readmission.^24-26 Nonetheless, 1 study suggested that easily measured socioeconomic factors may not have a strong influence on the readmission metric used by Medicare.²⁸ Fourth, while our study included over 47 million hospitalizations, our time trend analyses used calendar month as the primary independent variable. As our study included 77 months, we may not have had sufficient power to detect small changes in risk over time.

Medicare readmissions have declined steadily in recent years, presumably at least partly in response to policy changes including HRRP. We found that hospitals have been effective at reducing readmissions across a range of patient risk strata and clinical conditions. As a result, the overall risk of readmission for hospitalized patients has remained constant for some but not all conditions. Whether institutions can continue to reduce readmission rates for most types of patients remains to be seen.

 

Acknowledgments

This study was supported by the Agency for Healthcare Research and Quality (AHRQ) grant R01HS022882. Dr. Blecker was supported by the AHRQ grant K08HS23683. The authors would like to thank Shawn Hoke and Jane Padikkala for administrative support.

Disclosure

This study was supported by the Agency for Healthcare Research and Quality (AHRQ) grants R01HS022882 and K08HS23683. The authors have no conflicts to report.

Given the high cost of readmissions to the healthcare system, there has been a substantial push to reduce readmissions by policymakers.¹ Among these is the Hospital Readmissions Reduction Program (HRRP), in which hospitals with higher than expected readmission rates receive reduced payments from Medicare.² Recent evidence has suggested the success of such policy changes, with multiple reports demonstrating a decrease in 30-day readmission rates in the Medicare population starting in 2010.^3-8

Initiatives to reduce readmissions can also have an effect on total number of admissions.^9,10 Indeed, along with the recent reduction in readmission, there has been a reduction in all admissions among Medicare beneficiaries.^11,12 Some studies have found that as admissions have decreased, the burden of comorbidity has increased among hospitalized patients,^3,11 suggesting that hospitals may be increasingly filled with patients at high risk of readmission. However, whether readmission risk among hospitalized patients has changed remains unknown, and understanding changes in risk profile could help inform which patients to target with future interventions to reduce readmissions.

Hospital efforts to reduce readmissions may have differential effects on types of patients by risk. For instance, low-intensity, system-wide interventions such as standardized discharge instructions or medicine reconciliation may have a stronger effect on patients at relatively low risk of readmission who may have a few important drivers of readmission that are easily overcome. Alternatively, the impact of intensive care transitions management might be greatest for high-risk patients, who have the most need for postdischarge medications, follow-up, and self-care.

The purpose of this study was therefore twofold: (1) to observe changes in average monthly risk of readmission among hospitalized Medicare patients and (2) to examine changes in readmission rates for Medicare patients at various risk of readmission. We hypothesized that readmission risk in the Medicare population would increase in recent years, as overall number of admissions and readmissions have fallen.^7,11 Additionally, we hypothesized that standardized readmission rates would decline less in highest risk patients as compared with the lowest risk patients because transitional care interventions may not be able to mitigate the large burden of comorbidity and social issues present in many high-risk patients.^13,14

METHODS

We performed a retrospective cohort study of hospitalizations to US nonfederal short-term acute care facilities by Medicare beneficiaries between January 2009 and June 2015. The design involved 4 steps. First, we estimated a predictive model for unplanned readmissions within 30 days of discharge. Second, we assigned each hospitalization a predicted risk of readmission based on the model. Third, we studied trends in mean predicted risk of readmission during the study period. Fourth, we examined trends in observed to expected (O/E) readmission for hospitalizations in the lowest, middle, and highest categories of predicted risk of readmission to determine whether reductions in readmissions were more substantial in certain risk groups than in others.

Data were obtained from the Centers for Medicare and Medicaid Services (CMS) Inpatient Standard Analytic File and the Medicare Enrollment Data Base. We included hospitalizations of fee-for-service Medicare beneficiaries age ≥65 with continuous enrollment in Part A Medicare fee-for-service for at least 1 year prior and 30 days after the hospitalization.¹⁵ Hospitalizations with a discharge disposition of death, transfer to another acute hospital, and left against medical advice (AMA) were excluded. We also excluded patients with enrollment in hospice care prior to hospitalization. We excluded hospitalizations in June 2012 because of an irregularity in data availability for that month.

Hospitalizations were categorized into 5 specialty cohorts according to service line. The 5 cohorts were those used for the CMS hospital-wide readmission measure and included surgery/gynecology, medicine, cardiovascular, cardiorespiratory, and neurology.¹⁵ Among the 3 clinical conditions tracked as part of HRRP, heart failure and pneumonia were a subset of the cardiorespiratory cohort, while acute myocardial infarction was a subset of the cardiovascular cohort. Our use of cohorts was threefold: first, the average risk of readmission differs substantially across these cohorts, so pooling them produces heterogeneous risk strata; second, risk variables perform differently in different cohorts, so one single model may not be as accurate for calculating risk; and, third, the use of disease cohorts makes our results comparable to the CMS model and similar to other readmission studies in Medicare.^7,8,15

For development of the risk model, the outcome was 30-day unplanned hospital readmission. Planned readmissions were excluded; these were defined by the CMS algorithm as readmissions in which a typically planned procedure occurred in a hospitalization with a nonacute principal diagnosis.¹⁶ Independent variables included age and comorbidities in the final hospital-wide readmission models for each of the 5 specialty cohorts.¹⁵ In order to produce the best possible individual risk prediction for each patient, we added additional independent variables that CMS avoids for hospital quality measurement purposes but that contribute to risk of readmission: sex, race, dual eligibility status, number of prior AMA discharges, intensive care unit stay during current hospitalization, coronary care unit stay during current hospitalization, and hospitalization in the prior 30, 90, and 180 days. We also included an indicator variable for hospitalizations with more than 9 discharge diagnosis codes on or after January 2011, the time at which Medicare allowed an increase of the number of International Classification of Diseases, 9th Revision-Clinical Modification diagnosis billing codes from 9 to 25.¹⁷ This indicator adjusts for the increased availability of comorbidity codes, which might otherwise inflate the predicted risk relative to hospitalizations prior to that date.

Based on the risk models, each hospitalization was assigned a predicted risk of readmission. For each specialty cohort, we pooled all hospitalizations across all study years and divided them into risk quintiles. We categorized hospitalizations as high risk if in the highest quintile, medium risk if in the middle 3 quintiles, and low risk if in the lowest quintile of predicted risk for all study hospitalizations in a given specialty cohort.

For our time trend analyses, we studied 2 outcomes: monthly mean predicted risk and monthly ratio of observed readmissions to expected readmissions for patients in the lowest, middle, and highest categories of predicted risk of readmission. We studied monthly predicted risk to determine whether the average readmission risk of patients was changing over time as admission and readmission rates were declining. We studied the ratio of O/E readmissions to determine whether the decline in overall readmissions was more substantial in particular risk strata; we used the ratio of O/E readmissions, which measures number of readmissions divided by number of readmissions predicted by the model, rather than crude observed readmissions, as O/E readmissions account for any changes in risk profiles over time within each risk stratum. Independent variables in our trend analyses were year—entered as a continuous variable—and indicators for postintroduction of the Affordable Care Act (ACA, March 2010) and for postintroduction of HRRP (October 2012); these time indicators were included because of prior studies demonstrating that the introduction of ACA was associated with a decrease from baseline in readmission rates, which leveled off after introduction of HRRP.⁷ We also included an indicator for calendar quarter to account for seasonal effects.

 

Statistical Analysis

We developed generalized estimating equation models to predict 30-day unplanned readmission for each of the 5 specialty cohorts. The 5 models were fit using all patients in each cohort for the included time period and were adjusted for clustering by hospital. We assessed discrimination by calculating area under the receiver operating characteristic curve (AUC) for the 5 models; the AUCs measured the models’ ability to distinguish patients who were readmitted versus those who were not.¹⁸ We also calculated AUCs for each year to examine model performance over time.

Using these models, we calculated predicted risk for each hospitalization and averaged these to obtain mean predicted risk for each specialty cohort for each month. To test for trends in mean risk, we estimated 5 time series models, one for each cohort, with the dependent variable of monthly mean predicted risk. For each cohort, we first estimated a series of 12 empty autoregressive models, each with a different autoregressive term (1, 2...12). For each model, we calculated χ² for the test that the autocorrelation was 0; based on a comparison of chi-squared values, we specified an autocorrelation of 1 month for all models. Accordingly, a 1-month lag was used to estimate one final model for each cohort. Independent variables included year and indicators for post-ACA and post-HRRP; these variables captured the effect of trends over time and the introduction of these policy changes, respectively.¹⁹

To determine whether changes in risk over time were a result of changes in particular risk groups, we categorized hospitalizations into risk strata based on quintiles of predicted risk for each specialty cohort for the entire study period. For each individual year, we calculated the proportion of hospitalizations in the highest, middle, and lowest readmission risk strata for each cohort.

We calculated the monthly ratio of O/E readmission for hospitalizations in the lowest 20%, middle 60%, and highest 20% of readmission risk by month; O/E reflects the excess or deficit observed events relative to the number predicted by the model. Using this monthly O/E as the dependent variable, we developed autoregressive time series models as above, again with a 1-month lag, for each of these 3 risk strata in each cohort. As before, independent variables were year as a continuous variable, indicator variables for post-ACA and post-HRRP, and a categorical variable for calendar quarter.

All analyses were done in SAS version 9.3 (SAS Institute Inc., Cary, NC) and Stata version 14.2 (StataCorp LLC, College Station, TX).

RESULTS

We included 47,288,961 hospitalizations in the study, of which 11,231,242 (23.8%) were in the surgery/gynecology cohort, 19,548,711 (41.3%) were in the medicine cohort, 5,433,125 (11.5%) were in the cardiovascular cohort, 8,179,691 (17.3%) were in the cardiorespiratory cohort, and 2,896,192 (6.1%) were in the neurology cohort. The readmission rate was 16.2% (n = 7,642,161) overall, with the highest rates observed in the cardiorespiratory (20.5%) and medicine (17.6%) cohorts and the lowest rates observed in the surgery/gynecology (11.8%) and neurology (13.8%) cohorts.

The final predictive models for each cohort ranged in number of parameters from 56 for the cardiorespiratory cohort to 264 for the surgery/gynecology cohort. The models had AUCs of 0.70, 0.65, 0.67, 0.65, and 0.63 for the surgery/gynecology, medicine, cardiovascular, cardiorespiratory, and neurology cohorts, respectively; AUCs remained fairly stable over time for all disease cohorts (Appendix Table 1).

We observed an increase in the mean predicted readmission risk for hospitalizations in the surgery/gynecology and cardiovascular hospitalizations in early 2011 (Figure 1), a period between the introduction of ACA in March 2010 and the introduction of HRRP in October 2012. In time series models, the surgery/gynecology, cardiovascular, and neurology cohorts had increased predictive risks of readmission of 0.24%, 0.32%, and 0.13% per year, respectively, although this difference did not reach statistical significance for the cardiovascular cohort (Table 1). We found no association between introduction of ACA or HRRP and predicted risk for these cohorts (Table 1). There were no trends or differences in predicted readmission risk for hospitalizations in the medicine cohort. We observed a seasonal variation in predicted readmission risk for the cardiorespiratory cohort but no notable change in predicted risk over time (Figure 1); in the time series model, there was a slight decrease in risk following introduction of HRRP (Table 1).

After categorizing hospitalizations by predicted readmission risk, trends in the percent of hospitalizations in low, middle, and high risk strata differed by cohort. In the surgery/gynecology cohort, the proportion of hospitalizations in the lowest risk stratum increased only slightly, from 20.1% in 2009 to 21.1% of all surgery/gynecology hospitalizations in 2015 (Appendix Table 2). The proportion of surgery/gynecology hospitalizations in the high risk stratum (top quintile of risk) increased from 16.1% to 21.6% between 2009 and 2011 and remained at 21.8% in 2015, and the proportion of surgery/gynecology hospitalizations in the middle risk stratum (middle three quintiles of risk) decreased from 63.7% in 2009 to 59.4% in 2011 to 57.1% in 2015. Low-risk hospitalizations in the medicine cohort decreased from 21.7% in 2009 to 19.0% in 2015, while high-risk hospitalizations increased from 18.2% to 20.7% during the period. Hospitalizations in the lowest stratum of risk steadily declined in both the cardiovascular and neurology cohorts, from 24.9% to 14.8% and 22.6% to 17.3% of hospitalizations during the period, respectively; this was accompanied by an increase in the proportion of high-risk hospitalizations in each of these cohorts from 16.0% to 23.4% and 17.8% to 21.6%, respectively. The proportion of hospitalizations in each of the 3 risk strata remained relatively stable in the cardiorespiratory cohort (Appendix Table 2).

In each of the 5 cohorts, O/E readmissions steadily declined from 2009 to 2015 for hospitalizations with the lowest, middle, and highest predicted readmission risk (Figure 2). Each risk stratum had similar rates of decline during the study period for all cohorts (Table 2). Among surgery/gynecology hospitalizations, the monthly O/E readmission declined by 0.030 per year from an initial ratio of 0.936 for the lowest risk hospitalizations, by 0.037 per year for the middle risk hospitalizations, and by 0.036 per year for the highest risk hospitalizations (Table 2). Similarly, for hospitalizations in the lowest versus highest risk of readmission, annual decreases in O/E readmission rates were 0.018 versus 0.015, 0.034 versus 0.033, 0.020 versus 0.015, and 0.038 versus 0.029 for the medicine, cardiovascular, cardiorespiratory, and neurology cohorts, respectively. For all cohorts and in all risk strata, we found no significant change in O/E readmission risk with introduction of ACA or HRRP (Table 2).

 

DISCUSSION

In this 6-year, national study of Medicare hospitalizations, we found that readmission risk increased over time for surgical and neurological patients but did not increase in medicine or cardiorespiratory hospitalizations, even though those cohorts are known to have had substantial decreases in admissions and readmissions over the same time period.^7,8 Moreover, we found that O/E readmissions decreased similarly for all hospitalized Medicare patients, whether of low, moderate, or high risk of readmission. These findings suggest that hospital efforts have resulted in improved outcomes across the risk spectrum.

A number of mechanisms may account for the across-the-board improvements in readmission reduction. Many hospitals have instituted system-wide interventions, including patient education, medicine reconciliation, and early postdischarge follow-up,²⁰ which may have reduced readmissions across all patient risk strata. Alternatively, hospitals may have implemented interventions that disproportionally benefited low-risk patients while simultaneously utilizing interventions that only benefited high-risk patients. For instance, increasing threshold for admission⁷ may have the greatest effect on low-risk patients who could be most easily managed at home, while many intensive transitional care interventions have been developed to target only high-risk patients.^21,22

With the introduction of HRRP, there have been a number of concerns about the readmission measure used to penalize hospitals for high readmission rates. One major concern has been that the readmission metric may be flawed in its ability to capture continued improvement related to readmission.²³ Some have suggested that with better population health management, admissions will decrease, patient risk of the remaining patients will increase, and hospitals will be increasingly filled with patients who have high likelihood of readmission. This potential for increased risk with HRRP was suggested by a recent study that found that comorbidities increased in hospitalized Medicare beneficiaries between 2010 and 2013.¹¹ Our results were mixed in supporting this potential phenomenon because we examined global risk of readmission and found that some of the cohorts had increased risk over time while others did not. Others have expressed concern that readmission measure does not account for socioeconomic status, which has been associated with readmission rates.^24-27 Although we did not directly examine socioeconomic status in our study, we found that hospitals have been able to reduce readmission across all levels of risk, which includes markers of socioeconomic status, including race and Medicaid eligibility status.

Although we hypothesized that readmission risk would increase as number of hospitalizations decreased over time, we found no increase in readmission risk among the cohorts with HRRP diagnoses that had the largest decrease in readmission rates.^7,8 Conversely, readmission risk did increase—with a concurrent increase in the proportion of high-risk hospitalizations—in the surgery/gynecology and neurology cohorts that were not subject to HRRP penalties. Nonetheless, rehospitalizations were reduced for all risk categories in these 2 cohorts. Notably, surgery/gynecology and neurology had the lowest readmission rates overall. These findings suggest that initiatives to prevent initial hospitalizations, such as increasing the threshold for postoperative admission, may have had a greater effect on low- versus high-risk patients in low-risk hospitalizations. However, once a patient is hospitalized, multidisciplinary strategies appear to be effective at reducing readmissions for all risk classes in these cohorts.

For the 3 cohorts in which we observed an increase in readmission risk among hospitalized patients, the risk appeared to increase in early 2011. This time was about 10 months after passage of ACA, the timing of which was previously associated with a drop in readmission rates,^7,8 but well before HRRP went into effect in October 2012. The increase in readmission risk coincided with an increase in the number of diagnostic codes that could be included on a hospital claim to Medicare.¹⁷ This increase in allowable codes allowed us to capture more diagnoses for some patients, potentially resulting in an increase in apparent predicted risk of readmissions. While we adjusted for this in our predictive models, we may not have fully accounted for differences in risk related to coding change. As a result, some of the observed differences in risk in our study may be attributable to coding differences. More broadly, studies demonstrating the success of HRRP have typically examined risk-adjusted rates of readmission.^3,7 It is possible that a small portion of the observed reduction in risk-adjusted readmission rates may be related to the increase in predicted risk of readmission observed in our study. Future assessment of trends in readmission during this period should consider accounting for change in the number of allowed billing codes.

Other limitations should be considered in the interpretation of this study. First, like many predictive models for readmission,¹⁴ ours had imperfect discrimination, which could affect our results. Second, our study was based on older Medicare patients, so findings may not be applicable to younger patients. Third, while we accounted for surrogates for socioeconomic status, including dual eligibility and race, our models lacked other socioeconomic and community factors that can influence readmission.^24-26 Nonetheless, 1 study suggested that easily measured socioeconomic factors may not have a strong influence on the readmission metric used by Medicare.²⁸ Fourth, while our study included over 47 million hospitalizations, our time trend analyses used calendar month as the primary independent variable. As our study included 77 months, we may not have had sufficient power to detect small changes in risk over time.

Medicare readmissions have declined steadily in recent years, presumably at least partly in response to policy changes including HRRP. We found that hospitals have been effective at reducing readmissions across a range of patient risk strata and clinical conditions. As a result, the overall risk of readmission for hospitalized patients has remained constant for some but not all conditions. Whether institutions can continue to reduce readmission rates for most types of patients remains to be seen.

 

Acknowledgments

This study was supported by the Agency for Healthcare Research and Quality (AHRQ) grant R01HS022882. Dr. Blecker was supported by the AHRQ grant K08HS23683. The authors would like to thank Shawn Hoke and Jane Padikkala for administrative support.

Disclosure

This study was supported by the Agency for Healthcare Research and Quality (AHRQ) grants R01HS022882 and K08HS23683. The authors have no conflicts to report.

Given the high cost of readmissions to the healthcare system, there has been a substantial push to reduce readmissions by policymakers.¹ Among these is the Hospital Readmissions Reduction Program (HRRP), in which hospitals with higher than expected readmission rates receive reduced payments from Medicare.² Recent evidence has suggested the success of such policy changes, with multiple reports demonstrating a decrease in 30-day readmission rates in the Medicare population starting in 2010.^3-8

Initiatives to reduce readmissions can also have an effect on total number of admissions.^9,10 Indeed, along with the recent reduction in readmission, there has been a reduction in all admissions among Medicare beneficiaries.^11,12 Some studies have found that as admissions have decreased, the burden of comorbidity has increased among hospitalized patients,^3,11 suggesting that hospitals may be increasingly filled with patients at high risk of readmission. However, whether readmission risk among hospitalized patients has changed remains unknown, and understanding changes in risk profile could help inform which patients to target with future interventions to reduce readmissions.

Hospital efforts to reduce readmissions may have differential effects on types of patients by risk. For instance, low-intensity, system-wide interventions such as standardized discharge instructions or medicine reconciliation may have a stronger effect on patients at relatively low risk of readmission who may have a few important drivers of readmission that are easily overcome. Alternatively, the impact of intensive care transitions management might be greatest for high-risk patients, who have the most need for postdischarge medications, follow-up, and self-care.

The purpose of this study was therefore twofold: (1) to observe changes in average monthly risk of readmission among hospitalized Medicare patients and (2) to examine changes in readmission rates for Medicare patients at various risk of readmission. We hypothesized that readmission risk in the Medicare population would increase in recent years, as overall number of admissions and readmissions have fallen.^7,11 Additionally, we hypothesized that standardized readmission rates would decline less in highest risk patients as compared with the lowest risk patients because transitional care interventions may not be able to mitigate the large burden of comorbidity and social issues present in many high-risk patients.^13,14

METHODS

We performed a retrospective cohort study of hospitalizations to US nonfederal short-term acute care facilities by Medicare beneficiaries between January 2009 and June 2015. The design involved 4 steps. First, we estimated a predictive model for unplanned readmissions within 30 days of discharge. Second, we assigned each hospitalization a predicted risk of readmission based on the model. Third, we studied trends in mean predicted risk of readmission during the study period. Fourth, we examined trends in observed to expected (O/E) readmission for hospitalizations in the lowest, middle, and highest categories of predicted risk of readmission to determine whether reductions in readmissions were more substantial in certain risk groups than in others.

Data were obtained from the Centers for Medicare and Medicaid Services (CMS) Inpatient Standard Analytic File and the Medicare Enrollment Data Base. We included hospitalizations of fee-for-service Medicare beneficiaries age ≥65 with continuous enrollment in Part A Medicare fee-for-service for at least 1 year prior and 30 days after the hospitalization.¹⁵ Hospitalizations with a discharge disposition of death, transfer to another acute hospital, and left against medical advice (AMA) were excluded. We also excluded patients with enrollment in hospice care prior to hospitalization. We excluded hospitalizations in June 2012 because of an irregularity in data availability for that month.

Hospitalizations were categorized into 5 specialty cohorts according to service line. The 5 cohorts were those used for the CMS hospital-wide readmission measure and included surgery/gynecology, medicine, cardiovascular, cardiorespiratory, and neurology.¹⁵ Among the 3 clinical conditions tracked as part of HRRP, heart failure and pneumonia were a subset of the cardiorespiratory cohort, while acute myocardial infarction was a subset of the cardiovascular cohort. Our use of cohorts was threefold: first, the average risk of readmission differs substantially across these cohorts, so pooling them produces heterogeneous risk strata; second, risk variables perform differently in different cohorts, so one single model may not be as accurate for calculating risk; and, third, the use of disease cohorts makes our results comparable to the CMS model and similar to other readmission studies in Medicare.^7,8,15

For development of the risk model, the outcome was 30-day unplanned hospital readmission. Planned readmissions were excluded; these were defined by the CMS algorithm as readmissions in which a typically planned procedure occurred in a hospitalization with a nonacute principal diagnosis.¹⁶ Independent variables included age and comorbidities in the final hospital-wide readmission models for each of the 5 specialty cohorts.¹⁵ In order to produce the best possible individual risk prediction for each patient, we added additional independent variables that CMS avoids for hospital quality measurement purposes but that contribute to risk of readmission: sex, race, dual eligibility status, number of prior AMA discharges, intensive care unit stay during current hospitalization, coronary care unit stay during current hospitalization, and hospitalization in the prior 30, 90, and 180 days. We also included an indicator variable for hospitalizations with more than 9 discharge diagnosis codes on or after January 2011, the time at which Medicare allowed an increase of the number of International Classification of Diseases, 9th Revision-Clinical Modification diagnosis billing codes from 9 to 25.¹⁷ This indicator adjusts for the increased availability of comorbidity codes, which might otherwise inflate the predicted risk relative to hospitalizations prior to that date.

Based on the risk models, each hospitalization was assigned a predicted risk of readmission. For each specialty cohort, we pooled all hospitalizations across all study years and divided them into risk quintiles. We categorized hospitalizations as high risk if in the highest quintile, medium risk if in the middle 3 quintiles, and low risk if in the lowest quintile of predicted risk for all study hospitalizations in a given specialty cohort.

For our time trend analyses, we studied 2 outcomes: monthly mean predicted risk and monthly ratio of observed readmissions to expected readmissions for patients in the lowest, middle, and highest categories of predicted risk of readmission. We studied monthly predicted risk to determine whether the average readmission risk of patients was changing over time as admission and readmission rates were declining. We studied the ratio of O/E readmissions to determine whether the decline in overall readmissions was more substantial in particular risk strata; we used the ratio of O/E readmissions, which measures number of readmissions divided by number of readmissions predicted by the model, rather than crude observed readmissions, as O/E readmissions account for any changes in risk profiles over time within each risk stratum. Independent variables in our trend analyses were year—entered as a continuous variable—and indicators for postintroduction of the Affordable Care Act (ACA, March 2010) and for postintroduction of HRRP (October 2012); these time indicators were included because of prior studies demonstrating that the introduction of ACA was associated with a decrease from baseline in readmission rates, which leveled off after introduction of HRRP.⁷ We also included an indicator for calendar quarter to account for seasonal effects.

 

Statistical Analysis

We developed generalized estimating equation models to predict 30-day unplanned readmission for each of the 5 specialty cohorts. The 5 models were fit using all patients in each cohort for the included time period and were adjusted for clustering by hospital. We assessed discrimination by calculating area under the receiver operating characteristic curve (AUC) for the 5 models; the AUCs measured the models’ ability to distinguish patients who were readmitted versus those who were not.¹⁸ We also calculated AUCs for each year to examine model performance over time.

Using these models, we calculated predicted risk for each hospitalization and averaged these to obtain mean predicted risk for each specialty cohort for each month. To test for trends in mean risk, we estimated 5 time series models, one for each cohort, with the dependent variable of monthly mean predicted risk. For each cohort, we first estimated a series of 12 empty autoregressive models, each with a different autoregressive term (1, 2...12). For each model, we calculated χ² for the test that the autocorrelation was 0; based on a comparison of chi-squared values, we specified an autocorrelation of 1 month for all models. Accordingly, a 1-month lag was used to estimate one final model for each cohort. Independent variables included year and indicators for post-ACA and post-HRRP; these variables captured the effect of trends over time and the introduction of these policy changes, respectively.¹⁹

To determine whether changes in risk over time were a result of changes in particular risk groups, we categorized hospitalizations into risk strata based on quintiles of predicted risk for each specialty cohort for the entire study period. For each individual year, we calculated the proportion of hospitalizations in the highest, middle, and lowest readmission risk strata for each cohort.

We calculated the monthly ratio of O/E readmission for hospitalizations in the lowest 20%, middle 60%, and highest 20% of readmission risk by month; O/E reflects the excess or deficit observed events relative to the number predicted by the model. Using this monthly O/E as the dependent variable, we developed autoregressive time series models as above, again with a 1-month lag, for each of these 3 risk strata in each cohort. As before, independent variables were year as a continuous variable, indicator variables for post-ACA and post-HRRP, and a categorical variable for calendar quarter.

All analyses were done in SAS version 9.3 (SAS Institute Inc., Cary, NC) and Stata version 14.2 (StataCorp LLC, College Station, TX).

RESULTS

We included 47,288,961 hospitalizations in the study, of which 11,231,242 (23.8%) were in the surgery/gynecology cohort, 19,548,711 (41.3%) were in the medicine cohort, 5,433,125 (11.5%) were in the cardiovascular cohort, 8,179,691 (17.3%) were in the cardiorespiratory cohort, and 2,896,192 (6.1%) were in the neurology cohort. The readmission rate was 16.2% (n = 7,642,161) overall, with the highest rates observed in the cardiorespiratory (20.5%) and medicine (17.6%) cohorts and the lowest rates observed in the surgery/gynecology (11.8%) and neurology (13.8%) cohorts.

The final predictive models for each cohort ranged in number of parameters from 56 for the cardiorespiratory cohort to 264 for the surgery/gynecology cohort. The models had AUCs of 0.70, 0.65, 0.67, 0.65, and 0.63 for the surgery/gynecology, medicine, cardiovascular, cardiorespiratory, and neurology cohorts, respectively; AUCs remained fairly stable over time for all disease cohorts (Appendix Table 1).

We observed an increase in the mean predicted readmission risk for hospitalizations in the surgery/gynecology and cardiovascular hospitalizations in early 2011 (Figure 1), a period between the introduction of ACA in March 2010 and the introduction of HRRP in October 2012. In time series models, the surgery/gynecology, cardiovascular, and neurology cohorts had increased predictive risks of readmission of 0.24%, 0.32%, and 0.13% per year, respectively, although this difference did not reach statistical significance for the cardiovascular cohort (Table 1). We found no association between introduction of ACA or HRRP and predicted risk for these cohorts (Table 1). There were no trends or differences in predicted readmission risk for hospitalizations in the medicine cohort. We observed a seasonal variation in predicted readmission risk for the cardiorespiratory cohort but no notable change in predicted risk over time (Figure 1); in the time series model, there was a slight decrease in risk following introduction of HRRP (Table 1).

After categorizing hospitalizations by predicted readmission risk, trends in the percent of hospitalizations in low, middle, and high risk strata differed by cohort. In the surgery/gynecology cohort, the proportion of hospitalizations in the lowest risk stratum increased only slightly, from 20.1% in 2009 to 21.1% of all surgery/gynecology hospitalizations in 2015 (Appendix Table 2). The proportion of surgery/gynecology hospitalizations in the high risk stratum (top quintile of risk) increased from 16.1% to 21.6% between 2009 and 2011 and remained at 21.8% in 2015, and the proportion of surgery/gynecology hospitalizations in the middle risk stratum (middle three quintiles of risk) decreased from 63.7% in 2009 to 59.4% in 2011 to 57.1% in 2015. Low-risk hospitalizations in the medicine cohort decreased from 21.7% in 2009 to 19.0% in 2015, while high-risk hospitalizations increased from 18.2% to 20.7% during the period. Hospitalizations in the lowest stratum of risk steadily declined in both the cardiovascular and neurology cohorts, from 24.9% to 14.8% and 22.6% to 17.3% of hospitalizations during the period, respectively; this was accompanied by an increase in the proportion of high-risk hospitalizations in each of these cohorts from 16.0% to 23.4% and 17.8% to 21.6%, respectively. The proportion of hospitalizations in each of the 3 risk strata remained relatively stable in the cardiorespiratory cohort (Appendix Table 2).

In each of the 5 cohorts, O/E readmissions steadily declined from 2009 to 2015 for hospitalizations with the lowest, middle, and highest predicted readmission risk (Figure 2). Each risk stratum had similar rates of decline during the study period for all cohorts (Table 2). Among surgery/gynecology hospitalizations, the monthly O/E readmission declined by 0.030 per year from an initial ratio of 0.936 for the lowest risk hospitalizations, by 0.037 per year for the middle risk hospitalizations, and by 0.036 per year for the highest risk hospitalizations (Table 2). Similarly, for hospitalizations in the lowest versus highest risk of readmission, annual decreases in O/E readmission rates were 0.018 versus 0.015, 0.034 versus 0.033, 0.020 versus 0.015, and 0.038 versus 0.029 for the medicine, cardiovascular, cardiorespiratory, and neurology cohorts, respectively. For all cohorts and in all risk strata, we found no significant change in O/E readmission risk with introduction of ACA or HRRP (Table 2).

 

DISCUSSION

In this 6-year, national study of Medicare hospitalizations, we found that readmission risk increased over time for surgical and neurological patients but did not increase in medicine or cardiorespiratory hospitalizations, even though those cohorts are known to have had substantial decreases in admissions and readmissions over the same time period.^7,8 Moreover, we found that O/E readmissions decreased similarly for all hospitalized Medicare patients, whether of low, moderate, or high risk of readmission. These findings suggest that hospital efforts have resulted in improved outcomes across the risk spectrum.

A number of mechanisms may account for the across-the-board improvements in readmission reduction. Many hospitals have instituted system-wide interventions, including patient education, medicine reconciliation, and early postdischarge follow-up,²⁰ which may have reduced readmissions across all patient risk strata. Alternatively, hospitals may have implemented interventions that disproportionally benefited low-risk patients while simultaneously utilizing interventions that only benefited high-risk patients. For instance, increasing threshold for admission⁷ may have the greatest effect on low-risk patients who could be most easily managed at home, while many intensive transitional care interventions have been developed to target only high-risk patients.^21,22

With the introduction of HRRP, there have been a number of concerns about the readmission measure used to penalize hospitals for high readmission rates. One major concern has been that the readmission metric may be flawed in its ability to capture continued improvement related to readmission.²³ Some have suggested that with better population health management, admissions will decrease, patient risk of the remaining patients will increase, and hospitals will be increasingly filled with patients who have high likelihood of readmission. This potential for increased risk with HRRP was suggested by a recent study that found that comorbidities increased in hospitalized Medicare beneficiaries between 2010 and 2013.¹¹ Our results were mixed in supporting this potential phenomenon because we examined global risk of readmission and found that some of the cohorts had increased risk over time while others did not. Others have expressed concern that readmission measure does not account for socioeconomic status, which has been associated with readmission rates.^24-27 Although we did not directly examine socioeconomic status in our study, we found that hospitals have been able to reduce readmission across all levels of risk, which includes markers of socioeconomic status, including race and Medicaid eligibility status.

Although we hypothesized that readmission risk would increase as number of hospitalizations decreased over time, we found no increase in readmission risk among the cohorts with HRRP diagnoses that had the largest decrease in readmission rates.^7,8 Conversely, readmission risk did increase—with a concurrent increase in the proportion of high-risk hospitalizations—in the surgery/gynecology and neurology cohorts that were not subject to HRRP penalties. Nonetheless, rehospitalizations were reduced for all risk categories in these 2 cohorts. Notably, surgery/gynecology and neurology had the lowest readmission rates overall. These findings suggest that initiatives to prevent initial hospitalizations, such as increasing the threshold for postoperative admission, may have had a greater effect on low- versus high-risk patients in low-risk hospitalizations. However, once a patient is hospitalized, multidisciplinary strategies appear to be effective at reducing readmissions for all risk classes in these cohorts.

For the 3 cohorts in which we observed an increase in readmission risk among hospitalized patients, the risk appeared to increase in early 2011. This time was about 10 months after passage of ACA, the timing of which was previously associated with a drop in readmission rates,^7,8 but well before HRRP went into effect in October 2012. The increase in readmission risk coincided with an increase in the number of diagnostic codes that could be included on a hospital claim to Medicare.¹⁷ This increase in allowable codes allowed us to capture more diagnoses for some patients, potentially resulting in an increase in apparent predicted risk of readmissions. While we adjusted for this in our predictive models, we may not have fully accounted for differences in risk related to coding change. As a result, some of the observed differences in risk in our study may be attributable to coding differences. More broadly, studies demonstrating the success of HRRP have typically examined risk-adjusted rates of readmission.^3,7 It is possible that a small portion of the observed reduction in risk-adjusted readmission rates may be related to the increase in predicted risk of readmission observed in our study. Future assessment of trends in readmission during this period should consider accounting for change in the number of allowed billing codes.

Other limitations should be considered in the interpretation of this study. First, like many predictive models for readmission,¹⁴ ours had imperfect discrimination, which could affect our results. Second, our study was based on older Medicare patients, so findings may not be applicable to younger patients. Third, while we accounted for surrogates for socioeconomic status, including dual eligibility and race, our models lacked other socioeconomic and community factors that can influence readmission.^24-26 Nonetheless, 1 study suggested that easily measured socioeconomic factors may not have a strong influence on the readmission metric used by Medicare.²⁸ Fourth, while our study included over 47 million hospitalizations, our time trend analyses used calendar month as the primary independent variable. As our study included 77 months, we may not have had sufficient power to detect small changes in risk over time.

Medicare readmissions have declined steadily in recent years, presumably at least partly in response to policy changes including HRRP. We found that hospitals have been effective at reducing readmissions across a range of patient risk strata and clinical conditions. As a result, the overall risk of readmission for hospitalized patients has remained constant for some but not all conditions. Whether institutions can continue to reduce readmission rates for most types of patients remains to be seen.

 

Acknowledgments

This study was supported by the Agency for Healthcare Research and Quality (AHRQ) grant R01HS022882. Dr. Blecker was supported by the AHRQ grant K08HS23683. The authors would like to thank Shawn Hoke and Jane Padikkala for administrative support.

Disclosure

This study was supported by the Agency for Healthcare Research and Quality (AHRQ) grants R01HS022882 and K08HS23683. The authors have no conflicts to report.

Point-of-care ultrasound (POCUS) is a valuable tool to assist in the diagnosis and treatment of many common diseases.^1-11 Its use has increased in clinical settings over the years, primarily because of more portable, economical, high-quality devices and training availability.¹² POCUS improves procedural success and guides the diagnostic management of hospitalized patients.^2,9-12 Literature details the training of medical students,^13,14 residents,^15-21 and providers in emergency medicine²² and critical care,^23,24 as well as focused cardiac training with hospitalists.^25-27 However, no literature exists describing a comprehensive longitudinal training program for hospitalists or skills retention.

This document details the hospital medicine department’s ultrasound training program from Regions Hospital, part of HealthPartners in Saint Paul, Minnesota, a large tertiary care medical center. We describe the development and effectiveness of the Comprehensive Hospitalist Assessment and Mentorship with Portfolios (CHAMP) Ultrasound Program. This approach is intended to support the development of POCUS training programs at other organizations.

The aim of the program was to build a comprehensive bedside ultrasound training paradigm for hospitalists. The primary objective of the study was to assess the program’s effect on skills over time. Secondary objectives were confidence ratings in the use of ultrasound and with various patient care realms (volume management, quality of physical exam, and ability to narrow the differential diagnosis). We hypothesized there would be higher retention of ultrasound skills in those who completed portfolios and/or monthly scanning sessions as well as increased confidence through all secondary outcome measures (see below).

MATERIALS AND METHODS

This was a retrospective descriptive report of hospitalists who entered the CHAMP Ultrasound Program. Study participants were providers from the 454-bed Regions Hospital in Saint Paul, Minnesota. The study was deemed exempt by the HealthPartners Institutional Review Board. Three discrete 3-day courses and two 1-day in-person courses held at the Regions Hospital Simulation Center (Saint Paul, Minnesota) were studied.

Program Description

In 2014, a working group was developed in the hospital medicine department to support the hospital-wide POCUS committee with a charter to provide standardized training for providers to complete credentialing.²⁸ The goal of the hospital medicine ultrasound program was to establish the use of ultrasound by credentialed hospitalists into well-defined applications integrated into the practice of hospital medicine. Two providers were selected to lead the efforts and completed additional training through the American College of Chest Physicians (CHEST) Certificate of Completion Program.²⁹ An overall director was designated with the responsibilities delineated in supplementary Appendix 1. This director provided leadership on group practice, protocols, and equipment, creating the organizational framework for success with the training program. The hospital medicine training program had a 3-day in-person component built off the CHEST Critical Care Ultrasonography Program.²⁴ The curriculum was adapted from the American College of Chest Physicians/Société de Réanimation de Langue Française Statement on Competence in Critical Care Ultrasonography.³⁰ See Table 1 for the components of the training program.

All components of the training program are required to receive the certificate of completion with the exception of the refresher training. Learner feedback after each 3-day course and refresher training was incorporated into subsequent iterations of the training program. During initial phases, additional hands-on faculty were recruited from emergency medicine and critical care who had extensive experience with bedside ultrasound. Subsequently, faculty consisted of former course participants. All faculty followed a standard set of ultrasound and educational principles to guide the hands-on training of participants (supplementary Appendix 2).

Online Modules

As a prerequisite to the 3-day introductory course, hospitalists were required to complete modules for precourse knowledge involving a set of focused-topic online reading and videos with quizzes (supplementary Appendix 3).

3-Day In-Person Course with Assessments

The 3-day course provided 6 hours of didactics, 8 hours of image interpretation, and 9 hours of hands-on instruction (supplementary Appendix 4). Hospitalists first attended a large group didactic, followed by divided groups in image interpretation and hands-on scanning.²⁴

 

Didactics were provided in a room with a 2-screen set up. Providers used 1 screen to present primary content and the other for simultaneously scanning a human model.

Image interpretation sessions were interactive smaller group learning forums in which participants reviewed high-yield images related to the care of hospital medicine patients and received feedback. Approximately 45 videos with normal and abnormal findings were reviewed during each session.

The hands-on scanning component was accomplished with human models and a faculty-to-participant ratio between 1:2 and 1:3. Human models for this course were paid community models. A variety of ultrasound machine platforms were provided for participants. Learning objectives were clearly delineated prior to each scanning session to ensure the coverage of required content.

Portfolios

Portfolio development was a key aspect in overall POCUS competency for each participant. The hospital medicine department’s required portfolio files are presented in the Figure, with standards coinciding with the quality assurance grading rubric as developed by the POCUS committee at Regions Hospital and described by Mathews and Zwank.²⁸ Images taken with real patients were submitted without patient identifiers to a shared online portal. Faculty provided regular cycling feedback by entering the status of submission (accepted or declined) and specific comments on images and interpretations. Learners worked off of the feedback, practiced their skills, and resubmitted files. An image was considered acceptable if it met criteria of depth, axis, and gain and showed the required organ. Participants could use the same patient for different views but could not use the same patient for multiple images of the same view.

Refresher Training: 1-Day In-Person Course with Assessments and Monthly Scanning Sessions (Optional)

Only hospitalists who completed the 3-day course were eligible to take the 1-day in-person refresher course (supplementary Appendix 5). The first half of the course incorporated scanning with live human models, while the second half of the course had scanning with hospitalized patients focusing on pathology (pleural effusion, hydronephrosis, reduced left ventricular function, etc.). The course was offered at 3, 6, and 12 months after the initial 3-day course.

Monthly scanning sessions occurred for 2 hours every third Friday and were also available prior to the 1-day refresher. The first 90 minutes had a hands-on scanning component with hospitalized patients with faculty supervision (1:2 ratio). The last 30 minutes had an image interpretation component.

Assessments

Knowledge and skills assessment were adapted from the CHEST model (supplementary Appendix 6).²⁴ Before and after the 3-day and 1-day in-person courses, the same hands-on skills assessment with a checklist was provided (supplementary Appendix 7). Before and after the 3-day course, a written knowledge assessment with case-based image interpretation was provided (supplementary Appendix 6). A final knowledge and skills assessment was given at either of the in-person courses to those who completed the required components of the training. Passing scores for the final knowledge assessment were established at 85% items correct by an expert panel by using the Angoff method.³¹ This same standard was applied to the final skills examination. Participants who do not pass the final assessments are provided opportunities for further training and allowed to reattempt the assessments. In this regard, there is a standard training outcome but variances in length of training time for each participant. Pre- and postcourse skills assessments used the same faculty, checklist, and ultrasound device. Raters received an orientation the day prior to each in-person course, reviewing common learner pitfalls, reviewing the checklist, and discussing specific examples.

Measurement

Participant demographic and clinical information was collected at the initial 3-day course for all participants, including age, gender, specialty, years of experience, and number and type of ultrasound procedures personally conducted or supervised in the past year. For skills assessment, a 20-item dichotomous checklist was developed and scored as done correctly or not done/done incorrectly. This same assessment was provided both before and after each of the 3-day and 1-day courses. A 20-question image-based knowledge assessment was also developed and administered both before and after the 3-day course only. The same 20-item checklist was used for the final skills examination. However, a new more detailed 50-question examination was written for the final examination after the portfolio of images was complete. Self-reported measures were confidence in the use of ultrasound, volume management, quality of physical exam, and ability to narrow the differential diagnosis. Confidence in ultrasound use, confidence in volume management, and quality of physical exam were assessed by using a questionnaire both before and after the 3-day course and 1-day course. Participants rated confidence and quality on a 5-point scale, 1 being least confident and 5 being most confident.

 

Statistical Analysis

Demographics of the included hospitalist population and pre and post 3-day assessments, including knowledge score, skills score, confidence in ultrasound use, confidence in volume management, and quality of physical exam, were summarized. Values for all assessment variables are presented as percentages. Confidence scores were reported as a percentage of the Likert scale (eg, 4/5 was reported as 80%). Skills and written examinations were expressed as percentages of items correct. Data were reported as median and interquartile range or means and standard deviation based on variable distributions. Differences between pre- and postvalues for 3-day course variables were assessed by using 2-sample paired Wilcoxon signed rank tests with a 95% confidence level.

For the subset of hospitalists who also completed the 1-day course, pre and post 1-day course assessments, including skills score, confidence in ultrasound use, confidence in volume management, and quality of physical exam, were summarized. Differences between pre- and postvalues for 1-day assessment variables were assessed by using 2-sample paired Wilcoxon signed rank tests with a 95% confidence level.

For hospitalists who completed both the 3-day and 1-day courses, the change in course assessments, including skills score, confidence in ultrasound use, confidence in volume management, and quality of physical exam, was assessed by summarizing the change from post 3-day metrics to pre 1-day metrics (Table 2). The differences between these 2 assessments were evaluated by using 2-sample paired Wilcoxon signed rank tests with a 95% confidence level. Changes in skills score from post 3-day assessment to pre 1-day assessment were also compared for hospitalists completing any of the portfolio and those completing none, and for hospitalists attending any monthly scanning sessions and those who did not attend any, by using analysis of variance and Scheffe tests.

Multiple linear regression was performed with the change in skills assessment score from postcompletion of the 3-day course to precompletion of the 1-day course as the dependent variable. Hospitalists were split into 2 age groups (30-39 and 40-49) for the purpose of this analysis. The percent of monthly scanning sessions attended, age category, timing of 1-day course, and percent portfolio were assessed as possible predictors of the skills score by using simple linear regression with a P = .05 cutoff. A final model was chosen based on predictors significant in simple linear regression and included the percent of the portfolio completed and attendance of monthly scanning sessions.

RESULTS

Demographics

Of the 56 3-day course participants, 53 had complete data (Table 3). Three participants with incomplete data completed most of the course but left prior to postcourse assessments and were excluded from the analysis. Twenty-three hospitalists also completed the 1-day in-person course. Seven hospitalists completed the 1-day course 3 months after the initial course, 8 completed it at 6 months, and 8 completed it at 12 months. Completed portfolios required 164 approved video images. Fifteen of the 23 hospitalists at the 1-day course have started and are working towards completion of the online portfolio, while 9 of the 23 participated in the monthly scanning sessions.

3-Day In-Person Course

For the 53 hospitalists who completed skills-based assessments, performance increased significantly after the 3-day course. Knowledge scores also increased significantly from preassessment to postassessment. Self-reported confidence ratings for ultrasound use, confidence in volume management, and quality of physical exam all increased significantly from preassessment to postassessment (Table 2).

Refresher Training: 1-Day In-Person Course

Because the refresher training was encouraged but not required, only 25 of 53 hospitalists, 23 with complete data, completed the 1-day course. For the 23 hospitalists who completed skills-based assessments before and after the 1-day course, mean skills scores increased significantly (Table 2). Self-reported confidence ratings for ultrasound use, confidence in volume management, and quality of physical exam all increased significantly from preassessment to postassessment (Table 2).

Monthly Scanning Sessions and Portfolio Development

The skills retention from initial course to refresher course by portfolio completion and monthly scanning sessions is shown in Table 2. Multiple regression analysis showed that for every 10% increase in the percent of monthly sessions attended, the mean change in skills score was 3.7% (P = .017), and for every 10% increase in the percent of portfolio completed, the mean change in skills score was 2.5% (P = .04), showing that both monthly scanning session attendance and portfolio completion are significantly predictive of skills retention over time.

Final Assessments

Four providers met mastery at initial attempt. No providers to date have needed remediation. Many others are going through different stages of the process and are expected to attain mastery in a short period of time.

 

DISCUSSION

This is the first description of a successful longitudinal training program with assessments in POCUS for hospital medicine providers that shows an increase in skill retention with the use of a follow-up course and bedside scanning.

The CHAMP Ultrasound Program was developed to provide hospital medicine clinicians with a specialty focused in-house training pathway in POCUS and to assist in sustained skills acquisition by providing opportunities for regular feedback and practice. Practice with regular expert feedback is a critical aspect to develop and maintain skills in POCUS.^32,33 Arntfield³⁴ described the utility of remote supervision with feedback for ultrasound training in critical care, which demonstrated varying learning curves in the submission of portfolio images.^35,36 The CHAMP Ultrasound training program provided expert oversight, longitudinal supervision, and feedback for course participants. The educational method of mastery learning was employed by setting minimum standards and allowing learners to practice until they met that standard.^37-39

This unique program is made possible by the availability of expert-level faculty. Assessment scores improved with an initial 3-day course; however, they also decayed over time, most prominently with hospitalists that did not continue with POCUS scanning after their initial course. Ironically, those who performed more ultrasounds in the year prior to beginning the 3-day course had lower confidence ratings, likely explained by their awareness of their limitations and opportunities for improvement. The incorporation of refresher training to supplement the core 3-day course and portfolio development are key additions that differentiate this training program. These additions and the demonstration of successful training make this a durable pathway for other hospitalist programs. There are many workshops and short courses for medical students, residents, and practicing providers in POCUS.^40-43 However, without an opportunity for longitudinal supervision and feedback, there is a noted decrease in the skills for participants. The refresher training with its 2 components (1-day in-person course and monthly scanning sessions) provides evidence of the value of mentored training.

In the initial program development, refresher training was encouraged but optional. We intentionally tracked those that completed refresher training compared with those that did not. Based on the results showing significant skills retention among those attending some form of refresher training, the program is planned to change to make this a requirement. We recommend refresher training within 12 months of the initial introductory course. There were several hospitalists that were unable to accommodate taking a full-day refresher course and, therefore, monthly scanning sessions were provided as an alternative.

The main limitation of the study is that it was completed in a single hospital system with available training mentors in POCUS. This gave us the ability to perform longitudinal training but may make this less reproducible in other hospital systems. Another limitation is that our course participants did not complete the pre- and postknowledge assessments for the refresher training components of the program, though they did for the initial 3-day course. Our pre- and postassessments have not been externally shown to produce valid data, though they are based on the already validated CHEST ultrasound data.⁴⁴

Finally, our CHAMP Ultrasound Program required a significant time commitment by both faculty and learners. A relatively small percentage of hospitalists have completed the final assessments. The reasons are multifactorial, including program rigor, desire by certain hospitalists to know the basics but not pursue more expertise, and the challenges of developing a skillset that takes dedicated practice over time. We have aimed to address these barriers by providing additional hands-on scanning opportunities, giving timely feedback with portfolios, and obtaining more ultrasound machines. We expect more hospitalists to complete the final assessments in the coming year as evidenced by portfolio submissions to the shared online portal and many choosing to attend either the monthly scanning sessions and/or the 1-day course. We recognize that other institutions may need to adapt our program to suit their local environment.

CONCLUSION

A comprehensive longitudinal ultrasound training program including competency assessments significantly improved ultrasound acquisition skills with hospitalists. Those attending monthly scanning sessions and participating in the portfolio completion as well as a refresher course significantly retained and augmented their skills.

Acknowledgments

The authors would like to acknowledge Kelly Logue, Jason Robertson, MD, Jerome Siy, MD, Shauna Baer, and Jack Dressen for their support in the development and implementation of the POCUS program in hospital medicine.

Disclosure

The authors do not have any relevant financial disclosures to report.

Point-of-care ultrasound (POCUS) is a valuable tool to assist in the diagnosis and treatment of many common diseases.^1-11 Its use has increased in clinical settings over the years, primarily because of more portable, economical, high-quality devices and training availability.¹² POCUS improves procedural success and guides the diagnostic management of hospitalized patients.^2,9-12 Literature details the training of medical students,^13,14 residents,^15-21 and providers in emergency medicine²² and critical care,^23,24 as well as focused cardiac training with hospitalists.^25-27 However, no literature exists describing a comprehensive longitudinal training program for hospitalists or skills retention.

This document details the hospital medicine department’s ultrasound training program from Regions Hospital, part of HealthPartners in Saint Paul, Minnesota, a large tertiary care medical center. We describe the development and effectiveness of the Comprehensive Hospitalist Assessment and Mentorship with Portfolios (CHAMP) Ultrasound Program. This approach is intended to support the development of POCUS training programs at other organizations.

The aim of the program was to build a comprehensive bedside ultrasound training paradigm for hospitalists. The primary objective of the study was to assess the program’s effect on skills over time. Secondary objectives were confidence ratings in the use of ultrasound and with various patient care realms (volume management, quality of physical exam, and ability to narrow the differential diagnosis). We hypothesized there would be higher retention of ultrasound skills in those who completed portfolios and/or monthly scanning sessions as well as increased confidence through all secondary outcome measures (see below).

MATERIALS AND METHODS

This was a retrospective descriptive report of hospitalists who entered the CHAMP Ultrasound Program. Study participants were providers from the 454-bed Regions Hospital in Saint Paul, Minnesota. The study was deemed exempt by the HealthPartners Institutional Review Board. Three discrete 3-day courses and two 1-day in-person courses held at the Regions Hospital Simulation Center (Saint Paul, Minnesota) were studied.

Program Description

In 2014, a working group was developed in the hospital medicine department to support the hospital-wide POCUS committee with a charter to provide standardized training for providers to complete credentialing.²⁸ The goal of the hospital medicine ultrasound program was to establish the use of ultrasound by credentialed hospitalists into well-defined applications integrated into the practice of hospital medicine. Two providers were selected to lead the efforts and completed additional training through the American College of Chest Physicians (CHEST) Certificate of Completion Program.²⁹ An overall director was designated with the responsibilities delineated in supplementary Appendix 1. This director provided leadership on group practice, protocols, and equipment, creating the organizational framework for success with the training program. The hospital medicine training program had a 3-day in-person component built off the CHEST Critical Care Ultrasonography Program.²⁴ The curriculum was adapted from the American College of Chest Physicians/Société de Réanimation de Langue Française Statement on Competence in Critical Care Ultrasonography.³⁰ See Table 1 for the components of the training program.

All components of the training program are required to receive the certificate of completion with the exception of the refresher training. Learner feedback after each 3-day course and refresher training was incorporated into subsequent iterations of the training program. During initial phases, additional hands-on faculty were recruited from emergency medicine and critical care who had extensive experience with bedside ultrasound. Subsequently, faculty consisted of former course participants. All faculty followed a standard set of ultrasound and educational principles to guide the hands-on training of participants (supplementary Appendix 2).

Online Modules

As a prerequisite to the 3-day introductory course, hospitalists were required to complete modules for precourse knowledge involving a set of focused-topic online reading and videos with quizzes (supplementary Appendix 3).

3-Day In-Person Course with Assessments

The 3-day course provided 6 hours of didactics, 8 hours of image interpretation, and 9 hours of hands-on instruction (supplementary Appendix 4). Hospitalists first attended a large group didactic, followed by divided groups in image interpretation and hands-on scanning.²⁴

 

Didactics were provided in a room with a 2-screen set up. Providers used 1 screen to present primary content and the other for simultaneously scanning a human model.

Image interpretation sessions were interactive smaller group learning forums in which participants reviewed high-yield images related to the care of hospital medicine patients and received feedback. Approximately 45 videos with normal and abnormal findings were reviewed during each session.

The hands-on scanning component was accomplished with human models and a faculty-to-participant ratio between 1:2 and 1:3. Human models for this course were paid community models. A variety of ultrasound machine platforms were provided for participants. Learning objectives were clearly delineated prior to each scanning session to ensure the coverage of required content.

Portfolios

Portfolio development was a key aspect in overall POCUS competency for each participant. The hospital medicine department’s required portfolio files are presented in the Figure, with standards coinciding with the quality assurance grading rubric as developed by the POCUS committee at Regions Hospital and described by Mathews and Zwank.²⁸ Images taken with real patients were submitted without patient identifiers to a shared online portal. Faculty provided regular cycling feedback by entering the status of submission (accepted or declined) and specific comments on images and interpretations. Learners worked off of the feedback, practiced their skills, and resubmitted files. An image was considered acceptable if it met criteria of depth, axis, and gain and showed the required organ. Participants could use the same patient for different views but could not use the same patient for multiple images of the same view.

Refresher Training: 1-Day In-Person Course with Assessments and Monthly Scanning Sessions (Optional)

Only hospitalists who completed the 3-day course were eligible to take the 1-day in-person refresher course (supplementary Appendix 5). The first half of the course incorporated scanning with live human models, while the second half of the course had scanning with hospitalized patients focusing on pathology (pleural effusion, hydronephrosis, reduced left ventricular function, etc.). The course was offered at 3, 6, and 12 months after the initial 3-day course.

Monthly scanning sessions occurred for 2 hours every third Friday and were also available prior to the 1-day refresher. The first 90 minutes had a hands-on scanning component with hospitalized patients with faculty supervision (1:2 ratio). The last 30 minutes had an image interpretation component.

Assessments

Knowledge and skills assessment were adapted from the CHEST model (supplementary Appendix 6).²⁴ Before and after the 3-day and 1-day in-person courses, the same hands-on skills assessment with a checklist was provided (supplementary Appendix 7). Before and after the 3-day course, a written knowledge assessment with case-based image interpretation was provided (supplementary Appendix 6). A final knowledge and skills assessment was given at either of the in-person courses to those who completed the required components of the training. Passing scores for the final knowledge assessment were established at 85% items correct by an expert panel by using the Angoff method.³¹ This same standard was applied to the final skills examination. Participants who do not pass the final assessments are provided opportunities for further training and allowed to reattempt the assessments. In this regard, there is a standard training outcome but variances in length of training time for each participant. Pre- and postcourse skills assessments used the same faculty, checklist, and ultrasound device. Raters received an orientation the day prior to each in-person course, reviewing common learner pitfalls, reviewing the checklist, and discussing specific examples.

Measurement

Participant demographic and clinical information was collected at the initial 3-day course for all participants, including age, gender, specialty, years of experience, and number and type of ultrasound procedures personally conducted or supervised in the past year. For skills assessment, a 20-item dichotomous checklist was developed and scored as done correctly or not done/done incorrectly. This same assessment was provided both before and after each of the 3-day and 1-day courses. A 20-question image-based knowledge assessment was also developed and administered both before and after the 3-day course only. The same 20-item checklist was used for the final skills examination. However, a new more detailed 50-question examination was written for the final examination after the portfolio of images was complete. Self-reported measures were confidence in the use of ultrasound, volume management, quality of physical exam, and ability to narrow the differential diagnosis. Confidence in ultrasound use, confidence in volume management, and quality of physical exam were assessed by using a questionnaire both before and after the 3-day course and 1-day course. Participants rated confidence and quality on a 5-point scale, 1 being least confident and 5 being most confident.

 

Statistical Analysis

Demographics of the included hospitalist population and pre and post 3-day assessments, including knowledge score, skills score, confidence in ultrasound use, confidence in volume management, and quality of physical exam, were summarized. Values for all assessment variables are presented as percentages. Confidence scores were reported as a percentage of the Likert scale (eg, 4/5 was reported as 80%). Skills and written examinations were expressed as percentages of items correct. Data were reported as median and interquartile range or means and standard deviation based on variable distributions. Differences between pre- and postvalues for 3-day course variables were assessed by using 2-sample paired Wilcoxon signed rank tests with a 95% confidence level.

For the subset of hospitalists who also completed the 1-day course, pre and post 1-day course assessments, including skills score, confidence in ultrasound use, confidence in volume management, and quality of physical exam, were summarized. Differences between pre- and postvalues for 1-day assessment variables were assessed by using 2-sample paired Wilcoxon signed rank tests with a 95% confidence level.

For hospitalists who completed both the 3-day and 1-day courses, the change in course assessments, including skills score, confidence in ultrasound use, confidence in volume management, and quality of physical exam, was assessed by summarizing the change from post 3-day metrics to pre 1-day metrics (Table 2). The differences between these 2 assessments were evaluated by using 2-sample paired Wilcoxon signed rank tests with a 95% confidence level. Changes in skills score from post 3-day assessment to pre 1-day assessment were also compared for hospitalists completing any of the portfolio and those completing none, and for hospitalists attending any monthly scanning sessions and those who did not attend any, by using analysis of variance and Scheffe tests.

Multiple linear regression was performed with the change in skills assessment score from postcompletion of the 3-day course to precompletion of the 1-day course as the dependent variable. Hospitalists were split into 2 age groups (30-39 and 40-49) for the purpose of this analysis. The percent of monthly scanning sessions attended, age category, timing of 1-day course, and percent portfolio were assessed as possible predictors of the skills score by using simple linear regression with a P = .05 cutoff. A final model was chosen based on predictors significant in simple linear regression and included the percent of the portfolio completed and attendance of monthly scanning sessions.

RESULTS

Demographics

Of the 56 3-day course participants, 53 had complete data (Table 3). Three participants with incomplete data completed most of the course but left prior to postcourse assessments and were excluded from the analysis. Twenty-three hospitalists also completed the 1-day in-person course. Seven hospitalists completed the 1-day course 3 months after the initial course, 8 completed it at 6 months, and 8 completed it at 12 months. Completed portfolios required 164 approved video images. Fifteen of the 23 hospitalists at the 1-day course have started and are working towards completion of the online portfolio, while 9 of the 23 participated in the monthly scanning sessions.

3-Day In-Person Course

For the 53 hospitalists who completed skills-based assessments, performance increased significantly after the 3-day course. Knowledge scores also increased significantly from preassessment to postassessment. Self-reported confidence ratings for ultrasound use, confidence in volume management, and quality of physical exam all increased significantly from preassessment to postassessment (Table 2).

Refresher Training: 1-Day In-Person Course

Because the refresher training was encouraged but not required, only 25 of 53 hospitalists, 23 with complete data, completed the 1-day course. For the 23 hospitalists who completed skills-based assessments before and after the 1-day course, mean skills scores increased significantly (Table 2). Self-reported confidence ratings for ultrasound use, confidence in volume management, and quality of physical exam all increased significantly from preassessment to postassessment (Table 2).

Monthly Scanning Sessions and Portfolio Development

The skills retention from initial course to refresher course by portfolio completion and monthly scanning sessions is shown in Table 2. Multiple regression analysis showed that for every 10% increase in the percent of monthly sessions attended, the mean change in skills score was 3.7% (P = .017), and for every 10% increase in the percent of portfolio completed, the mean change in skills score was 2.5% (P = .04), showing that both monthly scanning session attendance and portfolio completion are significantly predictive of skills retention over time.

Final Assessments

Four providers met mastery at initial attempt. No providers to date have needed remediation. Many others are going through different stages of the process and are expected to attain mastery in a short period of time.

 

DISCUSSION

This is the first description of a successful longitudinal training program with assessments in POCUS for hospital medicine providers that shows an increase in skill retention with the use of a follow-up course and bedside scanning.

The CHAMP Ultrasound Program was developed to provide hospital medicine clinicians with a specialty focused in-house training pathway in POCUS and to assist in sustained skills acquisition by providing opportunities for regular feedback and practice. Practice with regular expert feedback is a critical aspect to develop and maintain skills in POCUS.^32,33 Arntfield³⁴ described the utility of remote supervision with feedback for ultrasound training in critical care, which demonstrated varying learning curves in the submission of portfolio images.^35,36 The CHAMP Ultrasound training program provided expert oversight, longitudinal supervision, and feedback for course participants. The educational method of mastery learning was employed by setting minimum standards and allowing learners to practice until they met that standard.^37-39

This unique program is made possible by the availability of expert-level faculty. Assessment scores improved with an initial 3-day course; however, they also decayed over time, most prominently with hospitalists that did not continue with POCUS scanning after their initial course. Ironically, those who performed more ultrasounds in the year prior to beginning the 3-day course had lower confidence ratings, likely explained by their awareness of their limitations and opportunities for improvement. The incorporation of refresher training to supplement the core 3-day course and portfolio development are key additions that differentiate this training program. These additions and the demonstration of successful training make this a durable pathway for other hospitalist programs. There are many workshops and short courses for medical students, residents, and practicing providers in POCUS.^40-43 However, without an opportunity for longitudinal supervision and feedback, there is a noted decrease in the skills for participants. The refresher training with its 2 components (1-day in-person course and monthly scanning sessions) provides evidence of the value of mentored training.

In the initial program development, refresher training was encouraged but optional. We intentionally tracked those that completed refresher training compared with those that did not. Based on the results showing significant skills retention among those attending some form of refresher training, the program is planned to change to make this a requirement. We recommend refresher training within 12 months of the initial introductory course. There were several hospitalists that were unable to accommodate taking a full-day refresher course and, therefore, monthly scanning sessions were provided as an alternative.

The main limitation of the study is that it was completed in a single hospital system with available training mentors in POCUS. This gave us the ability to perform longitudinal training but may make this less reproducible in other hospital systems. Another limitation is that our course participants did not complete the pre- and postknowledge assessments for the refresher training components of the program, though they did for the initial 3-day course. Our pre- and postassessments have not been externally shown to produce valid data, though they are based on the already validated CHEST ultrasound data.⁴⁴

Finally, our CHAMP Ultrasound Program required a significant time commitment by both faculty and learners. A relatively small percentage of hospitalists have completed the final assessments. The reasons are multifactorial, including program rigor, desire by certain hospitalists to know the basics but not pursue more expertise, and the challenges of developing a skillset that takes dedicated practice over time. We have aimed to address these barriers by providing additional hands-on scanning opportunities, giving timely feedback with portfolios, and obtaining more ultrasound machines. We expect more hospitalists to complete the final assessments in the coming year as evidenced by portfolio submissions to the shared online portal and many choosing to attend either the monthly scanning sessions and/or the 1-day course. We recognize that other institutions may need to adapt our program to suit their local environment.

CONCLUSION

A comprehensive longitudinal ultrasound training program including competency assessments significantly improved ultrasound acquisition skills with hospitalists. Those attending monthly scanning sessions and participating in the portfolio completion as well as a refresher course significantly retained and augmented their skills.

Acknowledgments

The authors would like to acknowledge Kelly Logue, Jason Robertson, MD, Jerome Siy, MD, Shauna Baer, and Jack Dressen for their support in the development and implementation of the POCUS program in hospital medicine.

Disclosure

The authors do not have any relevant financial disclosures to report.

Point-of-care ultrasound (POCUS) is a valuable tool to assist in the diagnosis and treatment of many common diseases.^1-11 Its use has increased in clinical settings over the years, primarily because of more portable, economical, high-quality devices and training availability.¹² POCUS improves procedural success and guides the diagnostic management of hospitalized patients.^2,9-12 Literature details the training of medical students,^13,14 residents,^15-21 and providers in emergency medicine²² and critical care,^23,24 as well as focused cardiac training with hospitalists.^25-27 However, no literature exists describing a comprehensive longitudinal training program for hospitalists or skills retention.

This document details the hospital medicine department’s ultrasound training program from Regions Hospital, part of HealthPartners in Saint Paul, Minnesota, a large tertiary care medical center. We describe the development and effectiveness of the Comprehensive Hospitalist Assessment and Mentorship with Portfolios (CHAMP) Ultrasound Program. This approach is intended to support the development of POCUS training programs at other organizations.

The aim of the program was to build a comprehensive bedside ultrasound training paradigm for hospitalists. The primary objective of the study was to assess the program’s effect on skills over time. Secondary objectives were confidence ratings in the use of ultrasound and with various patient care realms (volume management, quality of physical exam, and ability to narrow the differential diagnosis). We hypothesized there would be higher retention of ultrasound skills in those who completed portfolios and/or monthly scanning sessions as well as increased confidence through all secondary outcome measures (see below).

MATERIALS AND METHODS

This was a retrospective descriptive report of hospitalists who entered the CHAMP Ultrasound Program. Study participants were providers from the 454-bed Regions Hospital in Saint Paul, Minnesota. The study was deemed exempt by the HealthPartners Institutional Review Board. Three discrete 3-day courses and two 1-day in-person courses held at the Regions Hospital Simulation Center (Saint Paul, Minnesota) were studied.

Program Description

In 2014, a working group was developed in the hospital medicine department to support the hospital-wide POCUS committee with a charter to provide standardized training for providers to complete credentialing.²⁸ The goal of the hospital medicine ultrasound program was to establish the use of ultrasound by credentialed hospitalists into well-defined applications integrated into the practice of hospital medicine. Two providers were selected to lead the efforts and completed additional training through the American College of Chest Physicians (CHEST) Certificate of Completion Program.²⁹ An overall director was designated with the responsibilities delineated in supplementary Appendix 1. This director provided leadership on group practice, protocols, and equipment, creating the organizational framework for success with the training program. The hospital medicine training program had a 3-day in-person component built off the CHEST Critical Care Ultrasonography Program.²⁴ The curriculum was adapted from the American College of Chest Physicians/Société de Réanimation de Langue Française Statement on Competence in Critical Care Ultrasonography.³⁰ See Table 1 for the components of the training program.

All components of the training program are required to receive the certificate of completion with the exception of the refresher training. Learner feedback after each 3-day course and refresher training was incorporated into subsequent iterations of the training program. During initial phases, additional hands-on faculty were recruited from emergency medicine and critical care who had extensive experience with bedside ultrasound. Subsequently, faculty consisted of former course participants. All faculty followed a standard set of ultrasound and educational principles to guide the hands-on training of participants (supplementary Appendix 2).

Online Modules

As a prerequisite to the 3-day introductory course, hospitalists were required to complete modules for precourse knowledge involving a set of focused-topic online reading and videos with quizzes (supplementary Appendix 3).

3-Day In-Person Course with Assessments

The 3-day course provided 6 hours of didactics, 8 hours of image interpretation, and 9 hours of hands-on instruction (supplementary Appendix 4). Hospitalists first attended a large group didactic, followed by divided groups in image interpretation and hands-on scanning.²⁴

 

Didactics were provided in a room with a 2-screen set up. Providers used 1 screen to present primary content and the other for simultaneously scanning a human model.

Image interpretation sessions were interactive smaller group learning forums in which participants reviewed high-yield images related to the care of hospital medicine patients and received feedback. Approximately 45 videos with normal and abnormal findings were reviewed during each session.

The hands-on scanning component was accomplished with human models and a faculty-to-participant ratio between 1:2 and 1:3. Human models for this course were paid community models. A variety of ultrasound machine platforms were provided for participants. Learning objectives were clearly delineated prior to each scanning session to ensure the coverage of required content.

Portfolios

Portfolio development was a key aspect in overall POCUS competency for each participant. The hospital medicine department’s required portfolio files are presented in the Figure, with standards coinciding with the quality assurance grading rubric as developed by the POCUS committee at Regions Hospital and described by Mathews and Zwank.²⁸ Images taken with real patients were submitted without patient identifiers to a shared online portal. Faculty provided regular cycling feedback by entering the status of submission (accepted or declined) and specific comments on images and interpretations. Learners worked off of the feedback, practiced their skills, and resubmitted files. An image was considered acceptable if it met criteria of depth, axis, and gain and showed the required organ. Participants could use the same patient for different views but could not use the same patient for multiple images of the same view.

Refresher Training: 1-Day In-Person Course with Assessments and Monthly Scanning Sessions (Optional)

Only hospitalists who completed the 3-day course were eligible to take the 1-day in-person refresher course (supplementary Appendix 5). The first half of the course incorporated scanning with live human models, while the second half of the course had scanning with hospitalized patients focusing on pathology (pleural effusion, hydronephrosis, reduced left ventricular function, etc.). The course was offered at 3, 6, and 12 months after the initial 3-day course.

Monthly scanning sessions occurred for 2 hours every third Friday and were also available prior to the 1-day refresher. The first 90 minutes had a hands-on scanning component with hospitalized patients with faculty supervision (1:2 ratio). The last 30 minutes had an image interpretation component.

Assessments

Knowledge and skills assessment were adapted from the CHEST model (supplementary Appendix 6).²⁴ Before and after the 3-day and 1-day in-person courses, the same hands-on skills assessment with a checklist was provided (supplementary Appendix 7). Before and after the 3-day course, a written knowledge assessment with case-based image interpretation was provided (supplementary Appendix 6). A final knowledge and skills assessment was given at either of the in-person courses to those who completed the required components of the training. Passing scores for the final knowledge assessment were established at 85% items correct by an expert panel by using the Angoff method.³¹ This same standard was applied to the final skills examination. Participants who do not pass the final assessments are provided opportunities for further training and allowed to reattempt the assessments. In this regard, there is a standard training outcome but variances in length of training time for each participant. Pre- and postcourse skills assessments used the same faculty, checklist, and ultrasound device. Raters received an orientation the day prior to each in-person course, reviewing common learner pitfalls, reviewing the checklist, and discussing specific examples.

Measurement

Participant demographic and clinical information was collected at the initial 3-day course for all participants, including age, gender, specialty, years of experience, and number and type of ultrasound procedures personally conducted or supervised in the past year. For skills assessment, a 20-item dichotomous checklist was developed and scored as done correctly or not done/done incorrectly. This same assessment was provided both before and after each of the 3-day and 1-day courses. A 20-question image-based knowledge assessment was also developed and administered both before and after the 3-day course only. The same 20-item checklist was used for the final skills examination. However, a new more detailed 50-question examination was written for the final examination after the portfolio of images was complete. Self-reported measures were confidence in the use of ultrasound, volume management, quality of physical exam, and ability to narrow the differential diagnosis. Confidence in ultrasound use, confidence in volume management, and quality of physical exam were assessed by using a questionnaire both before and after the 3-day course and 1-day course. Participants rated confidence and quality on a 5-point scale, 1 being least confident and 5 being most confident.

 

Statistical Analysis

Demographics of the included hospitalist population and pre and post 3-day assessments, including knowledge score, skills score, confidence in ultrasound use, confidence in volume management, and quality of physical exam, were summarized. Values for all assessment variables are presented as percentages. Confidence scores were reported as a percentage of the Likert scale (eg, 4/5 was reported as 80%). Skills and written examinations were expressed as percentages of items correct. Data were reported as median and interquartile range or means and standard deviation based on variable distributions. Differences between pre- and postvalues for 3-day course variables were assessed by using 2-sample paired Wilcoxon signed rank tests with a 95% confidence level.

For the subset of hospitalists who also completed the 1-day course, pre and post 1-day course assessments, including skills score, confidence in ultrasound use, confidence in volume management, and quality of physical exam, were summarized. Differences between pre- and postvalues for 1-day assessment variables were assessed by using 2-sample paired Wilcoxon signed rank tests with a 95% confidence level.

For hospitalists who completed both the 3-day and 1-day courses, the change in course assessments, including skills score, confidence in ultrasound use, confidence in volume management, and quality of physical exam, was assessed by summarizing the change from post 3-day metrics to pre 1-day metrics (Table 2). The differences between these 2 assessments were evaluated by using 2-sample paired Wilcoxon signed rank tests with a 95% confidence level. Changes in skills score from post 3-day assessment to pre 1-day assessment were also compared for hospitalists completing any of the portfolio and those completing none, and for hospitalists attending any monthly scanning sessions and those who did not attend any, by using analysis of variance and Scheffe tests.

Multiple linear regression was performed with the change in skills assessment score from postcompletion of the 3-day course to precompletion of the 1-day course as the dependent variable. Hospitalists were split into 2 age groups (30-39 and 40-49) for the purpose of this analysis. The percent of monthly scanning sessions attended, age category, timing of 1-day course, and percent portfolio were assessed as possible predictors of the skills score by using simple linear regression with a P = .05 cutoff. A final model was chosen based on predictors significant in simple linear regression and included the percent of the portfolio completed and attendance of monthly scanning sessions.

RESULTS

Demographics

Of the 56 3-day course participants, 53 had complete data (Table 3). Three participants with incomplete data completed most of the course but left prior to postcourse assessments and were excluded from the analysis. Twenty-three hospitalists also completed the 1-day in-person course. Seven hospitalists completed the 1-day course 3 months after the initial course, 8 completed it at 6 months, and 8 completed it at 12 months. Completed portfolios required 164 approved video images. Fifteen of the 23 hospitalists at the 1-day course have started and are working towards completion of the online portfolio, while 9 of the 23 participated in the monthly scanning sessions.

3-Day In-Person Course

For the 53 hospitalists who completed skills-based assessments, performance increased significantly after the 3-day course. Knowledge scores also increased significantly from preassessment to postassessment. Self-reported confidence ratings for ultrasound use, confidence in volume management, and quality of physical exam all increased significantly from preassessment to postassessment (Table 2).

Refresher Training: 1-Day In-Person Course

Because the refresher training was encouraged but not required, only 25 of 53 hospitalists, 23 with complete data, completed the 1-day course. For the 23 hospitalists who completed skills-based assessments before and after the 1-day course, mean skills scores increased significantly (Table 2). Self-reported confidence ratings for ultrasound use, confidence in volume management, and quality of physical exam all increased significantly from preassessment to postassessment (Table 2).

Monthly Scanning Sessions and Portfolio Development

The skills retention from initial course to refresher course by portfolio completion and monthly scanning sessions is shown in Table 2. Multiple regression analysis showed that for every 10% increase in the percent of monthly sessions attended, the mean change in skills score was 3.7% (P = .017), and for every 10% increase in the percent of portfolio completed, the mean change in skills score was 2.5% (P = .04), showing that both monthly scanning session attendance and portfolio completion are significantly predictive of skills retention over time.

Final Assessments

Four providers met mastery at initial attempt. No providers to date have needed remediation. Many others are going through different stages of the process and are expected to attain mastery in a short period of time.

 

DISCUSSION

This is the first description of a successful longitudinal training program with assessments in POCUS for hospital medicine providers that shows an increase in skill retention with the use of a follow-up course and bedside scanning.

The CHAMP Ultrasound Program was developed to provide hospital medicine clinicians with a specialty focused in-house training pathway in POCUS and to assist in sustained skills acquisition by providing opportunities for regular feedback and practice. Practice with regular expert feedback is a critical aspect to develop and maintain skills in POCUS.^32,33 Arntfield³⁴ described the utility of remote supervision with feedback for ultrasound training in critical care, which demonstrated varying learning curves in the submission of portfolio images.^35,36 The CHAMP Ultrasound training program provided expert oversight, longitudinal supervision, and feedback for course participants. The educational method of mastery learning was employed by setting minimum standards and allowing learners to practice until they met that standard.^37-39

This unique program is made possible by the availability of expert-level faculty. Assessment scores improved with an initial 3-day course; however, they also decayed over time, most prominently with hospitalists that did not continue with POCUS scanning after their initial course. Ironically, those who performed more ultrasounds in the year prior to beginning the 3-day course had lower confidence ratings, likely explained by their awareness of their limitations and opportunities for improvement. The incorporation of refresher training to supplement the core 3-day course and portfolio development are key additions that differentiate this training program. These additions and the demonstration of successful training make this a durable pathway for other hospitalist programs. There are many workshops and short courses for medical students, residents, and practicing providers in POCUS.^40-43 However, without an opportunity for longitudinal supervision and feedback, there is a noted decrease in the skills for participants. The refresher training with its 2 components (1-day in-person course and monthly scanning sessions) provides evidence of the value of mentored training.

In the initial program development, refresher training was encouraged but optional. We intentionally tracked those that completed refresher training compared with those that did not. Based on the results showing significant skills retention among those attending some form of refresher training, the program is planned to change to make this a requirement. We recommend refresher training within 12 months of the initial introductory course. There were several hospitalists that were unable to accommodate taking a full-day refresher course and, therefore, monthly scanning sessions were provided as an alternative.

The main limitation of the study is that it was completed in a single hospital system with available training mentors in POCUS. This gave us the ability to perform longitudinal training but may make this less reproducible in other hospital systems. Another limitation is that our course participants did not complete the pre- and postknowledge assessments for the refresher training components of the program, though they did for the initial 3-day course. Our pre- and postassessments have not been externally shown to produce valid data, though they are based on the already validated CHEST ultrasound data.⁴⁴

Finally, our CHAMP Ultrasound Program required a significant time commitment by both faculty and learners. A relatively small percentage of hospitalists have completed the final assessments. The reasons are multifactorial, including program rigor, desire by certain hospitalists to know the basics but not pursue more expertise, and the challenges of developing a skillset that takes dedicated practice over time. We have aimed to address these barriers by providing additional hands-on scanning opportunities, giving timely feedback with portfolios, and obtaining more ultrasound machines. We expect more hospitalists to complete the final assessments in the coming year as evidenced by portfolio submissions to the shared online portal and many choosing to attend either the monthly scanning sessions and/or the 1-day course. We recognize that other institutions may need to adapt our program to suit their local environment.

CONCLUSION

A comprehensive longitudinal ultrasound training program including competency assessments significantly improved ultrasound acquisition skills with hospitalists. Those attending monthly scanning sessions and participating in the portfolio completion as well as a refresher course significantly retained and augmented their skills.

Acknowledgments

The authors would like to acknowledge Kelly Logue, Jason Robertson, MD, Jerome Siy, MD, Shauna Baer, and Jack Dressen for their support in the development and implementation of the POCUS program in hospital medicine.

Disclosure

The authors do not have any relevant financial disclosures to report.

ABSTRACT

Purpose In 2013-14, 2 clinics in the Watertown Regional Medical Center (WRMC; in southern Wisconsin) launched a new delivery model, “TEAM (Together Each person Achieves More) Primary Care,” as part of a quality improvement project to enhance the delivery experience for the patient, physician, and medical assistant (MA). New work flows, roles, and responsibilities were designed to reduce cycle time, increase patient time with physicians and staff, and reduce patient wait times.

Methods The new model increased the ratio of MAs to physicians from a baseline MA:MD ratio of 1:1 to 3:2, and trained MAs to assume expanded roles during exam-room entry and discharge, including assisting with documentation during the patient visit. A process engineer timed patient visits. The process engineer and a human resources associate conducted surveys to assess the level of satisfaction for patients, physicians, and MAs.

Results Cycle time decreased by a mean of 6 minutes, from 44 to 38 minutes per patient; time with staff increased a mean of 2 minutes, from 24 to 26 minutes per patient; and waiting time decreased from 9 to 2 minutes per patient. Qualitative interviews with patients, physicians, and MAs identified a high level of satisfaction with the new model.

Conclusion The higher staffing ratios and expanded roles for MAs in the new model improved workflow, increased the face time between patients and their physician and MA, and decreased patient wait times. The TEAM model also appeared to improve patient, physician, and MA satisfaction. We faced many challenges while implementing the new model, which could be further evaluated during wide adoption.

In recent years, we observed that our physicians, nurses, and medical assistants (MAs) appeared to be spending more time on administrative and clerical tasks—including tasks in the exam room with the patient—and less time engaged in direct patient care.^1,2 We recognized these factors contribute to burnout and threaten staff retention and anticipated that a new model would improve physician time spent in direct patient care, decrease the demands of administrative tasks, and increase patient, physician, and MA satisfaction.^3-6 Burnout, known to affect more than half of US physicians, has a negative impact on quality of care and patient safety and satisfaction.^7-11 Improving workflow has been shown to reduce burnout.¹²

Watertown Regional Medical Center (WRMC) is a small, financially stable integrated delivery system in rural southern Wisconsin, composed of a 90-bed hospital, 10 primary care clinics (7 owned and 3 affiliated), and 24 employed physicians in 9 specialties. Two clinics within WRMC launched a new delivery model, “TEAM (Together Each person Achieves More) Primary Care,” to improve the delivery experience for the entire team, defined as the patient, physician, and MA. New workflows, roles, and responsibilities were designed to reduce cycle time (the total amount of time patients spent in the clinic from check-in to check-out), increase the total time a patient spent with staff (physician and MA or in point-of-care testing and radiology), and reduce the total time a patient spent waiting.¹³

We describe here WRMC’s experience in developing and implementing workflow improvements as a means of reducing burnout and improving satisfaction.

Continue to: METHODS

 

METHODS

We selected 2 WRMC sites for TEAM re-engineering based on their experience with quality-improvement projects and perceived likelihood of success with a new transformation initiative. In early 2013, WRMC charged one physician (JM), 2 MAs, the clinic scheduler, and the clinic administrator with designing the details of the model including evaluation metrics. WRMC provided a .5 FTE process engineer (MS) to assist with the design and implementation of the model at no extra expense to the clinics. The model was implemented in late 2013 and into 2014 after regular TEAM planning meetings and observational visits to non-WRMC sites identified as examples of best practices in improving outpatient primary care patient satisfaction: Bellin Health (Green Bay, Wis); ThedaCare (Appleton, Wis); the University of Utah (Salt Lake City); and the University of Wisconsin Health Yahara Clinic (Madison, Wis).

TEAM model

The TEAM model—so named to create top-of-mind awareness of its benefits—increased the MA:MD ratio, maintained consistent team composition so that physician/MA teams learned to work together and become more efficient, and added new MA responsibilities. We trained MAs to assist with documentation in the exam room to ensure that physician time was spent in face-to-face direct patient care.^14-20 In these ways, we sought not only to increase patient satisfaction but also to enhance our own “joy in practice,” defined primarily by a high level of work-life satisfaction, a low level of burnout, and a feeling that the medical practice is fulfilling.²¹

In our traditional model, an MA retrieved the patient from the waiting room, conducted initial assessment in the exam room, and then left the patient to wait for the physician to enter. Once the physician entered and conducted the exam, the patient would be left alone again to wait for the MA to return. In our revised model (TABLE 1), we assigned one MA to each patient from arrival to discharge. After greeting the patient in the waiting room, the MA conducted an initial patient interview in the exam room, then remained in the room with the physician to document the visit. After the physician exited the exam room, the MA completed follow-up orders and provided the patient with a visit summary.

To facilitate consistency throughout the day, we designed a workflow similar to those created in lean models originally designed to increase efficiency in the manufacturing industry (TABLE 2). Visual and electronic cues triggered each step of the process and coordinated the movement of MAs and MDs. Cues included the conventional flag system outside each exam room, an electronic messaging system within the electronic health record (EHR) to indicate when a patient was ready to be seen, and a whiteboard in an area visible to all team members on which we wrote lab and radiology requests.

The TEAM model reduced wait time and increased staff interaction time with patients.

We experimented with the MA:MD ratio to identify the most effective and efficient team composition. On alternating weeks, we assigned one MA to one MD, 2 MAs to one MD, or 3 MAs to 2 MDs. Additionally, with the 2:1 MA:MD ratio, we varied the visit length in 2 tests; one spanning 30 minutes and the other 20 minutes. The MDs and MAs were seated at side-by-side workstations to make communication easier. We developed protocols and checklists that allowed MAs to enter health maintenance orders and conduct point-of-care testing before the physician entered the room. Such details included immunization management, strep screens, urine analyses, diabetic foot exams, extremity x-ray films, and mammogram and colonoscopy referrals.

Continue to: To prepare MAs...

 

To prepare MAs, we obtained special permission for team documentation from our Chief Information Officer and developed associated policies and procedures. A physician assistant (PA) trained each MA, introducing the structure and content of subjective, objective, assessment, and plan (SOAP) notes. Training was continuous, as PAs provided feedback when MAs began team documentation. The MAs documented visits using templates, free form, and quick text. We measured visit cycle-time, face time with staff, and patient waiting times. A process engineer with a stopwatch observed and timed the flow (but did not enter the exam room). We also conducted patient interviews immediately post-visit and administered anonymous questionnaires to clinic staff at different phases of the model. Physicians and MAs met weekly to evaluate the design.

We used qualitative interviews of patients, physicians, and MAs to identify the level of satisfaction with the new model. During the first week of implementation, a nurse and our process engineer conducted brief in-person surveys with approximately 20 post-visit patients. Patients, chosen by convenience, were asked if the visit addressed their concerns, whether they left with a thorough understanding of next steps, and if their wait time was acceptable. Twice during the implementation phase, a human resources associate distributed 9-item anonymous questionnaires to staff members during scheduled department meetings.

RESULTS

Times per activity with different MA:MD ratios and visit lengths are shown in TABLE 3. After 6 months, cycle time decreased by a mean of 6 minutes, from 44 to 38 minutes per patient; time with staff increased by a mean of 2 minutes, from 24 to 26 minutes per patient; and wait time decreased by a mean of 7 minutes, from 9 to 2 minutes per patient. We concluded the MA:MD ratio of 3:2 was most efficient because the 2:1 model left MAs with excess non-patient time.

Our delivery model received consistently positive comments from patients. Many expressed gratitude for the extra set of ears and eyes guiding them through the process. One recalled the “old days” when a nurse joined the doctor in the exam room. Another appreciated that both the MA and physician could answer follow-up questions over the phone.

Employee satisfaction

Surveys to assess satisfaction were distributed to all employees whether they were involved in the new model or not. Sixteen employees responded to the pre-implementation questionnaire and 18 responded to the post-implementation one distributed 7 months later. The questionnaires showed an increase in employee satisfaction scores from 3.70 to 3.89 on a 5-point Likert scale, with 5 ranking highest. “I am learning from [Dr. Milford] and understanding things more fully,” wrote one respondent. Another said, “Dr. Milford and his clinical support staff are less stressed.” Individual observations such as, “I can leave sooner with less work left to do,” and “All documentation is done before [the] patient leaves,” reflect the reduction in time that patient records remained open or incomplete. Some physicians reported a reduction in at-home or after-hours work, from about 2 to 4 hours per day to approximately one hour per day.

Continue to: Additional outcomes

 

 

Additional outcomes

The TEAM model allowed us to more easily integrate new initiatives into our practice and meet quality metrics by placing needed components within our workflow and checklist. For example, achieving Stage II Meaningful Use measures required that we print and furnish patients with a visit summary and a reminder to access our portal; something we easily incorporated into the MAs’ expanded responsibilities. We also met specific predetermined quality metrics that were part of a payment-withhold program. During the study period, we achieved scores at the 90th percentile and earned back our total withhold.

Finally, more of our patients completed advanced care planning discussions than the other 7 sites in our Honoring Choices Wisconsin cohort. This was achieved not only by integrating the process into our checklist, but because the MAs observed the MD-led patient conversations which they then emulated, presenting the advanced care planning information to patients before or after MD time with the patient.

Errors and defects in care

With ongoing provider guidance and reinforcement, MAs became integral members of the primary care team. They were empowered through protocols to manage and order health maintenance testing and provide needed immunizations. They also began to identify potentially overlooked aspects of care. For example, MAs prompted physicians to retake vital signs, adjust medications, order labs, discuss previous lab results, and pursue specialty referrals or follow-up care.

Billing

Although we tracked billing, the TEAM model was not designed to influence revenue. We noted no significant change in level of evaluation and management billed regardless of staffing ratio. While our panel size increased as we implemented the new process, this change could have been due to normal variation. We do see opportunity to affect future billing by having coders train MAs, which could enhance documentation and increase revenue.

DISCUSSION

The TEAM Primary Care model reduced the time our patients sat unattended, increased our opportunities to meaningfully interact with them, and seemed to reduce our administrative load. By identifying and implementing ways to work as a more cohesive, interconnected unit, we began to address our work as a team rather than as individuals. After implementing the model, we noted several instances where the MAs caught potential errors in care, although we did not consistently track or measure changes in the rate of these occurrences.

Continue to: Achieving these results also came with...

 

Achieving these results also came with challenges. Investing in and maintaining a new model opened our eyes to unforeseen inconsistencies in our staff profile and to the cost and administrative support needed for implementation. Moreover, our entire team (patients, MAs, and physicians) had to undergo a major cultural shift to adopt a new model.

Personnel variation

We discovered that implementing and sustaining organization change is highly dependent on constancy in human resources. When one team member was on vacation, sick, or leaving the practice, the process tended to fall apart. Hiring replacements and training employees well enough to fill in at a moment’s notice proved difficult. Bringing new employees into this process was also labor intensive. Despite team members being very engaged in change, these staffing inconsistencies caused significant stress and necessitated pauses in the implementation of the new model (reflected in the timeline of our measures). Larger organizational buy-in and support would allow us to hire and train a larger pool of MAs in anticipation of these fluctuations.

Cost

Our small, rural family practice took advantage of WRMC’s Primary Care Transformation project and the half-time process engineer and additional MA they provided. We question whether this model could be implemented without such support. While a process engineer might not prove necessary, expertise in process improvement is vital to help design and measure workflow and to identify opportunities for improvement.

Cultural change

Adopting a new model required asking every member of the team (patient, MA, and physician) to accommodate change and tolerate disruption. We anticipated patients might resist having an additional person in the room. All patients were informed of our new model at the beginning of the visit and told they could opt out. While we did not document patient resistance, JM recalled only 2 patients who expressed a desire not to have the MA present because of personal and sensitive issues. It’s possible some patients did not feel comfortable opting out. But many patients expressed gratitude for having an extra set of ears and eyes to guide them through the visit.

With the TEAM model, we noted several instances where the MAs caught potential errors in care.

It was more challenging to support MAs as they stepped out of their comfort zone to assist with documentation. It took time for MAs to grow accustomed to the protocols and checklists essential to our workflow. Without protocols, any point-of-care testing that could have been completed at the beginning of the appointment had to be done at the end. This disrupted our workflow and increased patient wait times.

Continue to: We correctly predicted MAs would have...

 

We correctly predicted MAs would have difficulty documenting the assessment, plan, and medical decision making. We discovered that MAs more easily categorized and articulated information when we reframed the assessment and plan in first-person and placed it under “Patient instructions.” For this to occur, physicians had to learn to accurately articulate their thought process and instructions to the patient.

When training was provided, as previously described, MAs grasped the subjective section quickly. Surprisingly, they had most difficulty understanding terminology within the objective section. In the future, we would avert this problem by working closely with the human resource department. We believe there should be a newly defined position and additional training for MAs in these roles, since duties such as patient-coaching and documentation assistance may warrant separate certification.

Limitations

Our findings should be interpreted in light of several limitations. Implementing the new model was carried out in a single organization. The patients who were selected and agreed to be interviewed may have differed from the patient population as a whole. We did not measure some important outcomes, such as cost effectiveness and patient morbidity. We did not analyze the data to determine whether the apparent improvements in wait time and cycle time were statistically significant. In addition, measurement of any adverse effects was beyond the scope of this study.

Looking forward

The traditional model of physicians working individually with minimal support staff is no longer viable. To echo our co-author (CAS)’s recent statements on physician dissatisfaction, “The days of hero medicine, with the doctor doing it all, belong in the past.”²¹ The new model appeared to alleviate some administrative burdens and increase physician time with patients. Pressures to achieve quality measures and growing administrative tasks have altered the role and responsibilities of each member of the team.

It's possible some patients might have felt uneasy with the TEAM approach, but many expressed gratitude for an extra set of ears and eyes to guide them through the visit.

Any sustainable system must address the larger crisis of physician dissatisfaction.^7,22 We cannot focus on a single perspective—patient, physician, or MA—at the expense of the system as a whole. If the health care system is to resolve the epidemic of burnout and physician dissatisfaction, new approaches to patient care must be imagined and realized. Although we faced many challenges in implementing and evaluating the TEAM model, attempts to overcome these challenges appear justified because of our overall favorable impression of it. Innovations like the TEAM Primary Care model may help us improve the well-being of not just our patients but also our health professionals and the health care industry as a whole.

CORRESPONDENCE
James Milford, MD, Three Oaks Health, S.C., 480 Village Walk Lane, Suite F, Johnson Creek, WI 53038; [email protected].

SUPPORT
Although the Watertown Regional Medical Center has provided general funding for its Primary Care Transformation project, no dollars were specifically earmarked for the TEAM Primary Care process. Support for editorial services in preparing this article was provided by Dr. James Milford.

PRIOR PRESENTATIONS
Co-author Michael R. Strasser, MPA, presented this project at the 2015 i-PrACTISE conference in Madison, Wis, April 12-14, 2015. http://www.fammed.wisc.edu/i-practise/. The proceedings were not published or recorded.

ACKNOWLEDGMENT
We thank Annalynn Skipper and Masarah Van Eyck for their valuable edits.

ABSTRACT

Purpose In 2013-14, 2 clinics in the Watertown Regional Medical Center (WRMC; in southern Wisconsin) launched a new delivery model, “TEAM (Together Each person Achieves More) Primary Care,” as part of a quality improvement project to enhance the delivery experience for the patient, physician, and medical assistant (MA). New work flows, roles, and responsibilities were designed to reduce cycle time, increase patient time with physicians and staff, and reduce patient wait times.

Methods The new model increased the ratio of MAs to physicians from a baseline MA:MD ratio of 1:1 to 3:2, and trained MAs to assume expanded roles during exam-room entry and discharge, including assisting with documentation during the patient visit. A process engineer timed patient visits. The process engineer and a human resources associate conducted surveys to assess the level of satisfaction for patients, physicians, and MAs.

Results Cycle time decreased by a mean of 6 minutes, from 44 to 38 minutes per patient; time with staff increased a mean of 2 minutes, from 24 to 26 minutes per patient; and waiting time decreased from 9 to 2 minutes per patient. Qualitative interviews with patients, physicians, and MAs identified a high level of satisfaction with the new model.

Conclusion The higher staffing ratios and expanded roles for MAs in the new model improved workflow, increased the face time between patients and their physician and MA, and decreased patient wait times. The TEAM model also appeared to improve patient, physician, and MA satisfaction. We faced many challenges while implementing the new model, which could be further evaluated during wide adoption.

In recent years, we observed that our physicians, nurses, and medical assistants (MAs) appeared to be spending more time on administrative and clerical tasks—including tasks in the exam room with the patient—and less time engaged in direct patient care.^1,2 We recognized these factors contribute to burnout and threaten staff retention and anticipated that a new model would improve physician time spent in direct patient care, decrease the demands of administrative tasks, and increase patient, physician, and MA satisfaction.^3-6 Burnout, known to affect more than half of US physicians, has a negative impact on quality of care and patient safety and satisfaction.^7-11 Improving workflow has been shown to reduce burnout.¹²

Watertown Regional Medical Center (WRMC) is a small, financially stable integrated delivery system in rural southern Wisconsin, composed of a 90-bed hospital, 10 primary care clinics (7 owned and 3 affiliated), and 24 employed physicians in 9 specialties. Two clinics within WRMC launched a new delivery model, “TEAM (Together Each person Achieves More) Primary Care,” to improve the delivery experience for the entire team, defined as the patient, physician, and MA. New workflows, roles, and responsibilities were designed to reduce cycle time (the total amount of time patients spent in the clinic from check-in to check-out), increase the total time a patient spent with staff (physician and MA or in point-of-care testing and radiology), and reduce the total time a patient spent waiting.¹³

We describe here WRMC’s experience in developing and implementing workflow improvements as a means of reducing burnout and improving satisfaction.

Continue to: METHODS

 

METHODS

We selected 2 WRMC sites for TEAM re-engineering based on their experience with quality-improvement projects and perceived likelihood of success with a new transformation initiative. In early 2013, WRMC charged one physician (JM), 2 MAs, the clinic scheduler, and the clinic administrator with designing the details of the model including evaluation metrics. WRMC provided a .5 FTE process engineer (MS) to assist with the design and implementation of the model at no extra expense to the clinics. The model was implemented in late 2013 and into 2014 after regular TEAM planning meetings and observational visits to non-WRMC sites identified as examples of best practices in improving outpatient primary care patient satisfaction: Bellin Health (Green Bay, Wis); ThedaCare (Appleton, Wis); the University of Utah (Salt Lake City); and the University of Wisconsin Health Yahara Clinic (Madison, Wis).

TEAM model

The TEAM model—so named to create top-of-mind awareness of its benefits—increased the MA:MD ratio, maintained consistent team composition so that physician/MA teams learned to work together and become more efficient, and added new MA responsibilities. We trained MAs to assist with documentation in the exam room to ensure that physician time was spent in face-to-face direct patient care.^14-20 In these ways, we sought not only to increase patient satisfaction but also to enhance our own “joy in practice,” defined primarily by a high level of work-life satisfaction, a low level of burnout, and a feeling that the medical practice is fulfilling.²¹

In our traditional model, an MA retrieved the patient from the waiting room, conducted initial assessment in the exam room, and then left the patient to wait for the physician to enter. Once the physician entered and conducted the exam, the patient would be left alone again to wait for the MA to return. In our revised model (TABLE 1), we assigned one MA to each patient from arrival to discharge. After greeting the patient in the waiting room, the MA conducted an initial patient interview in the exam room, then remained in the room with the physician to document the visit. After the physician exited the exam room, the MA completed follow-up orders and provided the patient with a visit summary.

To facilitate consistency throughout the day, we designed a workflow similar to those created in lean models originally designed to increase efficiency in the manufacturing industry (TABLE 2). Visual and electronic cues triggered each step of the process and coordinated the movement of MAs and MDs. Cues included the conventional flag system outside each exam room, an electronic messaging system within the electronic health record (EHR) to indicate when a patient was ready to be seen, and a whiteboard in an area visible to all team members on which we wrote lab and radiology requests.

The TEAM model reduced wait time and increased staff interaction time with patients.

We experimented with the MA:MD ratio to identify the most effective and efficient team composition. On alternating weeks, we assigned one MA to one MD, 2 MAs to one MD, or 3 MAs to 2 MDs. Additionally, with the 2:1 MA:MD ratio, we varied the visit length in 2 tests; one spanning 30 minutes and the other 20 minutes. The MDs and MAs were seated at side-by-side workstations to make communication easier. We developed protocols and checklists that allowed MAs to enter health maintenance orders and conduct point-of-care testing before the physician entered the room. Such details included immunization management, strep screens, urine analyses, diabetic foot exams, extremity x-ray films, and mammogram and colonoscopy referrals.

Continue to: To prepare MAs...

 

To prepare MAs, we obtained special permission for team documentation from our Chief Information Officer and developed associated policies and procedures. A physician assistant (PA) trained each MA, introducing the structure and content of subjective, objective, assessment, and plan (SOAP) notes. Training was continuous, as PAs provided feedback when MAs began team documentation. The MAs documented visits using templates, free form, and quick text. We measured visit cycle-time, face time with staff, and patient waiting times. A process engineer with a stopwatch observed and timed the flow (but did not enter the exam room). We also conducted patient interviews immediately post-visit and administered anonymous questionnaires to clinic staff at different phases of the model. Physicians and MAs met weekly to evaluate the design.

We used qualitative interviews of patients, physicians, and MAs to identify the level of satisfaction with the new model. During the first week of implementation, a nurse and our process engineer conducted brief in-person surveys with approximately 20 post-visit patients. Patients, chosen by convenience, were asked if the visit addressed their concerns, whether they left with a thorough understanding of next steps, and if their wait time was acceptable. Twice during the implementation phase, a human resources associate distributed 9-item anonymous questionnaires to staff members during scheduled department meetings.

RESULTS

Times per activity with different MA:MD ratios and visit lengths are shown in TABLE 3. After 6 months, cycle time decreased by a mean of 6 minutes, from 44 to 38 minutes per patient; time with staff increased by a mean of 2 minutes, from 24 to 26 minutes per patient; and wait time decreased by a mean of 7 minutes, from 9 to 2 minutes per patient. We concluded the MA:MD ratio of 3:2 was most efficient because the 2:1 model left MAs with excess non-patient time.

Our delivery model received consistently positive comments from patients. Many expressed gratitude for the extra set of ears and eyes guiding them through the process. One recalled the “old days” when a nurse joined the doctor in the exam room. Another appreciated that both the MA and physician could answer follow-up questions over the phone.

Employee satisfaction

Surveys to assess satisfaction were distributed to all employees whether they were involved in the new model or not. Sixteen employees responded to the pre-implementation questionnaire and 18 responded to the post-implementation one distributed 7 months later. The questionnaires showed an increase in employee satisfaction scores from 3.70 to 3.89 on a 5-point Likert scale, with 5 ranking highest. “I am learning from [Dr. Milford] and understanding things more fully,” wrote one respondent. Another said, “Dr. Milford and his clinical support staff are less stressed.” Individual observations such as, “I can leave sooner with less work left to do,” and “All documentation is done before [the] patient leaves,” reflect the reduction in time that patient records remained open or incomplete. Some physicians reported a reduction in at-home or after-hours work, from about 2 to 4 hours per day to approximately one hour per day.

Continue to: Additional outcomes

 

 

Additional outcomes

The TEAM model allowed us to more easily integrate new initiatives into our practice and meet quality metrics by placing needed components within our workflow and checklist. For example, achieving Stage II Meaningful Use measures required that we print and furnish patients with a visit summary and a reminder to access our portal; something we easily incorporated into the MAs’ expanded responsibilities. We also met specific predetermined quality metrics that were part of a payment-withhold program. During the study period, we achieved scores at the 90th percentile and earned back our total withhold.

Finally, more of our patients completed advanced care planning discussions than the other 7 sites in our Honoring Choices Wisconsin cohort. This was achieved not only by integrating the process into our checklist, but because the MAs observed the MD-led patient conversations which they then emulated, presenting the advanced care planning information to patients before or after MD time with the patient.

Errors and defects in care

With ongoing provider guidance and reinforcement, MAs became integral members of the primary care team. They were empowered through protocols to manage and order health maintenance testing and provide needed immunizations. They also began to identify potentially overlooked aspects of care. For example, MAs prompted physicians to retake vital signs, adjust medications, order labs, discuss previous lab results, and pursue specialty referrals or follow-up care.

Billing

Although we tracked billing, the TEAM model was not designed to influence revenue. We noted no significant change in level of evaluation and management billed regardless of staffing ratio. While our panel size increased as we implemented the new process, this change could have been due to normal variation. We do see opportunity to affect future billing by having coders train MAs, which could enhance documentation and increase revenue.

DISCUSSION

The TEAM Primary Care model reduced the time our patients sat unattended, increased our opportunities to meaningfully interact with them, and seemed to reduce our administrative load. By identifying and implementing ways to work as a more cohesive, interconnected unit, we began to address our work as a team rather than as individuals. After implementing the model, we noted several instances where the MAs caught potential errors in care, although we did not consistently track or measure changes in the rate of these occurrences.

Continue to: Achieving these results also came with...

 

Achieving these results also came with challenges. Investing in and maintaining a new model opened our eyes to unforeseen inconsistencies in our staff profile and to the cost and administrative support needed for implementation. Moreover, our entire team (patients, MAs, and physicians) had to undergo a major cultural shift to adopt a new model.

Personnel variation

We discovered that implementing and sustaining organization change is highly dependent on constancy in human resources. When one team member was on vacation, sick, or leaving the practice, the process tended to fall apart. Hiring replacements and training employees well enough to fill in at a moment’s notice proved difficult. Bringing new employees into this process was also labor intensive. Despite team members being very engaged in change, these staffing inconsistencies caused significant stress and necessitated pauses in the implementation of the new model (reflected in the timeline of our measures). Larger organizational buy-in and support would allow us to hire and train a larger pool of MAs in anticipation of these fluctuations.

Cost

Our small, rural family practice took advantage of WRMC’s Primary Care Transformation project and the half-time process engineer and additional MA they provided. We question whether this model could be implemented without such support. While a process engineer might not prove necessary, expertise in process improvement is vital to help design and measure workflow and to identify opportunities for improvement.

Cultural change

Adopting a new model required asking every member of the team (patient, MA, and physician) to accommodate change and tolerate disruption. We anticipated patients might resist having an additional person in the room. All patients were informed of our new model at the beginning of the visit and told they could opt out. While we did not document patient resistance, JM recalled only 2 patients who expressed a desire not to have the MA present because of personal and sensitive issues. It’s possible some patients did not feel comfortable opting out. But many patients expressed gratitude for having an extra set of ears and eyes to guide them through the visit.

With the TEAM model, we noted several instances where the MAs caught potential errors in care.

It was more challenging to support MAs as they stepped out of their comfort zone to assist with documentation. It took time for MAs to grow accustomed to the protocols and checklists essential to our workflow. Without protocols, any point-of-care testing that could have been completed at the beginning of the appointment had to be done at the end. This disrupted our workflow and increased patient wait times.

Continue to: We correctly predicted MAs would have...

 

We correctly predicted MAs would have difficulty documenting the assessment, plan, and medical decision making. We discovered that MAs more easily categorized and articulated information when we reframed the assessment and plan in first-person and placed it under “Patient instructions.” For this to occur, physicians had to learn to accurately articulate their thought process and instructions to the patient.

When training was provided, as previously described, MAs grasped the subjective section quickly. Surprisingly, they had most difficulty understanding terminology within the objective section. In the future, we would avert this problem by working closely with the human resource department. We believe there should be a newly defined position and additional training for MAs in these roles, since duties such as patient-coaching and documentation assistance may warrant separate certification.

Limitations

Our findings should be interpreted in light of several limitations. Implementing the new model was carried out in a single organization. The patients who were selected and agreed to be interviewed may have differed from the patient population as a whole. We did not measure some important outcomes, such as cost effectiveness and patient morbidity. We did not analyze the data to determine whether the apparent improvements in wait time and cycle time were statistically significant. In addition, measurement of any adverse effects was beyond the scope of this study.

Looking forward

The traditional model of physicians working individually with minimal support staff is no longer viable. To echo our co-author (CAS)’s recent statements on physician dissatisfaction, “The days of hero medicine, with the doctor doing it all, belong in the past.”²¹ The new model appeared to alleviate some administrative burdens and increase physician time with patients. Pressures to achieve quality measures and growing administrative tasks have altered the role and responsibilities of each member of the team.

It's possible some patients might have felt uneasy with the TEAM approach, but many expressed gratitude for an extra set of ears and eyes to guide them through the visit.

Any sustainable system must address the larger crisis of physician dissatisfaction.^7,22 We cannot focus on a single perspective—patient, physician, or MA—at the expense of the system as a whole. If the health care system is to resolve the epidemic of burnout and physician dissatisfaction, new approaches to patient care must be imagined and realized. Although we faced many challenges in implementing and evaluating the TEAM model, attempts to overcome these challenges appear justified because of our overall favorable impression of it. Innovations like the TEAM Primary Care model may help us improve the well-being of not just our patients but also our health professionals and the health care industry as a whole.

CORRESPONDENCE
James Milford, MD, Three Oaks Health, S.C., 480 Village Walk Lane, Suite F, Johnson Creek, WI 53038; [email protected].

SUPPORT
Although the Watertown Regional Medical Center has provided general funding for its Primary Care Transformation project, no dollars were specifically earmarked for the TEAM Primary Care process. Support for editorial services in preparing this article was provided by Dr. James Milford.

PRIOR PRESENTATIONS
Co-author Michael R. Strasser, MPA, presented this project at the 2015 i-PrACTISE conference in Madison, Wis, April 12-14, 2015. http://www.fammed.wisc.edu/i-practise/. The proceedings were not published or recorded.

ACKNOWLEDGMENT
We thank Annalynn Skipper and Masarah Van Eyck for their valuable edits.

ABSTRACT

Purpose In 2013-14, 2 clinics in the Watertown Regional Medical Center (WRMC; in southern Wisconsin) launched a new delivery model, “TEAM (Together Each person Achieves More) Primary Care,” as part of a quality improvement project to enhance the delivery experience for the patient, physician, and medical assistant (MA). New work flows, roles, and responsibilities were designed to reduce cycle time, increase patient time with physicians and staff, and reduce patient wait times.

Methods The new model increased the ratio of MAs to physicians from a baseline MA:MD ratio of 1:1 to 3:2, and trained MAs to assume expanded roles during exam-room entry and discharge, including assisting with documentation during the patient visit. A process engineer timed patient visits. The process engineer and a human resources associate conducted surveys to assess the level of satisfaction for patients, physicians, and MAs.

Results Cycle time decreased by a mean of 6 minutes, from 44 to 38 minutes per patient; time with staff increased a mean of 2 minutes, from 24 to 26 minutes per patient; and waiting time decreased from 9 to 2 minutes per patient. Qualitative interviews with patients, physicians, and MAs identified a high level of satisfaction with the new model.

Conclusion The higher staffing ratios and expanded roles for MAs in the new model improved workflow, increased the face time between patients and their physician and MA, and decreased patient wait times. The TEAM model also appeared to improve patient, physician, and MA satisfaction. We faced many challenges while implementing the new model, which could be further evaluated during wide adoption.

In recent years, we observed that our physicians, nurses, and medical assistants (MAs) appeared to be spending more time on administrative and clerical tasks—including tasks in the exam room with the patient—and less time engaged in direct patient care.^1,2 We recognized these factors contribute to burnout and threaten staff retention and anticipated that a new model would improve physician time spent in direct patient care, decrease the demands of administrative tasks, and increase patient, physician, and MA satisfaction.^3-6 Burnout, known to affect more than half of US physicians, has a negative impact on quality of care and patient safety and satisfaction.^7-11 Improving workflow has been shown to reduce burnout.¹²

Watertown Regional Medical Center (WRMC) is a small, financially stable integrated delivery system in rural southern Wisconsin, composed of a 90-bed hospital, 10 primary care clinics (7 owned and 3 affiliated), and 24 employed physicians in 9 specialties. Two clinics within WRMC launched a new delivery model, “TEAM (Together Each person Achieves More) Primary Care,” to improve the delivery experience for the entire team, defined as the patient, physician, and MA. New workflows, roles, and responsibilities were designed to reduce cycle time (the total amount of time patients spent in the clinic from check-in to check-out), increase the total time a patient spent with staff (physician and MA or in point-of-care testing and radiology), and reduce the total time a patient spent waiting.¹³

We describe here WRMC’s experience in developing and implementing workflow improvements as a means of reducing burnout and improving satisfaction.

Continue to: METHODS

 

METHODS

We selected 2 WRMC sites for TEAM re-engineering based on their experience with quality-improvement projects and perceived likelihood of success with a new transformation initiative. In early 2013, WRMC charged one physician (JM), 2 MAs, the clinic scheduler, and the clinic administrator with designing the details of the model including evaluation metrics. WRMC provided a .5 FTE process engineer (MS) to assist with the design and implementation of the model at no extra expense to the clinics. The model was implemented in late 2013 and into 2014 after regular TEAM planning meetings and observational visits to non-WRMC sites identified as examples of best practices in improving outpatient primary care patient satisfaction: Bellin Health (Green Bay, Wis); ThedaCare (Appleton, Wis); the University of Utah (Salt Lake City); and the University of Wisconsin Health Yahara Clinic (Madison, Wis).

TEAM model

The TEAM model—so named to create top-of-mind awareness of its benefits—increased the MA:MD ratio, maintained consistent team composition so that physician/MA teams learned to work together and become more efficient, and added new MA responsibilities. We trained MAs to assist with documentation in the exam room to ensure that physician time was spent in face-to-face direct patient care.^14-20 In these ways, we sought not only to increase patient satisfaction but also to enhance our own “joy in practice,” defined primarily by a high level of work-life satisfaction, a low level of burnout, and a feeling that the medical practice is fulfilling.²¹

In our traditional model, an MA retrieved the patient from the waiting room, conducted initial assessment in the exam room, and then left the patient to wait for the physician to enter. Once the physician entered and conducted the exam, the patient would be left alone again to wait for the MA to return. In our revised model (TABLE 1), we assigned one MA to each patient from arrival to discharge. After greeting the patient in the waiting room, the MA conducted an initial patient interview in the exam room, then remained in the room with the physician to document the visit. After the physician exited the exam room, the MA completed follow-up orders and provided the patient with a visit summary.

To facilitate consistency throughout the day, we designed a workflow similar to those created in lean models originally designed to increase efficiency in the manufacturing industry (TABLE 2). Visual and electronic cues triggered each step of the process and coordinated the movement of MAs and MDs. Cues included the conventional flag system outside each exam room, an electronic messaging system within the electronic health record (EHR) to indicate when a patient was ready to be seen, and a whiteboard in an area visible to all team members on which we wrote lab and radiology requests.

The TEAM model reduced wait time and increased staff interaction time with patients.

We experimented with the MA:MD ratio to identify the most effective and efficient team composition. On alternating weeks, we assigned one MA to one MD, 2 MAs to one MD, or 3 MAs to 2 MDs. Additionally, with the 2:1 MA:MD ratio, we varied the visit length in 2 tests; one spanning 30 minutes and the other 20 minutes. The MDs and MAs were seated at side-by-side workstations to make communication easier. We developed protocols and checklists that allowed MAs to enter health maintenance orders and conduct point-of-care testing before the physician entered the room. Such details included immunization management, strep screens, urine analyses, diabetic foot exams, extremity x-ray films, and mammogram and colonoscopy referrals.

Continue to: To prepare MAs...

 

To prepare MAs, we obtained special permission for team documentation from our Chief Information Officer and developed associated policies and procedures. A physician assistant (PA) trained each MA, introducing the structure and content of subjective, objective, assessment, and plan (SOAP) notes. Training was continuous, as PAs provided feedback when MAs began team documentation. The MAs documented visits using templates, free form, and quick text. We measured visit cycle-time, face time with staff, and patient waiting times. A process engineer with a stopwatch observed and timed the flow (but did not enter the exam room). We also conducted patient interviews immediately post-visit and administered anonymous questionnaires to clinic staff at different phases of the model. Physicians and MAs met weekly to evaluate the design.

We used qualitative interviews of patients, physicians, and MAs to identify the level of satisfaction with the new model. During the first week of implementation, a nurse and our process engineer conducted brief in-person surveys with approximately 20 post-visit patients. Patients, chosen by convenience, were asked if the visit addressed their concerns, whether they left with a thorough understanding of next steps, and if their wait time was acceptable. Twice during the implementation phase, a human resources associate distributed 9-item anonymous questionnaires to staff members during scheduled department meetings.

RESULTS

Times per activity with different MA:MD ratios and visit lengths are shown in TABLE 3. After 6 months, cycle time decreased by a mean of 6 minutes, from 44 to 38 minutes per patient; time with staff increased by a mean of 2 minutes, from 24 to 26 minutes per patient; and wait time decreased by a mean of 7 minutes, from 9 to 2 minutes per patient. We concluded the MA:MD ratio of 3:2 was most efficient because the 2:1 model left MAs with excess non-patient time.

Our delivery model received consistently positive comments from patients. Many expressed gratitude for the extra set of ears and eyes guiding them through the process. One recalled the “old days” when a nurse joined the doctor in the exam room. Another appreciated that both the MA and physician could answer follow-up questions over the phone.

Employee satisfaction

Surveys to assess satisfaction were distributed to all employees whether they were involved in the new model or not. Sixteen employees responded to the pre-implementation questionnaire and 18 responded to the post-implementation one distributed 7 months later. The questionnaires showed an increase in employee satisfaction scores from 3.70 to 3.89 on a 5-point Likert scale, with 5 ranking highest. “I am learning from [Dr. Milford] and understanding things more fully,” wrote one respondent. Another said, “Dr. Milford and his clinical support staff are less stressed.” Individual observations such as, “I can leave sooner with less work left to do,” and “All documentation is done before [the] patient leaves,” reflect the reduction in time that patient records remained open or incomplete. Some physicians reported a reduction in at-home or after-hours work, from about 2 to 4 hours per day to approximately one hour per day.

Continue to: Additional outcomes

 

 

Additional outcomes

The TEAM model allowed us to more easily integrate new initiatives into our practice and meet quality metrics by placing needed components within our workflow and checklist. For example, achieving Stage II Meaningful Use measures required that we print and furnish patients with a visit summary and a reminder to access our portal; something we easily incorporated into the MAs’ expanded responsibilities. We also met specific predetermined quality metrics that were part of a payment-withhold program. During the study period, we achieved scores at the 90th percentile and earned back our total withhold.

Finally, more of our patients completed advanced care planning discussions than the other 7 sites in our Honoring Choices Wisconsin cohort. This was achieved not only by integrating the process into our checklist, but because the MAs observed the MD-led patient conversations which they then emulated, presenting the advanced care planning information to patients before or after MD time with the patient.

Errors and defects in care

With ongoing provider guidance and reinforcement, MAs became integral members of the primary care team. They were empowered through protocols to manage and order health maintenance testing and provide needed immunizations. They also began to identify potentially overlooked aspects of care. For example, MAs prompted physicians to retake vital signs, adjust medications, order labs, discuss previous lab results, and pursue specialty referrals or follow-up care.

Billing

Although we tracked billing, the TEAM model was not designed to influence revenue. We noted no significant change in level of evaluation and management billed regardless of staffing ratio. While our panel size increased as we implemented the new process, this change could have been due to normal variation. We do see opportunity to affect future billing by having coders train MAs, which could enhance documentation and increase revenue.

DISCUSSION

The TEAM Primary Care model reduced the time our patients sat unattended, increased our opportunities to meaningfully interact with them, and seemed to reduce our administrative load. By identifying and implementing ways to work as a more cohesive, interconnected unit, we began to address our work as a team rather than as individuals. After implementing the model, we noted several instances where the MAs caught potential errors in care, although we did not consistently track or measure changes in the rate of these occurrences.

Continue to: Achieving these results also came with...

 

Achieving these results also came with challenges. Investing in and maintaining a new model opened our eyes to unforeseen inconsistencies in our staff profile and to the cost and administrative support needed for implementation. Moreover, our entire team (patients, MAs, and physicians) had to undergo a major cultural shift to adopt a new model.

Personnel variation

We discovered that implementing and sustaining organization change is highly dependent on constancy in human resources. When one team member was on vacation, sick, or leaving the practice, the process tended to fall apart. Hiring replacements and training employees well enough to fill in at a moment’s notice proved difficult. Bringing new employees into this process was also labor intensive. Despite team members being very engaged in change, these staffing inconsistencies caused significant stress and necessitated pauses in the implementation of the new model (reflected in the timeline of our measures). Larger organizational buy-in and support would allow us to hire and train a larger pool of MAs in anticipation of these fluctuations.

Cost

Our small, rural family practice took advantage of WRMC’s Primary Care Transformation project and the half-time process engineer and additional MA they provided. We question whether this model could be implemented without such support. While a process engineer might not prove necessary, expertise in process improvement is vital to help design and measure workflow and to identify opportunities for improvement.

Cultural change

Adopting a new model required asking every member of the team (patient, MA, and physician) to accommodate change and tolerate disruption. We anticipated patients might resist having an additional person in the room. All patients were informed of our new model at the beginning of the visit and told they could opt out. While we did not document patient resistance, JM recalled only 2 patients who expressed a desire not to have the MA present because of personal and sensitive issues. It’s possible some patients did not feel comfortable opting out. But many patients expressed gratitude for having an extra set of ears and eyes to guide them through the visit.

With the TEAM model, we noted several instances where the MAs caught potential errors in care.

It was more challenging to support MAs as they stepped out of their comfort zone to assist with documentation. It took time for MAs to grow accustomed to the protocols and checklists essential to our workflow. Without protocols, any point-of-care testing that could have been completed at the beginning of the appointment had to be done at the end. This disrupted our workflow and increased patient wait times.

Continue to: We correctly predicted MAs would have...

 

We correctly predicted MAs would have difficulty documenting the assessment, plan, and medical decision making. We discovered that MAs more easily categorized and articulated information when we reframed the assessment and plan in first-person and placed it under “Patient instructions.” For this to occur, physicians had to learn to accurately articulate their thought process and instructions to the patient.

When training was provided, as previously described, MAs grasped the subjective section quickly. Surprisingly, they had most difficulty understanding terminology within the objective section. In the future, we would avert this problem by working closely with the human resource department. We believe there should be a newly defined position and additional training for MAs in these roles, since duties such as patient-coaching and documentation assistance may warrant separate certification.

Limitations

Our findings should be interpreted in light of several limitations. Implementing the new model was carried out in a single organization. The patients who were selected and agreed to be interviewed may have differed from the patient population as a whole. We did not measure some important outcomes, such as cost effectiveness and patient morbidity. We did not analyze the data to determine whether the apparent improvements in wait time and cycle time were statistically significant. In addition, measurement of any adverse effects was beyond the scope of this study.

Looking forward

The traditional model of physicians working individually with minimal support staff is no longer viable. To echo our co-author (CAS)’s recent statements on physician dissatisfaction, “The days of hero medicine, with the doctor doing it all, belong in the past.”²¹ The new model appeared to alleviate some administrative burdens and increase physician time with patients. Pressures to achieve quality measures and growing administrative tasks have altered the role and responsibilities of each member of the team.

It's possible some patients might have felt uneasy with the TEAM approach, but many expressed gratitude for an extra set of ears and eyes to guide them through the visit.

Any sustainable system must address the larger crisis of physician dissatisfaction.^7,22 We cannot focus on a single perspective—patient, physician, or MA—at the expense of the system as a whole. If the health care system is to resolve the epidemic of burnout and physician dissatisfaction, new approaches to patient care must be imagined and realized. Although we faced many challenges in implementing and evaluating the TEAM model, attempts to overcome these challenges appear justified because of our overall favorable impression of it. Innovations like the TEAM Primary Care model may help us improve the well-being of not just our patients but also our health professionals and the health care industry as a whole.

CORRESPONDENCE
James Milford, MD, Three Oaks Health, S.C., 480 Village Walk Lane, Suite F, Johnson Creek, WI 53038; [email protected].

SUPPORT
Although the Watertown Regional Medical Center has provided general funding for its Primary Care Transformation project, no dollars were specifically earmarked for the TEAM Primary Care process. Support for editorial services in preparing this article was provided by Dr. James Milford.

PRIOR PRESENTATIONS
Co-author Michael R. Strasser, MPA, presented this project at the 2015 i-PrACTISE conference in Madison, Wis, April 12-14, 2015. http://www.fammed.wisc.edu/i-practise/. The proceedings were not published or recorded.

ACKNOWLEDGMENT
We thank Annalynn Skipper and Masarah Van Eyck for their valuable edits.

ABSTRACT

Fiberglass casts are frequently valved to accommodate swelling following injury or surgery. The use of cast spacers has been recommended to bridge this gap between pressure reduction and cast strength, but no studies have assessed their effect on cast pressure.

We applied 30 long-arm fiberglass casts to adult volunteers, divided between a univalve group and a bivalve group. A pediatric blood pressure bladder was applied under the cast to simulate soft tissue swelling. Valved casts were secured using an elastic wrap, 10-mm cast spacer, or 15-mm cast spacer. Measurements of cast pressure and circumference were performed at each stage and compared on the basis of type of valve and securement.

Our results indicated that cast univalving resulted in an approximately 60% reduction in cast pressures, with a 75% reduction seen in the bivalve group. The addition of cast spacers resulted in significant pressure reductions for both valving groups. The univalve group secured with a 10-mm cast spacer produced reductions in cast pressure similar to those of the elastic-wrapped bivalve cast, both with the cast padding intact and with it released.

The use of cast spacers results in significant cast pressure reductions, regardless of valving technique. A univalved cast secured with a cast spacer can produce decreases in cast pressures similar to those seen with an elastic-wrapped bivalved cast, and it is a viable option for reducing cast pressure without compromising cast structural integrity with a bivalve technique.

Continue to: Complications following closed reduction...

 

Complications following closed reduction and casting of pediatric forearm fractures are rare, but they do occur. Arguably the most devastating of these complications is the risk of developing compartment syndrome or Volkmann contracture secondary to injury-associated swelling under a circumferential cast.^1-4 The peak in swelling can develop from 4 to 24 hours following the initial cast application,⁵ and as such, medical providers may not be able to identify it early because most children are discharged following closed reductions. For this reason, many providers implement prophylactic measures to minimize pressure-related complications.

A popular method for reducing pressure accumulation within a cast is to valve, or cut, the cast. Previous investigations have shown that cast valving results in significant reductions in cast pressure.^2,6-9 Bivalving a circumferential cast results in significantly greater reductions in cast pressure when compared with univalve techniques;^6,7,9 however, bivalving has also been shown to result in significant impairment in the structural integrity of the cast.¹⁰ An additional method to facilitate cast pressure reduction without impairing the structural integrity of the cast that accompanies a bivalve is to incorporate a cast spacer with a univalve technique to hold the split cast open.¹¹ Although this method is commonly used in clinical practice, its ability to mitigate cast pressures has not previously been investigated.

The goal of this study is to investigate the influence of incorporating cast spacers with valved long-arm casts. We hypothesized that cast spacers would provide a greater pressure reduction for both univalved and bivalved casts when compared with the use of an elastic wrap. Additionally, we proposed that by incorporating a cast spacer with a univalved cast, we could attain pressure reduction equivalent to that of a bivalved cast secured with an elastic wrap.

MATERIALS AND METHODS

Upon receiving approval from the Institutional Review Board, experimental testing began with the application of 30 total casts performed on uninjured adult human volunteers. Pressure readings were provided with the use of a bladder from a pediatric blood pressure cuff (Welch Allyn Inc), as previously described.⁶ The bladder was placed on the volar aspect of the volunteer’s forearm, held in place with a 3-in diameter cotton stockinet (3M). Cotton cast padding (Webril-Kendall) was applied, 3 in wide and 2 layers thick, and a long-arm cast was applied, 2 layers thick with 3-in wide fiberglass casting material (Scotchcast Plus Casting Tape; 3M).

Once the cast was applied and allowed to set, the blood pressure bladder was inflated to 100 mm Hg. After inflation, forearm cast circumference was measured at 2 set points, assessed at points 2 cm distal to the elbow flexor crease and 10 cm distal to the previous point (Figure 1). Using these data, we calculated estimated cast volume using the volumetric equation for a frustum. Following this point, casts were split into 2 experimental groups, univalve or bivalve, with 15 casts comprising each group. The univalve group consisted of a single cut along the dorsum of the extremity, and the bivalve group incorporated a second cut to the volar extremity. Cast valving was performed using an oscillating cast saw (Cast Vac; Stryker Instruments), with care taken to ensure the continuity of the underlying cast padding.

Continue to: Following valving, casts were secured via...

 

Following valving, casts were secured via 3 separate techniques: overwrap with a 3-in elastic wrap (Econo Wrap; Vitality Medical), application of two 10-mm and 15-mm cast spacers (CastWedge; DM Systems) (Figure 2). After securement, cast pressures were recorded, and circumference measurements were performed at the 2 previously identified points. The cast padding was then cut at the valve site and secured via the 3 listed techniques. Cast pressure and circumference measurements were performed at set time points (Figure 3). Changes in cast pressure were recorded in terms of the amount of change from the initial cast placement to account for differences in the size of volunteers’ forearms. Volumetric calculations were performed only for the spacer subgroups owing to the added material in the elastic wrap group. Estimated cast volume was calculated using the equation for volume of a frustum (Figure 4).

We used a 2-cast type (univalve and bivalve) by 4 securement subgroups (initial, elastic wrap, 10-mm spacer, and 15-mm spacer) design, with cast type serving as a between-subject measure and securement serving as a within-subject variable. An a priori power analysis showed that a minimum sample size of 15 subjects per condition should provide sufficient power of .80 and alpha set at .05, for a total of 30 casts. Statistical analyses were performed using IBM SPSS Statistics software version 21 (IBM). Experimental groups were analyzed using mixed-design analysis of variance (ANOVA). Post hoc comparisons between valving groups and cast securement were performed using Scheffe’s test to control for type II errors. Change in cast volume between the initial cast and cast spacers groups was compared using paired Student’s t tests. Statistical significance was predetermined as P < .05.

RESULTS

A summary of collected data for cast pressure and volume is detailed in Table 1, subdividing the variables on the basis of cast type and type of securement. Recorded pressures of the different subgroups are depicted in Figures 5 and 6 according to type of securement (initial, elastic wrap, 10-mm spacer, or 15-mm spacer). Results of the mixed-design ANOVA demonstrated significant differences between the initial cast pressure and univalve and bivalve groups (P < .05). There was a main effect for bivalve having lower pressure overall (F [1, 1)] = 3321.51, P < .001). There was also a main effect indicating that pressure was different for each type of securement (elastic wrap, 10-mm spacer, 15-mm spacer) (F [1, 28] = 538.54, P <. 01). Post hoc testing confirmed pressure decreased significantly, in descending order from elastic wrap, to 10-mm spacers, to 15-mm spacers (P < .05).

Table 1. Cumulative Data for Two Casting groups at Each Timepoint

Cast	Pressure	Standard Deviation	Volume
Univalve
Initial	100	---	2654.3
Elastic Wrap	39.47	3.33	---
10-mm Spacer	23.93	2.73	2708.23
15-mm Spacer	18.87	2.94	2734.86
Padding and Elastic Wrap	20.93	2.91	---
Padding and 10-mm Spacer	15.46	2.19	2733.24
Padding and 15-mm Spacer	0	---	2819.27
Bivalve
Initial	100	---	2839.3
Elastic Wrap	25.9	3.17	---
10-mm Spacer	16.53	2.32	3203.13
15-mm Spacer	13.6	2.74	3380.32
Padding and Elastic Wrap	12.67	1.95	---
Padding and 10-mm Spacer	0	---	3296.55
Padding and 15- mm Spacer	0	---	3438.67

Continue to: Table 2...

 

The summary of volumetric changes is listed in Table 2. The decrease in pressure correlated with an associated increase in cast volume, as demonstrated in Figure 7. The degree of increase in cast volume was more pronounced in the bivalve group (P < .001). The volume increased in the 15-mm group compared with the 10-mm group for both groups (P < .001) and increased for each spacer group with the release of the underlying padding (P < .05).

 

Table 2. Volumetric Data

Cast	Average Volumetric change (cm3)	Standard Deviation
Univalve
10-mm Spacer	175.6	65.4
15-mm Spacer	269.4	73.3
Padding and 10-mm Spacer	202.3	62.5
Padding and 15-mm Spacer	294.1	66.9
Bivalve
10-mm Spacer	363.7	67.2
15-mm Spacer	540.9	85.7
Padding and 10-mm Spacer	457.2	97.9
Padding and 15-mm Spacer	599.3	84.2

Analysis of the planned comparisons demonstrated no significant difference between the bivalve with elastic wrap and univalve with 10-mm spacer subgroups (t [28] = 1.85, P = .075, d = .68). In comparing the bivalve with elastic wrap group with the univalve and 15-mm spacer subgroup, the univalve group showed significantly lower pressures [t [28] = 6.32, P < .001, d = .2.31).

DISCUSSION

Valving of circumferential casting is a well-established technique to minimize potential pressure-related complications. Previous studies have demonstrated that univalving techniques produce a 65% reduction in cast pressure, whereas bivalving produces an 80% decrease.^6,7,9 Our results showed comparable pressure reductions of 75% with bivalving and 60% with univalving. The type of cast padding has been shown to have a significant effect on the cast pressure, favoring lower pressures with cotton padding over synthetic and waterproof padding, which, when released, can provide an additional 10% pressure reduction.^6,7

Although bivalving techniques are superior in pressure reduction, the reduction comes at the cost of the cast’s structural integrity. Crickard and colleagues¹⁰ performed a biomechanical assessment of the structural integrity by 3-point bending of casts following univalve and bivalve compared with an intact cast. The authors found that valving resulted in a significant decrease in the casts’ bending stiffness and load to failure, with bivalved casts demonstrating a significantly lower load to failure than univalved casts. One technique that has been used to enhance the pressure reduction in univalved casting techniques is the application of a cast spacer. Rang and colleagues¹¹ recommended this technique as part of a graded cast-splitting approach for the treatment of children’s fractures. This technique was applied to fractures with only modest anticipated swelling, which accounted for approximately 95% of casts applied in their children’s hospital. Our results support the use of cast spacers, demonstrating significant reduction in cast pressure in both univalve and bivalve techniques. Additionally, we found that a univalved cast with a 10-mm cast spacer provided pressure reduction similar to that of a bivalved cast.

The theory behind the application of cast spacers is that a split fiberglass cast will not remain open unless held in position.¹¹ Holding the cast open is less of a restraint to pressure reduction in bivalving techniques, because the split cast no longer has the contralateral intact hinge point to resist cast opening, demonstrated in the compromise in structural integrity seen with this technique.¹⁰ By maintaining the split cast in an opened position, the effective volume of the cast is increased, which allows for the reduction in cast pressure. This is demonstrated in our results indicating an increase in estimated cast volume with an associated incremental reduction in cast pressure with the application of incrementally sized cast spacers. Although this technique does have the potential for skin irritation caused by cast expansion, as well as local swelling at the cast window location, it is a cost-effective treatment method compared with overwrapping a bivalved cast, $1.55 for 1 cast spacer vs an estimated $200 for a forearm cast application.

This study is not without its limitations. Our model does not account for the soft tissue injury associated with forearm fractures. However, by using human volunteers, we were able to include the viscoelastic properties that are omitted with nonliving models, and our results do align with those of previous investigations regarding pressure change following valving. We did not incorporate a 3-point molding technique commonly used with reduction and casting of acute forearm fractures, owing to the lack of a standardized method for applying the mold to healthy volunteers. Although molding is necessary for most fractures in which valving is considered, we believe our data still provide valuable information. Additionally, valving of circumferential casts has not been shown, prospectively, to result in a reduction of cast-related compartment syndrome, maintenance of reduction, or need for surgery.^12,13 However, these results are reflective of reliable patients who completed the requisite follow-up care necessary for inclusion in a randomized controlled trial and may be applicable to unreliable patients or patient situations, a setting in which the compromise in cast structural integrity may be unacceptable.

CONCLUSION

We demonstrated that incorporating cast spacers into valved long-arm casts provides pressure reduction comparable to that achieved with the use of an elastic wrap. The addition of a 10-mm cast spacer to a univalved long-arm cast provides pressure reduction equivalent to that of a bivalved cast secured with an elastic wrap. A univalved cast secured with a cast spacer is a viable option for treatment of displaced pediatric forearm fractures, without compromising the cast’s structural integrity as required with bivalved techniques.

This paper will be judged for the Resident Writer’s Award.

ABSTRACT

Fiberglass casts are frequently valved to accommodate swelling following injury or surgery. The use of cast spacers has been recommended to bridge this gap between pressure reduction and cast strength, but no studies have assessed their effect on cast pressure.

We applied 30 long-arm fiberglass casts to adult volunteers, divided between a univalve group and a bivalve group. A pediatric blood pressure bladder was applied under the cast to simulate soft tissue swelling. Valved casts were secured using an elastic wrap, 10-mm cast spacer, or 15-mm cast spacer. Measurements of cast pressure and circumference were performed at each stage and compared on the basis of type of valve and securement.

Our results indicated that cast univalving resulted in an approximately 60% reduction in cast pressures, with a 75% reduction seen in the bivalve group. The addition of cast spacers resulted in significant pressure reductions for both valving groups. The univalve group secured with a 10-mm cast spacer produced reductions in cast pressure similar to those of the elastic-wrapped bivalve cast, both with the cast padding intact and with it released.

The use of cast spacers results in significant cast pressure reductions, regardless of valving technique. A univalved cast secured with a cast spacer can produce decreases in cast pressures similar to those seen with an elastic-wrapped bivalved cast, and it is a viable option for reducing cast pressure without compromising cast structural integrity with a bivalve technique.

Continue to: Complications following closed reduction...

 

Complications following closed reduction and casting of pediatric forearm fractures are rare, but they do occur. Arguably the most devastating of these complications is the risk of developing compartment syndrome or Volkmann contracture secondary to injury-associated swelling under a circumferential cast.^1-4 The peak in swelling can develop from 4 to 24 hours following the initial cast application,⁵ and as such, medical providers may not be able to identify it early because most children are discharged following closed reductions. For this reason, many providers implement prophylactic measures to minimize pressure-related complications.

A popular method for reducing pressure accumulation within a cast is to valve, or cut, the cast. Previous investigations have shown that cast valving results in significant reductions in cast pressure.^2,6-9 Bivalving a circumferential cast results in significantly greater reductions in cast pressure when compared with univalve techniques;^6,7,9 however, bivalving has also been shown to result in significant impairment in the structural integrity of the cast.¹⁰ An additional method to facilitate cast pressure reduction without impairing the structural integrity of the cast that accompanies a bivalve is to incorporate a cast spacer with a univalve technique to hold the split cast open.¹¹ Although this method is commonly used in clinical practice, its ability to mitigate cast pressures has not previously been investigated.

The goal of this study is to investigate the influence of incorporating cast spacers with valved long-arm casts. We hypothesized that cast spacers would provide a greater pressure reduction for both univalved and bivalved casts when compared with the use of an elastic wrap. Additionally, we proposed that by incorporating a cast spacer with a univalved cast, we could attain pressure reduction equivalent to that of a bivalved cast secured with an elastic wrap.

MATERIALS AND METHODS

Upon receiving approval from the Institutional Review Board, experimental testing began with the application of 30 total casts performed on uninjured adult human volunteers. Pressure readings were provided with the use of a bladder from a pediatric blood pressure cuff (Welch Allyn Inc), as previously described.⁶ The bladder was placed on the volar aspect of the volunteer’s forearm, held in place with a 3-in diameter cotton stockinet (3M). Cotton cast padding (Webril-Kendall) was applied, 3 in wide and 2 layers thick, and a long-arm cast was applied, 2 layers thick with 3-in wide fiberglass casting material (Scotchcast Plus Casting Tape; 3M).

Once the cast was applied and allowed to set, the blood pressure bladder was inflated to 100 mm Hg. After inflation, forearm cast circumference was measured at 2 set points, assessed at points 2 cm distal to the elbow flexor crease and 10 cm distal to the previous point (Figure 1). Using these data, we calculated estimated cast volume using the volumetric equation for a frustum. Following this point, casts were split into 2 experimental groups, univalve or bivalve, with 15 casts comprising each group. The univalve group consisted of a single cut along the dorsum of the extremity, and the bivalve group incorporated a second cut to the volar extremity. Cast valving was performed using an oscillating cast saw (Cast Vac; Stryker Instruments), with care taken to ensure the continuity of the underlying cast padding.

Continue to: Following valving, casts were secured via...

 

Following valving, casts were secured via 3 separate techniques: overwrap with a 3-in elastic wrap (Econo Wrap; Vitality Medical), application of two 10-mm and 15-mm cast spacers (CastWedge; DM Systems) (Figure 2). After securement, cast pressures were recorded, and circumference measurements were performed at the 2 previously identified points. The cast padding was then cut at the valve site and secured via the 3 listed techniques. Cast pressure and circumference measurements were performed at set time points (Figure 3). Changes in cast pressure were recorded in terms of the amount of change from the initial cast placement to account for differences in the size of volunteers’ forearms. Volumetric calculations were performed only for the spacer subgroups owing to the added material in the elastic wrap group. Estimated cast volume was calculated using the equation for volume of a frustum (Figure 4).

We used a 2-cast type (univalve and bivalve) by 4 securement subgroups (initial, elastic wrap, 10-mm spacer, and 15-mm spacer) design, with cast type serving as a between-subject measure and securement serving as a within-subject variable. An a priori power analysis showed that a minimum sample size of 15 subjects per condition should provide sufficient power of .80 and alpha set at .05, for a total of 30 casts. Statistical analyses were performed using IBM SPSS Statistics software version 21 (IBM). Experimental groups were analyzed using mixed-design analysis of variance (ANOVA). Post hoc comparisons between valving groups and cast securement were performed using Scheffe’s test to control for type II errors. Change in cast volume between the initial cast and cast spacers groups was compared using paired Student’s t tests. Statistical significance was predetermined as P < .05.

RESULTS

A summary of collected data for cast pressure and volume is detailed in Table 1, subdividing the variables on the basis of cast type and type of securement. Recorded pressures of the different subgroups are depicted in Figures 5 and 6 according to type of securement (initial, elastic wrap, 10-mm spacer, or 15-mm spacer). Results of the mixed-design ANOVA demonstrated significant differences between the initial cast pressure and univalve and bivalve groups (P < .05). There was a main effect for bivalve having lower pressure overall (F [1, 1)] = 3321.51, P < .001). There was also a main effect indicating that pressure was different for each type of securement (elastic wrap, 10-mm spacer, 15-mm spacer) (F [1, 28] = 538.54, P <. 01). Post hoc testing confirmed pressure decreased significantly, in descending order from elastic wrap, to 10-mm spacers, to 15-mm spacers (P < .05).

Table 1. Cumulative Data for Two Casting groups at Each Timepoint

Cast	Pressure	Standard Deviation	Volume
Univalve
Initial	100	---	2654.3
Elastic Wrap	39.47	3.33	---
10-mm Spacer	23.93	2.73	2708.23
15-mm Spacer	18.87	2.94	2734.86
Padding and Elastic Wrap	20.93	2.91	---
Padding and 10-mm Spacer	15.46	2.19	2733.24
Padding and 15-mm Spacer	0	---	2819.27
Bivalve
Initial	100	---	2839.3
Elastic Wrap	25.9	3.17	---
10-mm Spacer	16.53	2.32	3203.13
15-mm Spacer	13.6	2.74	3380.32
Padding and Elastic Wrap	12.67	1.95	---
Padding and 10-mm Spacer	0	---	3296.55
Padding and 15- mm Spacer	0	---	3438.67

Continue to: Table 2...

 

The summary of volumetric changes is listed in Table 2. The decrease in pressure correlated with an associated increase in cast volume, as demonstrated in Figure 7. The degree of increase in cast volume was more pronounced in the bivalve group (P < .001). The volume increased in the 15-mm group compared with the 10-mm group for both groups (P < .001) and increased for each spacer group with the release of the underlying padding (P < .05).

 

Table 2. Volumetric Data

Cast	Average Volumetric change (cm3)	Standard Deviation
Univalve
10-mm Spacer	175.6	65.4
15-mm Spacer	269.4	73.3
Padding and 10-mm Spacer	202.3	62.5
Padding and 15-mm Spacer	294.1	66.9
Bivalve
10-mm Spacer	363.7	67.2
15-mm Spacer	540.9	85.7
Padding and 10-mm Spacer	457.2	97.9
Padding and 15-mm Spacer	599.3	84.2

Analysis of the planned comparisons demonstrated no significant difference between the bivalve with elastic wrap and univalve with 10-mm spacer subgroups (t [28] = 1.85, P = .075, d = .68). In comparing the bivalve with elastic wrap group with the univalve and 15-mm spacer subgroup, the univalve group showed significantly lower pressures [t [28] = 6.32, P < .001, d = .2.31).

DISCUSSION

Valving of circumferential casting is a well-established technique to minimize potential pressure-related complications. Previous studies have demonstrated that univalving techniques produce a 65% reduction in cast pressure, whereas bivalving produces an 80% decrease.^6,7,9 Our results showed comparable pressure reductions of 75% with bivalving and 60% with univalving. The type of cast padding has been shown to have a significant effect on the cast pressure, favoring lower pressures with cotton padding over synthetic and waterproof padding, which, when released, can provide an additional 10% pressure reduction.^6,7

Although bivalving techniques are superior in pressure reduction, the reduction comes at the cost of the cast’s structural integrity. Crickard and colleagues¹⁰ performed a biomechanical assessment of the structural integrity by 3-point bending of casts following univalve and bivalve compared with an intact cast. The authors found that valving resulted in a significant decrease in the casts’ bending stiffness and load to failure, with bivalved casts demonstrating a significantly lower load to failure than univalved casts. One technique that has been used to enhance the pressure reduction in univalved casting techniques is the application of a cast spacer. Rang and colleagues¹¹ recommended this technique as part of a graded cast-splitting approach for the treatment of children’s fractures. This technique was applied to fractures with only modest anticipated swelling, which accounted for approximately 95% of casts applied in their children’s hospital. Our results support the use of cast spacers, demonstrating significant reduction in cast pressure in both univalve and bivalve techniques. Additionally, we found that a univalved cast with a 10-mm cast spacer provided pressure reduction similar to that of a bivalved cast.

The theory behind the application of cast spacers is that a split fiberglass cast will not remain open unless held in position.¹¹ Holding the cast open is less of a restraint to pressure reduction in bivalving techniques, because the split cast no longer has the contralateral intact hinge point to resist cast opening, demonstrated in the compromise in structural integrity seen with this technique.¹⁰ By maintaining the split cast in an opened position, the effective volume of the cast is increased, which allows for the reduction in cast pressure. This is demonstrated in our results indicating an increase in estimated cast volume with an associated incremental reduction in cast pressure with the application of incrementally sized cast spacers. Although this technique does have the potential for skin irritation caused by cast expansion, as well as local swelling at the cast window location, it is a cost-effective treatment method compared with overwrapping a bivalved cast, $1.55 for 1 cast spacer vs an estimated $200 for a forearm cast application.

This study is not without its limitations. Our model does not account for the soft tissue injury associated with forearm fractures. However, by using human volunteers, we were able to include the viscoelastic properties that are omitted with nonliving models, and our results do align with those of previous investigations regarding pressure change following valving. We did not incorporate a 3-point molding technique commonly used with reduction and casting of acute forearm fractures, owing to the lack of a standardized method for applying the mold to healthy volunteers. Although molding is necessary for most fractures in which valving is considered, we believe our data still provide valuable information. Additionally, valving of circumferential casts has not been shown, prospectively, to result in a reduction of cast-related compartment syndrome, maintenance of reduction, or need for surgery.^12,13 However, these results are reflective of reliable patients who completed the requisite follow-up care necessary for inclusion in a randomized controlled trial and may be applicable to unreliable patients or patient situations, a setting in which the compromise in cast structural integrity may be unacceptable.

CONCLUSION

We demonstrated that incorporating cast spacers into valved long-arm casts provides pressure reduction comparable to that achieved with the use of an elastic wrap. The addition of a 10-mm cast spacer to a univalved long-arm cast provides pressure reduction equivalent to that of a bivalved cast secured with an elastic wrap. A univalved cast secured with a cast spacer is a viable option for treatment of displaced pediatric forearm fractures, without compromising the cast’s structural integrity as required with bivalved techniques.

This paper will be judged for the Resident Writer’s Award.

ABSTRACT

Fiberglass casts are frequently valved to accommodate swelling following injury or surgery. The use of cast spacers has been recommended to bridge this gap between pressure reduction and cast strength, but no studies have assessed their effect on cast pressure.

We applied 30 long-arm fiberglass casts to adult volunteers, divided between a univalve group and a bivalve group. A pediatric blood pressure bladder was applied under the cast to simulate soft tissue swelling. Valved casts were secured using an elastic wrap, 10-mm cast spacer, or 15-mm cast spacer. Measurements of cast pressure and circumference were performed at each stage and compared on the basis of type of valve and securement.

Our results indicated that cast univalving resulted in an approximately 60% reduction in cast pressures, with a 75% reduction seen in the bivalve group. The addition of cast spacers resulted in significant pressure reductions for both valving groups. The univalve group secured with a 10-mm cast spacer produced reductions in cast pressure similar to those of the elastic-wrapped bivalve cast, both with the cast padding intact and with it released.

The use of cast spacers results in significant cast pressure reductions, regardless of valving technique. A univalved cast secured with a cast spacer can produce decreases in cast pressures similar to those seen with an elastic-wrapped bivalved cast, and it is a viable option for reducing cast pressure without compromising cast structural integrity with a bivalve technique.

Continue to: Complications following closed reduction...

 

Complications following closed reduction and casting of pediatric forearm fractures are rare, but they do occur. Arguably the most devastating of these complications is the risk of developing compartment syndrome or Volkmann contracture secondary to injury-associated swelling under a circumferential cast.^1-4 The peak in swelling can develop from 4 to 24 hours following the initial cast application,⁵ and as such, medical providers may not be able to identify it early because most children are discharged following closed reductions. For this reason, many providers implement prophylactic measures to minimize pressure-related complications.

A popular method for reducing pressure accumulation within a cast is to valve, or cut, the cast. Previous investigations have shown that cast valving results in significant reductions in cast pressure.^2,6-9 Bivalving a circumferential cast results in significantly greater reductions in cast pressure when compared with univalve techniques;^6,7,9 however, bivalving has also been shown to result in significant impairment in the structural integrity of the cast.¹⁰ An additional method to facilitate cast pressure reduction without impairing the structural integrity of the cast that accompanies a bivalve is to incorporate a cast spacer with a univalve technique to hold the split cast open.¹¹ Although this method is commonly used in clinical practice, its ability to mitigate cast pressures has not previously been investigated.

The goal of this study is to investigate the influence of incorporating cast spacers with valved long-arm casts. We hypothesized that cast spacers would provide a greater pressure reduction for both univalved and bivalved casts when compared with the use of an elastic wrap. Additionally, we proposed that by incorporating a cast spacer with a univalved cast, we could attain pressure reduction equivalent to that of a bivalved cast secured with an elastic wrap.

MATERIALS AND METHODS

Upon receiving approval from the Institutional Review Board, experimental testing began with the application of 30 total casts performed on uninjured adult human volunteers. Pressure readings were provided with the use of a bladder from a pediatric blood pressure cuff (Welch Allyn Inc), as previously described.⁶ The bladder was placed on the volar aspect of the volunteer’s forearm, held in place with a 3-in diameter cotton stockinet (3M). Cotton cast padding (Webril-Kendall) was applied, 3 in wide and 2 layers thick, and a long-arm cast was applied, 2 layers thick with 3-in wide fiberglass casting material (Scotchcast Plus Casting Tape; 3M).

Once the cast was applied and allowed to set, the blood pressure bladder was inflated to 100 mm Hg. After inflation, forearm cast circumference was measured at 2 set points, assessed at points 2 cm distal to the elbow flexor crease and 10 cm distal to the previous point (Figure 1). Using these data, we calculated estimated cast volume using the volumetric equation for a frustum. Following this point, casts were split into 2 experimental groups, univalve or bivalve, with 15 casts comprising each group. The univalve group consisted of a single cut along the dorsum of the extremity, and the bivalve group incorporated a second cut to the volar extremity. Cast valving was performed using an oscillating cast saw (Cast Vac; Stryker Instruments), with care taken to ensure the continuity of the underlying cast padding.

Continue to: Following valving, casts were secured via...

 

Following valving, casts were secured via 3 separate techniques: overwrap with a 3-in elastic wrap (Econo Wrap; Vitality Medical), application of two 10-mm and 15-mm cast spacers (CastWedge; DM Systems) (Figure 2). After securement, cast pressures were recorded, and circumference measurements were performed at the 2 previously identified points. The cast padding was then cut at the valve site and secured via the 3 listed techniques. Cast pressure and circumference measurements were performed at set time points (Figure 3). Changes in cast pressure were recorded in terms of the amount of change from the initial cast placement to account for differences in the size of volunteers’ forearms. Volumetric calculations were performed only for the spacer subgroups owing to the added material in the elastic wrap group. Estimated cast volume was calculated using the equation for volume of a frustum (Figure 4).

We used a 2-cast type (univalve and bivalve) by 4 securement subgroups (initial, elastic wrap, 10-mm spacer, and 15-mm spacer) design, with cast type serving as a between-subject measure and securement serving as a within-subject variable. An a priori power analysis showed that a minimum sample size of 15 subjects per condition should provide sufficient power of .80 and alpha set at .05, for a total of 30 casts. Statistical analyses were performed using IBM SPSS Statistics software version 21 (IBM). Experimental groups were analyzed using mixed-design analysis of variance (ANOVA). Post hoc comparisons between valving groups and cast securement were performed using Scheffe’s test to control for type II errors. Change in cast volume between the initial cast and cast spacers groups was compared using paired Student’s t tests. Statistical significance was predetermined as P < .05.

RESULTS

A summary of collected data for cast pressure and volume is detailed in Table 1, subdividing the variables on the basis of cast type and type of securement. Recorded pressures of the different subgroups are depicted in Figures 5 and 6 according to type of securement (initial, elastic wrap, 10-mm spacer, or 15-mm spacer). Results of the mixed-design ANOVA demonstrated significant differences between the initial cast pressure and univalve and bivalve groups (P < .05). There was a main effect for bivalve having lower pressure overall (F [1, 1)] = 3321.51, P < .001). There was also a main effect indicating that pressure was different for each type of securement (elastic wrap, 10-mm spacer, 15-mm spacer) (F [1, 28] = 538.54, P <. 01). Post hoc testing confirmed pressure decreased significantly, in descending order from elastic wrap, to 10-mm spacers, to 15-mm spacers (P < .05).

Table 1. Cumulative Data for Two Casting groups at Each Timepoint

Cast	Pressure	Standard Deviation	Volume
Univalve
Initial	100	---	2654.3
Elastic Wrap	39.47	3.33	---
10-mm Spacer	23.93	2.73	2708.23
15-mm Spacer	18.87	2.94	2734.86
Padding and Elastic Wrap	20.93	2.91	---
Padding and 10-mm Spacer	15.46	2.19	2733.24
Padding and 15-mm Spacer	0	---	2819.27
Bivalve
Initial	100	---	2839.3
Elastic Wrap	25.9	3.17	---
10-mm Spacer	16.53	2.32	3203.13
15-mm Spacer	13.6	2.74	3380.32
Padding and Elastic Wrap	12.67	1.95	---
Padding and 10-mm Spacer	0	---	3296.55
Padding and 15- mm Spacer	0	---	3438.67

Continue to: Table 2...

 

The summary of volumetric changes is listed in Table 2. The decrease in pressure correlated with an associated increase in cast volume, as demonstrated in Figure 7. The degree of increase in cast volume was more pronounced in the bivalve group (P < .001). The volume increased in the 15-mm group compared with the 10-mm group for both groups (P < .001) and increased for each spacer group with the release of the underlying padding (P < .05).

 

Table 2. Volumetric Data

Cast	Average Volumetric change (cm3)	Standard Deviation
Univalve
10-mm Spacer	175.6	65.4
15-mm Spacer	269.4	73.3
Padding and 10-mm Spacer	202.3	62.5
Padding and 15-mm Spacer	294.1	66.9
Bivalve
10-mm Spacer	363.7	67.2
15-mm Spacer	540.9	85.7
Padding and 10-mm Spacer	457.2	97.9
Padding and 15-mm Spacer	599.3	84.2

Analysis of the planned comparisons demonstrated no significant difference between the bivalve with elastic wrap and univalve with 10-mm spacer subgroups (t [28] = 1.85, P = .075, d = .68). In comparing the bivalve with elastic wrap group with the univalve and 15-mm spacer subgroup, the univalve group showed significantly lower pressures [t [28] = 6.32, P < .001, d = .2.31).

DISCUSSION

Valving of circumferential casting is a well-established technique to minimize potential pressure-related complications. Previous studies have demonstrated that univalving techniques produce a 65% reduction in cast pressure, whereas bivalving produces an 80% decrease.^6,7,9 Our results showed comparable pressure reductions of 75% with bivalving and 60% with univalving. The type of cast padding has been shown to have a significant effect on the cast pressure, favoring lower pressures with cotton padding over synthetic and waterproof padding, which, when released, can provide an additional 10% pressure reduction.^6,7

Although bivalving techniques are superior in pressure reduction, the reduction comes at the cost of the cast’s structural integrity. Crickard and colleagues¹⁰ performed a biomechanical assessment of the structural integrity by 3-point bending of casts following univalve and bivalve compared with an intact cast. The authors found that valving resulted in a significant decrease in the casts’ bending stiffness and load to failure, with bivalved casts demonstrating a significantly lower load to failure than univalved casts. One technique that has been used to enhance the pressure reduction in univalved casting techniques is the application of a cast spacer. Rang and colleagues¹¹ recommended this technique as part of a graded cast-splitting approach for the treatment of children’s fractures. This technique was applied to fractures with only modest anticipated swelling, which accounted for approximately 95% of casts applied in their children’s hospital. Our results support the use of cast spacers, demonstrating significant reduction in cast pressure in both univalve and bivalve techniques. Additionally, we found that a univalved cast with a 10-mm cast spacer provided pressure reduction similar to that of a bivalved cast.

The theory behind the application of cast spacers is that a split fiberglass cast will not remain open unless held in position.¹¹ Holding the cast open is less of a restraint to pressure reduction in bivalving techniques, because the split cast no longer has the contralateral intact hinge point to resist cast opening, demonstrated in the compromise in structural integrity seen with this technique.¹⁰ By maintaining the split cast in an opened position, the effective volume of the cast is increased, which allows for the reduction in cast pressure. This is demonstrated in our results indicating an increase in estimated cast volume with an associated incremental reduction in cast pressure with the application of incrementally sized cast spacers. Although this technique does have the potential for skin irritation caused by cast expansion, as well as local swelling at the cast window location, it is a cost-effective treatment method compared with overwrapping a bivalved cast, $1.55 for 1 cast spacer vs an estimated $200 for a forearm cast application.

This study is not without its limitations. Our model does not account for the soft tissue injury associated with forearm fractures. However, by using human volunteers, we were able to include the viscoelastic properties that are omitted with nonliving models, and our results do align with those of previous investigations regarding pressure change following valving. We did not incorporate a 3-point molding technique commonly used with reduction and casting of acute forearm fractures, owing to the lack of a standardized method for applying the mold to healthy volunteers. Although molding is necessary for most fractures in which valving is considered, we believe our data still provide valuable information. Additionally, valving of circumferential casts has not been shown, prospectively, to result in a reduction of cast-related compartment syndrome, maintenance of reduction, or need for surgery.^12,13 However, these results are reflective of reliable patients who completed the requisite follow-up care necessary for inclusion in a randomized controlled trial and may be applicable to unreliable patients or patient situations, a setting in which the compromise in cast structural integrity may be unacceptable.

CONCLUSION

We demonstrated that incorporating cast spacers into valved long-arm casts provides pressure reduction comparable to that achieved with the use of an elastic wrap. The addition of a 10-mm cast spacer to a univalved long-arm cast provides pressure reduction equivalent to that of a bivalved cast secured with an elastic wrap. A univalved cast secured with a cast spacer is a viable option for treatment of displaced pediatric forearm fractures, without compromising the cast’s structural integrity as required with bivalved techniques.

This paper will be judged for the Resident Writer’s Award.

ABSTRACT

A patient’s perception of hospital or provider quality can have far-reaching effects, as it can impact reimbursement, patient selection of a surgeon, and healthcare competition. A variety of organizations offer quality designations for orthopedic surgery and its subspecialties. Our goal is to compare total joint arthroplasty (TJA) quality designation methodology across key quality rating organizations. One researcher conducted an initial Google search to determine organizations providing quality designations for hospitals and surgeons providing orthopedic procedures with a focus on TJA. Organizations that offer quality designation specific to TJA were determined. Organizations that provided general orthopedic surgery or only surgeon-specific quality designation were excluded from the analysis. The senior author confirmed the inclusion of the final organizations. Seven organizations fit our inclusion criteria. Only the private payers and The Joint Commission required hospital accreditation to meet quality designation criteria. Total arthroplasty volume was considered in 86% of the organizations’ methodologies, and 57% of organizations utilized process measurements such as antibiotic prophylaxis and care pathways. In addition, 57% of organizations included patient experience in their methodologies. Only 29% of organizations included a cost element in their methodology. All organizations utilized outcome data and publicly reported all hospitals receiving their quality designation. Hospital quality designation methodologies are inconsistent in the context of TJA. All stakeholders (ie, providers, payers, and patients) should be involved in deciding the definition of quality.

Continue to: Healthcare in the United States...

 

Healthcare in the United States has begun to move toward a system focused on value for patients, defined as health outcome per dollar expended.¹ Indeed, an estimated 30% of Medicare payments are now made using the so-called alternative payment models (eg, bundled payments),² and there is an expectation that consumerism in medicine will continue to expand.³ In addition, although there is a continuing debate regarding the benefits and pitfalls of hospital mergers, there is no question whether provider consolidation has increased dramatically in recent years.⁴ At the core of many of these changes is the push to improve healthcare quality and reduce costs.

Quality has the ability to affect payment, patient selection of providers, and hospital competition. Patients (ie, healthcare consumers) are increasingly using the Internet to find a variety of health information.⁵ Accessible provider quality information online would allow patients to make more informed decisions about where to seek care. In addition, the development of transparent quality ratings could assist payers in driving beneficiaries to higher quality and better value providers, which could mean more business for the highest quality physicians and better patient outcomes with fewer complications. Some payers such as the Centers for Medicare and Medicaid Services (CMS) have already started using quality measures as part of their reimbursement strategy.⁶ Because CMS is the largest payer in the United States, private insurers tend to follow their lead; thus, quality measurements will become even more common as a factor in reimbursement over the coming years.

To make quality ratings useful, “quality” must be clearly defined. Clarity around which factors are considered in a quality designation will create transparency for patients and allow providers to understand how their performance is being measured so that they focus on improving outcomes for their patients. Numerous organizations, including private payers, public payers, and both not-for-profit and for-profit entities, have created quality designation programs to rate providers. However, within orthopedics and several other medical specialties, there has been an ongoing debate about what measures best reflect quality.⁷ Although inconsistencies in quality ratings in arthroplasty care have been noted,⁸ it remains unknown how each quality designation program compares with the others in terms of the factors considered in deciding quality designations.

The purpose of this study is to evaluate publicly available information from key quality designation programs for total joint arthroplasty (TJA) providers to determine what factors are considered by each organization in awarding quality designations; what similarities and differences in quality designations exist across the different organizations; and how many of the organizations publish their quality designation methodologies and final rating results.

MATERIALS AND METHODS

A directed Google search was conducted to determine organizations (ie, payers, independent firms, and government entities) that rate hospitals and/or surgeons in orthopedic surgery. The identified organizations were then examined to determine whether they provided hospital ratings for total hip and/or knee arthroplasty. Entities were included if they provided quality designations for hospitals specifically addressing TJA. Organizations that provided only general hospital, other surgical procedures, orthopedic surgery, or orthopedic surgeon-specific quality designations were excluded. A list of all organizations determined to fit the inclusion criteria was then reviewed for completeness and approved by the senior author.

Continue to: One investigator reviewed the website of each organization...

 

One investigator reviewed the website of each organization fitting the inclusion criteria to determine the full rating methodology in 1 sitting on July 2, 2016. Detailed notes were taken on each program using publicly available information. For organizations that used proprietary criteria for quality designation (eg, The Joint Commission [TJC]), only publicly available information was used in the analysis. Therefore, the information reported is solely based on data available online to the public.

Detailed quality designation criteria were condensed into broader categories (accreditation, volume, structural, process, outcomes, patient experience, and cost/efficiency) to capture differences between each organization reviewed. In addition, we recorded whether each organization published a list of providers that received its quality designation.  

RESULTS

A total of 7 organizations fit our inclusion criteria^9-15 (Table). Of these 7 organizations, 3 were private payers (Aetna, UnitedHealth, and Blue Cross Blue Shield [BCBS]), 2 were nongovernmental not-for-profit organizations (TJC and Consumer Reports), and 2 were consumer-based and/or for-profit organizations (HealthGrades and US News & World Report [USNWR]). There were no government agencies that fit our inclusion criteria. BCBS had the following 2 separate quality designations: BCBS Blue Distinction and BCBS Blue Distinction+. The only difference between the 2 BCBS ratings is that BCBS Blue Distinction+ includes cost efficiency ratings, whereas BCBS Blue Distinction does not.

Only the 3 private payers and TJC, the primary hospital accreditation body in the United States, required accreditation as part of its quality designation criteria. TJC requires its own accreditation for quality designation consideration, whereas the 3 private payers allow accreditation from one of a variety of sources. Aetna Institutes of Quality for Orthopedic Surgery requires accreditation by TJC, Healthcare Facilities Accreditation Program, American Osteopathic Association, National Integrated Accreditation for Healthcare Organizations, or Det Norske Veritas Healthcare. UnitedHealth Premium Total Joint Replacement (TJR) Specialty Center requires accreditation by TJC and/or equivalent of TJC accreditation. However, TJC accreditation equivalents are not noted in the UnitedHealth handbook. BCBS Blue Distinction and Distinction+ require accreditation by TJC, Healthcare Facilities Accreditation Program, National Integrated Accreditation for Healthcare Organizations, or Center for Improvement in Healthcare Quality. In addition, BCBS is willing to consider alternative accreditations that are at least as stringent as the national alternatives noted. However, no detailed criteria that must be met to be equivalent to the national standards are noted in the relevant quality designation handbook.

The volume of completed total hip and knee arthroplasty procedures was considered in 6 of the organizations’ quality ratings methodologies. Of those 6, all private payers, TJC (not-for-profit), and 2 for-profit rating agencies were included. Surgeon specialization in TJA was only explicitly noted as a factor considered in UnitedHealth Premium TJR Specialty Center criteria; however, the requirements for surgeon specialization were not clearly defined. In addition, the presence of a multidisciplinary clinical pathway was only explicitly considered for Aetna Institutes of Quality for Orthopedic Surgery.

Structural requirements (eg, use of electronic health records [EHR], staffing levels, etc.) were taken into account in private payer and USNWR quality methodologies. Process measures (eg, antibiotic prophylaxis and other care pathways) were considered for the private payers and TJC but not for USNWR quality designation. Cost and/or efficiency measures were factors in the quality formula for Aetna Institutes of Quality for Orthopedic Surgery and BCBS Distinction+. Aetna utilizes its own cost data and risk-adjusts using a product known as Symmetry Episode Risk Groups to determine cost-effectiveness, while BCBS uses its own Composite Facility Cost Index. Patient experience (eg, Hospital Consumer Assessment of Healthcare Providers and Systems [HCAHPS]) was incorporated into the quality formulas for 4 of the 7 quality designation programs examined.

Continue to: All of the 7 quality designation programs included...

 

All of the 7 quality designation programs included outcomes (ie, readmission rates and/or mortality rates) and publicly reported the hospitals receiving their quality designation. In contrast, only Aetna explicitly included the presence of multidisciplinary clinical care pathways as part of their quality designation criteria. In addition, only UnitedHealth included surgeon specialization in joint arthroplasty as a factor for quality consideration for its quality designation program. BCBS Distinction+ and Aetna Institutes of Quality for Orthopedic Surgery were the only 2 quality designations that included at least 1 variable that fit into each of the 7 characteristics considered (accreditation, volume, structural, process, outcomes, patient experience, and cost/efficiency).

DISCUSSION

As healthcare continues to shift toward value-based delivery and payment models, quality becomes a critical factor in reimbursement and provider rankings. However, quality is a vague term. Several providers probably do not know what is required to be designated as high quality by a particular rating agency. Moreover, there are multiple quality designation programs, all using distinct criteria to determine “quality,” which further complicates the matter. Our objective was to determine the key stakeholders that provide quality designations in TJA and what criteria each organization uses in assessing quality.

Our idea of comprehensive quality is based on Avedis Donabedian’s enduring framework for healthcare quality focused on structure, process, and outcome.¹⁶ We expanded on these 3 areas and analyzed quality designations based on variables fitting into the following categories: accreditation, volume, structural, process, outcomes, patient experience, and cost/efficiency. We believe that these categories encompass a comprehensive rating system that addresses key elements of patient care. However, our results suggest that only 2 major quality designations (BCBS Distinction+ and Aetna Institutes of Quality for Orthopedic Surgery) take all such variables into account.

All quality designation programs that we analyzed required outcome data (ie, readmission and/or mortality rates within 30 days); however, only 2 programs utilized cost in their quality designation criteria (BCBS Distinction+ and Aetna Institutes of Quality for Orthopedic Surgery). Aetna Institutes of Quality for Orthopedic Surgery risk-adjusted for its cost-effectiveness calculations based on age, sex, and other unspecified conditions using a product known as Symmetry Episode Risk Groups. However, the organization also noted that although it did risk-adjust for inpatient mortality, it did not do so for pulmonary embolism or deep vein thrombosis. BCBS Distinction+ also utilized risk adjustment for its cost efficiency measure, and its step-by-step methodology is available online. Further, Consumer Reports does risk-adjust using logistic regression models in their quality analysis, but the description provided is minimal; it is noted that such risk adjustments are already completed by CMS prior to Consumer Reports acquiring the data. The CMS Compare model information is available on the CMS website. The data utilized by several organizations and presented on CMS Compare are already risk-adjusted using CMS’ approach. In contrast, UnitedHealth Premium TJR Specialty Center gathers its own data from providers and does not describe a risk adjustment methodology. Risk adjustment is important because the lack of risk adjustment may lead to physicians “cherry-picking” easy cases to boost positive outcomes, leading to increased financial benefits and higher quality ratings. Having a consistent risk adjustment formula will ensure accurate comparisons across outcomes and cost-effectiveness measures used by quality designation programs.

Factors considered for quality designation varied greatly from one organization to the other. The range of categories of factors considered varied from 1 (Consumer Reports only considered outcome data) to all 7 categories (BCBS Distinction+ and Aetna Institutes of Quality for Orthopedic Surgery). Our findings are consistent with the work by Keswani and colleagues,⁸ which showed that there is likely variation in factors considered when rating hospital quality more broadly. Our work suggests that quality designation formulas do not appear to get more consistent when focused on TJA.

We found that all organizations in our analysis published the providers earning their quality designation. However, TJC does not provide publicly a detailed methodology on how to qualify for its quality designation. The price to purchase the necessary manual for this information is $146.00 for accredited organizations and $186.00 for all others.¹⁷ For large healthcare providers, this is not a large sum of money. Nonetheless, this provides an additional hurdle for stakeholders to gain a full understanding of the requirements to receive a TJC Gold Seal for Orthopedics.

Previous work has evaluated the consistency of and the variety of means of gauging healthcare quality. Previous work by Rothberg and colleagues¹⁸ comparing hospital rankings across 5 common consumer-oriented websites found disagreement on hospital rankings within any diagnosis and even among metrics such as mortality. Another study by Halasyamani and Davis¹⁹ found that CMS Compare and USNWR rankings were dissimilar and the authors attributed the discrepancy to different methodologies. In addition, a study by Krumholz and colleagues²⁰ focused on Internet report cards, which measured the appropriate use of select medications and mortality rates for acute myocardial infarction as the quality metrics. The authors found that, in aggregate, there was a clear difference in quality of care and outcomes but that comparisons between 2 hospitals provided poor discrimination.²⁰ Other work has analyzed the increasing trend of online ratings of orthopedic surgeons by patients.²¹ However, there remains no agreed-upon definition of quality. Thus, the use of the term “quality” in several studies may be misleading.

Our results must be interpreted keeping the limitations of our work in mind. First, we used expert knowledge and a public search engine to develop our list of organizations that provide TJA quality designations. However, there is a possibility that we did not include all relevant organizations. Second, although all authors reviewed the final data, it is possible that there was human error in the analysis of each organization’s quality designation criteria.

CONCLUSION

As healthcare progresses further toward a system that rewards providers for delivering value to patients, accurately defining and measuring quality becomes critical because it can be suggestive of value to patients, payers, and providers. Furthermore, it gives providers a goal to focus on as they strive to improve the value of care they deliver to patients. Measuring healthcare quality is currently a novel, imperfect science,²² and there continues to be a debate about what factors should be included in a quality designation formula. Nonetheless, more and more quality designations and performance measurements are being created for orthopedic care, including total hip and total knee arthroplasty. In fact, in 2016, The Leapfrog Group added readmission for patients undergoing TJA to its survey.²³ Consensus on a quality definition may facilitate the movement toward a value-based healthcare system. Future research should evaluate strategies for gaining consensus among stakeholders for a universal quality metric in TJA. Surgeons, hospitals, payers, and most importantly patients should play critical roles in defining quality.

ABSTRACT

A patient’s perception of hospital or provider quality can have far-reaching effects, as it can impact reimbursement, patient selection of a surgeon, and healthcare competition. A variety of organizations offer quality designations for orthopedic surgery and its subspecialties. Our goal is to compare total joint arthroplasty (TJA) quality designation methodology across key quality rating organizations. One researcher conducted an initial Google search to determine organizations providing quality designations for hospitals and surgeons providing orthopedic procedures with a focus on TJA. Organizations that offer quality designation specific to TJA were determined. Organizations that provided general orthopedic surgery or only surgeon-specific quality designation were excluded from the analysis. The senior author confirmed the inclusion of the final organizations. Seven organizations fit our inclusion criteria. Only the private payers and The Joint Commission required hospital accreditation to meet quality designation criteria. Total arthroplasty volume was considered in 86% of the organizations’ methodologies, and 57% of organizations utilized process measurements such as antibiotic prophylaxis and care pathways. In addition, 57% of organizations included patient experience in their methodologies. Only 29% of organizations included a cost element in their methodology. All organizations utilized outcome data and publicly reported all hospitals receiving their quality designation. Hospital quality designation methodologies are inconsistent in the context of TJA. All stakeholders (ie, providers, payers, and patients) should be involved in deciding the definition of quality.

Continue to: Healthcare in the United States...

 

Healthcare in the United States has begun to move toward a system focused on value for patients, defined as health outcome per dollar expended.¹ Indeed, an estimated 30% of Medicare payments are now made using the so-called alternative payment models (eg, bundled payments),² and there is an expectation that consumerism in medicine will continue to expand.³ In addition, although there is a continuing debate regarding the benefits and pitfalls of hospital mergers, there is no question whether provider consolidation has increased dramatically in recent years.⁴ At the core of many of these changes is the push to improve healthcare quality and reduce costs.

Quality has the ability to affect payment, patient selection of providers, and hospital competition. Patients (ie, healthcare consumers) are increasingly using the Internet to find a variety of health information.⁵ Accessible provider quality information online would allow patients to make more informed decisions about where to seek care. In addition, the development of transparent quality ratings could assist payers in driving beneficiaries to higher quality and better value providers, which could mean more business for the highest quality physicians and better patient outcomes with fewer complications. Some payers such as the Centers for Medicare and Medicaid Services (CMS) have already started using quality measures as part of their reimbursement strategy.⁶ Because CMS is the largest payer in the United States, private insurers tend to follow their lead; thus, quality measurements will become even more common as a factor in reimbursement over the coming years.

To make quality ratings useful, “quality” must be clearly defined. Clarity around which factors are considered in a quality designation will create transparency for patients and allow providers to understand how their performance is being measured so that they focus on improving outcomes for their patients. Numerous organizations, including private payers, public payers, and both not-for-profit and for-profit entities, have created quality designation programs to rate providers. However, within orthopedics and several other medical specialties, there has been an ongoing debate about what measures best reflect quality.⁷ Although inconsistencies in quality ratings in arthroplasty care have been noted,⁸ it remains unknown how each quality designation program compares with the others in terms of the factors considered in deciding quality designations.

The purpose of this study is to evaluate publicly available information from key quality designation programs for total joint arthroplasty (TJA) providers to determine what factors are considered by each organization in awarding quality designations; what similarities and differences in quality designations exist across the different organizations; and how many of the organizations publish their quality designation methodologies and final rating results.

MATERIALS AND METHODS

A directed Google search was conducted to determine organizations (ie, payers, independent firms, and government entities) that rate hospitals and/or surgeons in orthopedic surgery. The identified organizations were then examined to determine whether they provided hospital ratings for total hip and/or knee arthroplasty. Entities were included if they provided quality designations for hospitals specifically addressing TJA. Organizations that provided only general hospital, other surgical procedures, orthopedic surgery, or orthopedic surgeon-specific quality designations were excluded. A list of all organizations determined to fit the inclusion criteria was then reviewed for completeness and approved by the senior author.

Continue to: One investigator reviewed the website of each organization...

 

One investigator reviewed the website of each organization fitting the inclusion criteria to determine the full rating methodology in 1 sitting on July 2, 2016. Detailed notes were taken on each program using publicly available information. For organizations that used proprietary criteria for quality designation (eg, The Joint Commission [TJC]), only publicly available information was used in the analysis. Therefore, the information reported is solely based on data available online to the public.

Detailed quality designation criteria were condensed into broader categories (accreditation, volume, structural, process, outcomes, patient experience, and cost/efficiency) to capture differences between each organization reviewed. In addition, we recorded whether each organization published a list of providers that received its quality designation.  

RESULTS

A total of 7 organizations fit our inclusion criteria^9-15 (Table). Of these 7 organizations, 3 were private payers (Aetna, UnitedHealth, and Blue Cross Blue Shield [BCBS]), 2 were nongovernmental not-for-profit organizations (TJC and Consumer Reports), and 2 were consumer-based and/or for-profit organizations (HealthGrades and US News & World Report [USNWR]). There were no government agencies that fit our inclusion criteria. BCBS had the following 2 separate quality designations: BCBS Blue Distinction and BCBS Blue Distinction+. The only difference between the 2 BCBS ratings is that BCBS Blue Distinction+ includes cost efficiency ratings, whereas BCBS Blue Distinction does not.

Only the 3 private payers and TJC, the primary hospital accreditation body in the United States, required accreditation as part of its quality designation criteria. TJC requires its own accreditation for quality designation consideration, whereas the 3 private payers allow accreditation from one of a variety of sources. Aetna Institutes of Quality for Orthopedic Surgery requires accreditation by TJC, Healthcare Facilities Accreditation Program, American Osteopathic Association, National Integrated Accreditation for Healthcare Organizations, or Det Norske Veritas Healthcare. UnitedHealth Premium Total Joint Replacement (TJR) Specialty Center requires accreditation by TJC and/or equivalent of TJC accreditation. However, TJC accreditation equivalents are not noted in the UnitedHealth handbook. BCBS Blue Distinction and Distinction+ require accreditation by TJC, Healthcare Facilities Accreditation Program, National Integrated Accreditation for Healthcare Organizations, or Center for Improvement in Healthcare Quality. In addition, BCBS is willing to consider alternative accreditations that are at least as stringent as the national alternatives noted. However, no detailed criteria that must be met to be equivalent to the national standards are noted in the relevant quality designation handbook.

The volume of completed total hip and knee arthroplasty procedures was considered in 6 of the organizations’ quality ratings methodologies. Of those 6, all private payers, TJC (not-for-profit), and 2 for-profit rating agencies were included. Surgeon specialization in TJA was only explicitly noted as a factor considered in UnitedHealth Premium TJR Specialty Center criteria; however, the requirements for surgeon specialization were not clearly defined. In addition, the presence of a multidisciplinary clinical pathway was only explicitly considered for Aetna Institutes of Quality for Orthopedic Surgery.

Structural requirements (eg, use of electronic health records [EHR], staffing levels, etc.) were taken into account in private payer and USNWR quality methodologies. Process measures (eg, antibiotic prophylaxis and other care pathways) were considered for the private payers and TJC but not for USNWR quality designation. Cost and/or efficiency measures were factors in the quality formula for Aetna Institutes of Quality for Orthopedic Surgery and BCBS Distinction+. Aetna utilizes its own cost data and risk-adjusts using a product known as Symmetry Episode Risk Groups to determine cost-effectiveness, while BCBS uses its own Composite Facility Cost Index. Patient experience (eg, Hospital Consumer Assessment of Healthcare Providers and Systems [HCAHPS]) was incorporated into the quality formulas for 4 of the 7 quality designation programs examined.

Continue to: All of the 7 quality designation programs included...

 

All of the 7 quality designation programs included outcomes (ie, readmission rates and/or mortality rates) and publicly reported the hospitals receiving their quality designation. In contrast, only Aetna explicitly included the presence of multidisciplinary clinical care pathways as part of their quality designation criteria. In addition, only UnitedHealth included surgeon specialization in joint arthroplasty as a factor for quality consideration for its quality designation program. BCBS Distinction+ and Aetna Institutes of Quality for Orthopedic Surgery were the only 2 quality designations that included at least 1 variable that fit into each of the 7 characteristics considered (accreditation, volume, structural, process, outcomes, patient experience, and cost/efficiency).

DISCUSSION

As healthcare continues to shift toward value-based delivery and payment models, quality becomes a critical factor in reimbursement and provider rankings. However, quality is a vague term. Several providers probably do not know what is required to be designated as high quality by a particular rating agency. Moreover, there are multiple quality designation programs, all using distinct criteria to determine “quality,” which further complicates the matter. Our objective was to determine the key stakeholders that provide quality designations in TJA and what criteria each organization uses in assessing quality.

Our idea of comprehensive quality is based on Avedis Donabedian’s enduring framework for healthcare quality focused on structure, process, and outcome.¹⁶ We expanded on these 3 areas and analyzed quality designations based on variables fitting into the following categories: accreditation, volume, structural, process, outcomes, patient experience, and cost/efficiency. We believe that these categories encompass a comprehensive rating system that addresses key elements of patient care. However, our results suggest that only 2 major quality designations (BCBS Distinction+ and Aetna Institutes of Quality for Orthopedic Surgery) take all such variables into account.

All quality designation programs that we analyzed required outcome data (ie, readmission and/or mortality rates within 30 days); however, only 2 programs utilized cost in their quality designation criteria (BCBS Distinction+ and Aetna Institutes of Quality for Orthopedic Surgery). Aetna Institutes of Quality for Orthopedic Surgery risk-adjusted for its cost-effectiveness calculations based on age, sex, and other unspecified conditions using a product known as Symmetry Episode Risk Groups. However, the organization also noted that although it did risk-adjust for inpatient mortality, it did not do so for pulmonary embolism or deep vein thrombosis. BCBS Distinction+ also utilized risk adjustment for its cost efficiency measure, and its step-by-step methodology is available online. Further, Consumer Reports does risk-adjust using logistic regression models in their quality analysis, but the description provided is minimal; it is noted that such risk adjustments are already completed by CMS prior to Consumer Reports acquiring the data. The CMS Compare model information is available on the CMS website. The data utilized by several organizations and presented on CMS Compare are already risk-adjusted using CMS’ approach. In contrast, UnitedHealth Premium TJR Specialty Center gathers its own data from providers and does not describe a risk adjustment methodology. Risk adjustment is important because the lack of risk adjustment may lead to physicians “cherry-picking” easy cases to boost positive outcomes, leading to increased financial benefits and higher quality ratings. Having a consistent risk adjustment formula will ensure accurate comparisons across outcomes and cost-effectiveness measures used by quality designation programs.

Factors considered for quality designation varied greatly from one organization to the other. The range of categories of factors considered varied from 1 (Consumer Reports only considered outcome data) to all 7 categories (BCBS Distinction+ and Aetna Institutes of Quality for Orthopedic Surgery). Our findings are consistent with the work by Keswani and colleagues,⁸ which showed that there is likely variation in factors considered when rating hospital quality more broadly. Our work suggests that quality designation formulas do not appear to get more consistent when focused on TJA.

We found that all organizations in our analysis published the providers earning their quality designation. However, TJC does not provide publicly a detailed methodology on how to qualify for its quality designation. The price to purchase the necessary manual for this information is $146.00 for accredited organizations and $186.00 for all others.¹⁷ For large healthcare providers, this is not a large sum of money. Nonetheless, this provides an additional hurdle for stakeholders to gain a full understanding of the requirements to receive a TJC Gold Seal for Orthopedics.

Previous work has evaluated the consistency of and the variety of means of gauging healthcare quality. Previous work by Rothberg and colleagues¹⁸ comparing hospital rankings across 5 common consumer-oriented websites found disagreement on hospital rankings within any diagnosis and even among metrics such as mortality. Another study by Halasyamani and Davis¹⁹ found that CMS Compare and USNWR rankings were dissimilar and the authors attributed the discrepancy to different methodologies. In addition, a study by Krumholz and colleagues²⁰ focused on Internet report cards, which measured the appropriate use of select medications and mortality rates for acute myocardial infarction as the quality metrics. The authors found that, in aggregate, there was a clear difference in quality of care and outcomes but that comparisons between 2 hospitals provided poor discrimination.²⁰ Other work has analyzed the increasing trend of online ratings of orthopedic surgeons by patients.²¹ However, there remains no agreed-upon definition of quality. Thus, the use of the term “quality” in several studies may be misleading.

Our results must be interpreted keeping the limitations of our work in mind. First, we used expert knowledge and a public search engine to develop our list of organizations that provide TJA quality designations. However, there is a possibility that we did not include all relevant organizations. Second, although all authors reviewed the final data, it is possible that there was human error in the analysis of each organization’s quality designation criteria.

CONCLUSION

As healthcare progresses further toward a system that rewards providers for delivering value to patients, accurately defining and measuring quality becomes critical because it can be suggestive of value to patients, payers, and providers. Furthermore, it gives providers a goal to focus on as they strive to improve the value of care they deliver to patients. Measuring healthcare quality is currently a novel, imperfect science,²² and there continues to be a debate about what factors should be included in a quality designation formula. Nonetheless, more and more quality designations and performance measurements are being created for orthopedic care, including total hip and total knee arthroplasty. In fact, in 2016, The Leapfrog Group added readmission for patients undergoing TJA to its survey.²³ Consensus on a quality definition may facilitate the movement toward a value-based healthcare system. Future research should evaluate strategies for gaining consensus among stakeholders for a universal quality metric in TJA. Surgeons, hospitals, payers, and most importantly patients should play critical roles in defining quality.