Resident duty hour restrictions were initially implemented in New York in 1989 with New York State Code 405 in response to a patient death in a New York City Emergency Department.1 This case initiated an evaluation of potential risks to patient safety when residents were inadequately supervised and overfatigued. In 2003, the Accreditation Council for Graduate Medical Education (ACGME) implemented resident duty hours nationally due to concerns for patient safety and quality of care.2 These restrictions involved the implementation of the 80‐hour work week (averaged over 4 weeks), a maximum duty length of 30 hours, and prescriptive supervision guidelines. In December 2008, the Institute of Medicine (IOM) proposed additional changes to further restrict resident duty hours which also included overnight protected sleep periods and additional days off per month.3 The ACGME responded by mandating new resident duty hour restrictions in October 2010 which will be implemented in July 2011. The ACGME's new changes include a change in the maximum duty hour length for residents in their first year of training (PGY‐1) of 16 hours. Residents in their second year of training (PGY‐2) level and above may work a maximum of 24 hours with an additional 4 hours for transition of care and resident education. The ACGME strongly recommends strategic napping, but do not have a protected overnight sleep period in place4 (Table 1).

There is growing concern regarding the impact of these new resident duty hour restrictions on the coverage of inpatient services, particularly during the overnight period. To our knowledge, there is no published national data on how pediatric inpatient teaching services are staffed at night. The objective of this study was to survey the current landscape of pediatric resident coverage of noncritical care inpatient teaching services. In addition, we sought to explore how changes in work hour restrictions might affect the role of pediatric hospitalists in training programs.

METHODS

We developed an institutional review board (IRB)‐approved Web‐based electronic survey. The survey consisted of 17 questions. The survey obtained information regarding the demographics of the program including: number of residents, daily patient census per ward intern, information regarding staff‐only pediatric ward services, overnight coverage, and current attending in‐house overnight coverage (see Appendix). We also examined the prevalence of pediatric hospitalists in training programs, their current role in staffing patients, and how that role may change with the implementation of additional resident duty hour restrictions. Initially, the survey was reviewed and tested by several pediatric hospitalists and program directors. It was then reviewed and approved by the Association of Pediatric Program Director (APPD) research task force. The survey was sent out to 196 US pediatric residency programs via the APPD listserve in January 2010. Program directors were given the option of completing it themselves or specifically designating someone else to complete it. Two reminders were sent. We then sent an additional request for program participation on the pediatric hospitalist listserve. All data was collected by February 2010.

RESULTS

One hundred twenty unique responses were received (61% of total pediatric residency programs). As of 2009, this represented 5201 pediatric residents (58% of total pediatric residents). The average program size was 43 residents (range: 12‐156 residents, median 43). The average daily patient census per ward intern during daytime hours was 6.65 patients (range: 3‐17, median 6). Twenty percent of training programs had staff‐only (no residents) pediatric ward services during daytime hours. In the programs with both staff‐only and resident pediatric ward services, only 19% of patients were covered by the staff‐only teams and 81% of patients were covered by resident teams.

During the overnight period, 86% of resident teams did not have caps on the number of new patient admissions. An average of 3.6 providers per training program were in‐house overnight to accept patient admissions to pediatric wards. Ninety‐four percent of these providers in‐house were residents (399 residents in‐house/425 total providers in‐house each night).

Twenty‐five percent of the training programs that responded to the survey had pediatric hospitalist attendings in‐house at night. This included both overnight and partial nights (ie, until midnight). Other attendings in‐house at night include: neonatal intensive care unit (NICU) attendings (53% of programs), pediatric intensive care unit (PICU) attendings (46% of programs), Pediatric Emergency Medicine attendings (65% of programs), and Pediatric Surgery attendings (6.4% of programs). Twenty‐two percent of programs had no in‐house attendings at night (Figure 1).

Figure 1

Pediatric hospitalists were involved with 84% (n = 97) of training programs. Sixty percent (n = 58) of the pediatric hospitalist teams were staffed with both teaching attendings and residents. Fourteen percent (n = 14) of the pediatric hospitalist teams did not involve residents (staff‐only) and 25% (n = 25) had both types of teams. Specifically, of the programs that had pediatric hospitalists, 20% (n = 19) of them had hospitalist attendings in‐house 24 hours per day and 13% (n = 12) of teams had hospitalist attendings in‐house into the evening hours for a varying amount of time. Of the programs with hospitalist attendings in‐house 24 hours per day, 52% (n = 11) had started this coverage within the past 3 years.

Looking towards the future, and prior to the enactment of the October 2010 ACGME standards, 31% (n = 35) of the training programs that lacked 24/7 hospitalist in‐house coverage in January 2010 anticipated adding this level of coverage within the next 5 years. Notably, 70% (n = 81) of training programs felt that further resident work hour restrictions, which have since been enacted, would likely require the addition of more hospitalist attendings at night. Our survey allowed program directors to make open‐ended comments on how further work hour restrictions may change inpatient staffing in noncritical care inpatient teaching services.

DISCUSSION

To our knowledge, this was the first national study of pediatric resident coverage in noncritical care inpatient teaching services. While there was significant variation in how inpatient teaching services were covered across these programs, in January 2010, residents were involved in the majority of patient care with only 20% of programs having attending‐only hospitalist teams during the daytime. During the overnight period, the proportion of patient care provided by residents became even more significant with residents representing 94% of the total in‐house providers accepting new admissions. While pediatric hospitalists were prevalent at these training programs, their role in direct patient care overnight was limited. Only 6% of total in‐house providers accepting admissions at night were pediatric hospitalists.

The comments made by program directors are representative of the overall concerns regarding changes to resident work hours (see Table 2). In a position statement by the Association of Pediatric Program Directors in regards to the IOM recommendations, concerns were raised stating that the recommendations of the IOM Committee are intended to enhance patient safety without appropriate consideration for the educational and professional development of trainees.5 While the newly mandated ACGME standards are different than the previous IOM recommendations, it is clear that there will be very significant changes to accommodate these new standards. Our study was done prior to the new ACGME's standards. At the time of the survey, less than a third of programs were anticipating the addition of 24/7 pediatric hospitalist coverage; however, if resident work hours were further restricted, 70% of programs felt that additional hospitalists would be needed. This is a significant increase in the previously anticipated need for overnight attending hospitalist coverage, especially in light of the further restrictions mandated by the ACGME. We know that the response of New York State programs to the 405 regulations varied by program size, but all made significant changes to accommodate the new standards.6 It is clear that many program directors nationally are anticipating significant changes to their residencies when these new restrictions are enacted. The respondents in our survey felt that pediatric hospitalists are going to have to play an even bigger role at night when additional resident work hour restrictions are put into place.

Pediatric hospital medicine remains a rapidly growing field.7 Eighty‐four percent of pediatric training programs utilize pediatric hospitalists. Over 60% of these pediatric hospitalist teams are involved in teaching teams with residents. While we did not directly study the supply and demand of pediatric hospitalists, there is some concern that even despite its rapid growth, the supply of pediatric hospitalists will not keep up with the demand when further resident work hours restrictions are implemented. At time of submission, a cost‐analysis has not yet been publicly published on the ACGME's new changes. There is data available based on the IOM's 2008 recommendations. A study by Nuckols and Escarce8 suggests that if the IOM's recommendations were implemented, the entire healthcare system nationally would have to develop and fill new full‐time positions equal to 5001 attending physicians, 5984 midlevel providers (nurse practitioners or physician assistants), 320 licensed vocational nurses, 229 nursing aides, and 45 laboratory technicians. This would be equivalent to adding an additional 8247 residency positions across all specialties.810 While the ACGME's new mandated changes are different than the IOM's recommendations, they will also restrict resident duty hours that we believe could lead to gaps in patient care requiring significant personnel changes in the healthcare system.

There are several limitations to our study. We did not study the role of pediatric subspecialty fellows and their involvement in pediatric inpatient services in these training programs. We also did not study the prevalence and use of resident night float systems. While night floats may be used in some programs, it may become more prevalent with the possible restriction in intern work hours down to 16 hours. As with any survey, there remains both volunteer and nonresponse bias with the programs that decide to complete or disregard the survey. Finally, there remains some concern over the data collection after the survey was sent out to the hospitalist listserve. Pediatric hospitalists may have incorrectly filled out the data for their program after their program director had already completed the survey. We attempted to minimize this problem by specifically instructing hospitalists to encourage their program director to fill out the survey if they had not already done so. We also compared computer Internet Protocol (IP) addresses and actual program responses, before and after the hospitalist e‐mail was sent, in an attempt to minimize the chance of including duplicated responses from the same program. Lastly, the January 2010 survey predated the October 2010 ACGME response to the IOM recommendations, and the responses may be different now that the specific restrictions have been mandated with an actual implementation date.

CONCLUSIONS

This study shows that pediatric teaching services varied significantly in how they provided overnight coverage in 2010 prior to new ACGME recommendations. Overall, residents were providing the overwhelming majority of the patient care overnight in pediatric training programs. While hospitalists were prevalent in pediatric training programs, in 2010 they had limited roles in direct patient care at night. The ACGME has now mandated additional residency work hour restrictions to be implemented July 2011. With these restrictions, hospitalists will likely need to expand their services, and additional hospitalists will be needed to provide overnight coverage. It is unclear where those hospitalists will come from and what their role will be. It is also unclear what the impact of increased demand and changed job description will be on the continued evolution of the field of Pediatric Hospital Medicine.

Future work needs to be done to establish benchmarks for inpatient coverage. The benchmarks could include guidelines on balancing patient safety with resident education. This may also involve the implementation of resident night float models. There needs to be monitoring on how changes in resident work hours and staffing affect coverage and, ultimately, how changes affect patient and resident outcomes.

Resident duty hour restrictions were initially implemented in New York in 1989 with New York State Code 405 in response to a patient death in a New York City Emergency Department.1 This case initiated an evaluation of potential risks to patient safety when residents were inadequately supervised and overfatigued. In 2003, the Accreditation Council for Graduate Medical Education (ACGME) implemented resident duty hours nationally due to concerns for patient safety and quality of care.2 These restrictions involved the implementation of the 80‐hour work week (averaged over 4 weeks), a maximum duty length of 30 hours, and prescriptive supervision guidelines. In December 2008, the Institute of Medicine (IOM) proposed additional changes to further restrict resident duty hours which also included overnight protected sleep periods and additional days off per month.3 The ACGME responded by mandating new resident duty hour restrictions in October 2010 which will be implemented in July 2011. The ACGME's new changes include a change in the maximum duty hour length for residents in their first year of training (PGY‐1) of 16 hours. Residents in their second year of training (PGY‐2) level and above may work a maximum of 24 hours with an additional 4 hours for transition of care and resident education. The ACGME strongly recommends strategic napping, but do not have a protected overnight sleep period in place4 (Table 1).

There is growing concern regarding the impact of these new resident duty hour restrictions on the coverage of inpatient services, particularly during the overnight period. To our knowledge, there is no published national data on how pediatric inpatient teaching services are staffed at night. The objective of this study was to survey the current landscape of pediatric resident coverage of noncritical care inpatient teaching services. In addition, we sought to explore how changes in work hour restrictions might affect the role of pediatric hospitalists in training programs.

METHODS

We developed an institutional review board (IRB)‐approved Web‐based electronic survey. The survey consisted of 17 questions. The survey obtained information regarding the demographics of the program including: number of residents, daily patient census per ward intern, information regarding staff‐only pediatric ward services, overnight coverage, and current attending in‐house overnight coverage (see Appendix). We also examined the prevalence of pediatric hospitalists in training programs, their current role in staffing patients, and how that role may change with the implementation of additional resident duty hour restrictions. Initially, the survey was reviewed and tested by several pediatric hospitalists and program directors. It was then reviewed and approved by the Association of Pediatric Program Director (APPD) research task force. The survey was sent out to 196 US pediatric residency programs via the APPD listserve in January 2010. Program directors were given the option of completing it themselves or specifically designating someone else to complete it. Two reminders were sent. We then sent an additional request for program participation on the pediatric hospitalist listserve. All data was collected by February 2010.

RESULTS

One hundred twenty unique responses were received (61% of total pediatric residency programs). As of 2009, this represented 5201 pediatric residents (58% of total pediatric residents). The average program size was 43 residents (range: 12‐156 residents, median 43). The average daily patient census per ward intern during daytime hours was 6.65 patients (range: 3‐17, median 6). Twenty percent of training programs had staff‐only (no residents) pediatric ward services during daytime hours. In the programs with both staff‐only and resident pediatric ward services, only 19% of patients were covered by the staff‐only teams and 81% of patients were covered by resident teams.

During the overnight period, 86% of resident teams did not have caps on the number of new patient admissions. An average of 3.6 providers per training program were in‐house overnight to accept patient admissions to pediatric wards. Ninety‐four percent of these providers in‐house were residents (399 residents in‐house/425 total providers in‐house each night).

Twenty‐five percent of the training programs that responded to the survey had pediatric hospitalist attendings in‐house at night. This included both overnight and partial nights (ie, until midnight). Other attendings in‐house at night include: neonatal intensive care unit (NICU) attendings (53% of programs), pediatric intensive care unit (PICU) attendings (46% of programs), Pediatric Emergency Medicine attendings (65% of programs), and Pediatric Surgery attendings (6.4% of programs). Twenty‐two percent of programs had no in‐house attendings at night (Figure 1).

Figure 1

Pediatric hospitalists were involved with 84% (n = 97) of training programs. Sixty percent (n = 58) of the pediatric hospitalist teams were staffed with both teaching attendings and residents. Fourteen percent (n = 14) of the pediatric hospitalist teams did not involve residents (staff‐only) and 25% (n = 25) had both types of teams. Specifically, of the programs that had pediatric hospitalists, 20% (n = 19) of them had hospitalist attendings in‐house 24 hours per day and 13% (n = 12) of teams had hospitalist attendings in‐house into the evening hours for a varying amount of time. Of the programs with hospitalist attendings in‐house 24 hours per day, 52% (n = 11) had started this coverage within the past 3 years.

Looking towards the future, and prior to the enactment of the October 2010 ACGME standards, 31% (n = 35) of the training programs that lacked 24/7 hospitalist in‐house coverage in January 2010 anticipated adding this level of coverage within the next 5 years. Notably, 70% (n = 81) of training programs felt that further resident work hour restrictions, which have since been enacted, would likely require the addition of more hospitalist attendings at night. Our survey allowed program directors to make open‐ended comments on how further work hour restrictions may change inpatient staffing in noncritical care inpatient teaching services.

DISCUSSION

To our knowledge, this was the first national study of pediatric resident coverage in noncritical care inpatient teaching services. While there was significant variation in how inpatient teaching services were covered across these programs, in January 2010, residents were involved in the majority of patient care with only 20% of programs having attending‐only hospitalist teams during the daytime. During the overnight period, the proportion of patient care provided by residents became even more significant with residents representing 94% of the total in‐house providers accepting new admissions. While pediatric hospitalists were prevalent at these training programs, their role in direct patient care overnight was limited. Only 6% of total in‐house providers accepting admissions at night were pediatric hospitalists.

The comments made by program directors are representative of the overall concerns regarding changes to resident work hours (see Table 2). In a position statement by the Association of Pediatric Program Directors in regards to the IOM recommendations, concerns were raised stating that the recommendations of the IOM Committee are intended to enhance patient safety without appropriate consideration for the educational and professional development of trainees.5 While the newly mandated ACGME standards are different than the previous IOM recommendations, it is clear that there will be very significant changes to accommodate these new standards. Our study was done prior to the new ACGME's standards. At the time of the survey, less than a third of programs were anticipating the addition of 24/7 pediatric hospitalist coverage; however, if resident work hours were further restricted, 70% of programs felt that additional hospitalists would be needed. This is a significant increase in the previously anticipated need for overnight attending hospitalist coverage, especially in light of the further restrictions mandated by the ACGME. We know that the response of New York State programs to the 405 regulations varied by program size, but all made significant changes to accommodate the new standards.6 It is clear that many program directors nationally are anticipating significant changes to their residencies when these new restrictions are enacted. The respondents in our survey felt that pediatric hospitalists are going to have to play an even bigger role at night when additional resident work hour restrictions are put into place.

Pediatric hospital medicine remains a rapidly growing field.7 Eighty‐four percent of pediatric training programs utilize pediatric hospitalists. Over 60% of these pediatric hospitalist teams are involved in teaching teams with residents. While we did not directly study the supply and demand of pediatric hospitalists, there is some concern that even despite its rapid growth, the supply of pediatric hospitalists will not keep up with the demand when further resident work hours restrictions are implemented. At time of submission, a cost‐analysis has not yet been publicly published on the ACGME's new changes. There is data available based on the IOM's 2008 recommendations. A study by Nuckols and Escarce8 suggests that if the IOM's recommendations were implemented, the entire healthcare system nationally would have to develop and fill new full‐time positions equal to 5001 attending physicians, 5984 midlevel providers (nurse practitioners or physician assistants), 320 licensed vocational nurses, 229 nursing aides, and 45 laboratory technicians. This would be equivalent to adding an additional 8247 residency positions across all specialties.810 While the ACGME's new mandated changes are different than the IOM's recommendations, they will also restrict resident duty hours that we believe could lead to gaps in patient care requiring significant personnel changes in the healthcare system.

There are several limitations to our study. We did not study the role of pediatric subspecialty fellows and their involvement in pediatric inpatient services in these training programs. We also did not study the prevalence and use of resident night float systems. While night floats may be used in some programs, it may become more prevalent with the possible restriction in intern work hours down to 16 hours. As with any survey, there remains both volunteer and nonresponse bias with the programs that decide to complete or disregard the survey. Finally, there remains some concern over the data collection after the survey was sent out to the hospitalist listserve. Pediatric hospitalists may have incorrectly filled out the data for their program after their program director had already completed the survey. We attempted to minimize this problem by specifically instructing hospitalists to encourage their program director to fill out the survey if they had not already done so. We also compared computer Internet Protocol (IP) addresses and actual program responses, before and after the hospitalist e‐mail was sent, in an attempt to minimize the chance of including duplicated responses from the same program. Lastly, the January 2010 survey predated the October 2010 ACGME response to the IOM recommendations, and the responses may be different now that the specific restrictions have been mandated with an actual implementation date.

CONCLUSIONS

This study shows that pediatric teaching services varied significantly in how they provided overnight coverage in 2010 prior to new ACGME recommendations. Overall, residents were providing the overwhelming majority of the patient care overnight in pediatric training programs. While hospitalists were prevalent in pediatric training programs, in 2010 they had limited roles in direct patient care at night. The ACGME has now mandated additional residency work hour restrictions to be implemented July 2011. With these restrictions, hospitalists will likely need to expand their services, and additional hospitalists will be needed to provide overnight coverage. It is unclear where those hospitalists will come from and what their role will be. It is also unclear what the impact of increased demand and changed job description will be on the continued evolution of the field of Pediatric Hospital Medicine.

Future work needs to be done to establish benchmarks for inpatient coverage. The benchmarks could include guidelines on balancing patient safety with resident education. This may also involve the implementation of resident night float models. There needs to be monitoring on how changes in resident work hours and staffing affect coverage and, ultimately, how changes affect patient and resident outcomes.

Resident duty hour restrictions were initially implemented in New York in 1989 with New York State Code 405 in response to a patient death in a New York City Emergency Department.1 This case initiated an evaluation of potential risks to patient safety when residents were inadequately supervised and overfatigued. In 2003, the Accreditation Council for Graduate Medical Education (ACGME) implemented resident duty hours nationally due to concerns for patient safety and quality of care.2 These restrictions involved the implementation of the 80‐hour work week (averaged over 4 weeks), a maximum duty length of 30 hours, and prescriptive supervision guidelines. In December 2008, the Institute of Medicine (IOM) proposed additional changes to further restrict resident duty hours which also included overnight protected sleep periods and additional days off per month.3 The ACGME responded by mandating new resident duty hour restrictions in October 2010 which will be implemented in July 2011. The ACGME's new changes include a change in the maximum duty hour length for residents in their first year of training (PGY‐1) of 16 hours. Residents in their second year of training (PGY‐2) level and above may work a maximum of 24 hours with an additional 4 hours for transition of care and resident education. The ACGME strongly recommends strategic napping, but do not have a protected overnight sleep period in place4 (Table 1).

There is growing concern regarding the impact of these new resident duty hour restrictions on the coverage of inpatient services, particularly during the overnight period. To our knowledge, there is no published national data on how pediatric inpatient teaching services are staffed at night. The objective of this study was to survey the current landscape of pediatric resident coverage of noncritical care inpatient teaching services. In addition, we sought to explore how changes in work hour restrictions might affect the role of pediatric hospitalists in training programs.

METHODS

We developed an institutional review board (IRB)‐approved Web‐based electronic survey. The survey consisted of 17 questions. The survey obtained information regarding the demographics of the program including: number of residents, daily patient census per ward intern, information regarding staff‐only pediatric ward services, overnight coverage, and current attending in‐house overnight coverage (see Appendix). We also examined the prevalence of pediatric hospitalists in training programs, their current role in staffing patients, and how that role may change with the implementation of additional resident duty hour restrictions. Initially, the survey was reviewed and tested by several pediatric hospitalists and program directors. It was then reviewed and approved by the Association of Pediatric Program Director (APPD) research task force. The survey was sent out to 196 US pediatric residency programs via the APPD listserve in January 2010. Program directors were given the option of completing it themselves or specifically designating someone else to complete it. Two reminders were sent. We then sent an additional request for program participation on the pediatric hospitalist listserve. All data was collected by February 2010.

RESULTS

One hundred twenty unique responses were received (61% of total pediatric residency programs). As of 2009, this represented 5201 pediatric residents (58% of total pediatric residents). The average program size was 43 residents (range: 12‐156 residents, median 43). The average daily patient census per ward intern during daytime hours was 6.65 patients (range: 3‐17, median 6). Twenty percent of training programs had staff‐only (no residents) pediatric ward services during daytime hours. In the programs with both staff‐only and resident pediatric ward services, only 19% of patients were covered by the staff‐only teams and 81% of patients were covered by resident teams.

During the overnight period, 86% of resident teams did not have caps on the number of new patient admissions. An average of 3.6 providers per training program were in‐house overnight to accept patient admissions to pediatric wards. Ninety‐four percent of these providers in‐house were residents (399 residents in‐house/425 total providers in‐house each night).

Twenty‐five percent of the training programs that responded to the survey had pediatric hospitalist attendings in‐house at night. This included both overnight and partial nights (ie, until midnight). Other attendings in‐house at night include: neonatal intensive care unit (NICU) attendings (53% of programs), pediatric intensive care unit (PICU) attendings (46% of programs), Pediatric Emergency Medicine attendings (65% of programs), and Pediatric Surgery attendings (6.4% of programs). Twenty‐two percent of programs had no in‐house attendings at night (Figure 1).

Figure 1

Pediatric hospitalists were involved with 84% (n = 97) of training programs. Sixty percent (n = 58) of the pediatric hospitalist teams were staffed with both teaching attendings and residents. Fourteen percent (n = 14) of the pediatric hospitalist teams did not involve residents (staff‐only) and 25% (n = 25) had both types of teams. Specifically, of the programs that had pediatric hospitalists, 20% (n = 19) of them had hospitalist attendings in‐house 24 hours per day and 13% (n = 12) of teams had hospitalist attendings in‐house into the evening hours for a varying amount of time. Of the programs with hospitalist attendings in‐house 24 hours per day, 52% (n = 11) had started this coverage within the past 3 years.

Looking towards the future, and prior to the enactment of the October 2010 ACGME standards, 31% (n = 35) of the training programs that lacked 24/7 hospitalist in‐house coverage in January 2010 anticipated adding this level of coverage within the next 5 years. Notably, 70% (n = 81) of training programs felt that further resident work hour restrictions, which have since been enacted, would likely require the addition of more hospitalist attendings at night. Our survey allowed program directors to make open‐ended comments on how further work hour restrictions may change inpatient staffing in noncritical care inpatient teaching services.

DISCUSSION

To our knowledge, this was the first national study of pediatric resident coverage in noncritical care inpatient teaching services. While there was significant variation in how inpatient teaching services were covered across these programs, in January 2010, residents were involved in the majority of patient care with only 20% of programs having attending‐only hospitalist teams during the daytime. During the overnight period, the proportion of patient care provided by residents became even more significant with residents representing 94% of the total in‐house providers accepting new admissions. While pediatric hospitalists were prevalent at these training programs, their role in direct patient care overnight was limited. Only 6% of total in‐house providers accepting admissions at night were pediatric hospitalists.

The comments made by program directors are representative of the overall concerns regarding changes to resident work hours (see Table 2). In a position statement by the Association of Pediatric Program Directors in regards to the IOM recommendations, concerns were raised stating that the recommendations of the IOM Committee are intended to enhance patient safety without appropriate consideration for the educational and professional development of trainees.5 While the newly mandated ACGME standards are different than the previous IOM recommendations, it is clear that there will be very significant changes to accommodate these new standards. Our study was done prior to the new ACGME's standards. At the time of the survey, less than a third of programs were anticipating the addition of 24/7 pediatric hospitalist coverage; however, if resident work hours were further restricted, 70% of programs felt that additional hospitalists would be needed. This is a significant increase in the previously anticipated need for overnight attending hospitalist coverage, especially in light of the further restrictions mandated by the ACGME. We know that the response of New York State programs to the 405 regulations varied by program size, but all made significant changes to accommodate the new standards.6 It is clear that many program directors nationally are anticipating significant changes to their residencies when these new restrictions are enacted. The respondents in our survey felt that pediatric hospitalists are going to have to play an even bigger role at night when additional resident work hour restrictions are put into place.

Pediatric hospital medicine remains a rapidly growing field.7 Eighty‐four percent of pediatric training programs utilize pediatric hospitalists. Over 60% of these pediatric hospitalist teams are involved in teaching teams with residents. While we did not directly study the supply and demand of pediatric hospitalists, there is some concern that even despite its rapid growth, the supply of pediatric hospitalists will not keep up with the demand when further resident work hours restrictions are implemented. At time of submission, a cost‐analysis has not yet been publicly published on the ACGME's new changes. There is data available based on the IOM's 2008 recommendations. A study by Nuckols and Escarce8 suggests that if the IOM's recommendations were implemented, the entire healthcare system nationally would have to develop and fill new full‐time positions equal to 5001 attending physicians, 5984 midlevel providers (nurse practitioners or physician assistants), 320 licensed vocational nurses, 229 nursing aides, and 45 laboratory technicians. This would be equivalent to adding an additional 8247 residency positions across all specialties.810 While the ACGME's new mandated changes are different than the IOM's recommendations, they will also restrict resident duty hours that we believe could lead to gaps in patient care requiring significant personnel changes in the healthcare system.

There are several limitations to our study. We did not study the role of pediatric subspecialty fellows and their involvement in pediatric inpatient services in these training programs. We also did not study the prevalence and use of resident night float systems. While night floats may be used in some programs, it may become more prevalent with the possible restriction in intern work hours down to 16 hours. As with any survey, there remains both volunteer and nonresponse bias with the programs that decide to complete or disregard the survey. Finally, there remains some concern over the data collection after the survey was sent out to the hospitalist listserve. Pediatric hospitalists may have incorrectly filled out the data for their program after their program director had already completed the survey. We attempted to minimize this problem by specifically instructing hospitalists to encourage their program director to fill out the survey if they had not already done so. We also compared computer Internet Protocol (IP) addresses and actual program responses, before and after the hospitalist e‐mail was sent, in an attempt to minimize the chance of including duplicated responses from the same program. Lastly, the January 2010 survey predated the October 2010 ACGME response to the IOM recommendations, and the responses may be different now that the specific restrictions have been mandated with an actual implementation date.

CONCLUSIONS

This study shows that pediatric teaching services varied significantly in how they provided overnight coverage in 2010 prior to new ACGME recommendations. Overall, residents were providing the overwhelming majority of the patient care overnight in pediatric training programs. While hospitalists were prevalent in pediatric training programs, in 2010 they had limited roles in direct patient care at night. The ACGME has now mandated additional residency work hour restrictions to be implemented July 2011. With these restrictions, hospitalists will likely need to expand their services, and additional hospitalists will be needed to provide overnight coverage. It is unclear where those hospitalists will come from and what their role will be. It is also unclear what the impact of increased demand and changed job description will be on the continued evolution of the field of Pediatric Hospital Medicine.

Future work needs to be done to establish benchmarks for inpatient coverage. The benchmarks could include guidelines on balancing patient safety with resident education. This may also involve the implementation of resident night float models. There needs to be monitoring on how changes in resident work hours and staffing affect coverage and, ultimately, how changes affect patient and resident outcomes.

Feedback has long been recognized as pivotal to the attainment of clinical acumen and skills in medical training.1 Formative feedback can give trainees insight into their strengths and weaknesses, and provide them with clear goals and methods to attain those goals.1, 2 In fact, feedback given regularly over time by a respected figure has shown to improve physician performance.3 However, most faculty are not trained to provide effective feedback. As a result, supervisors often believe they are giving more feedback than trainees believe they are receiving, and residents receive little feedback that they perceive as useful.4 Most residents receive little to no feedback on their communications skills4 or professionalism,5 and rarely receive corrective feedback.6, 7

Faculty may fail to give feedback to residents for a number of reasons. Those barriers most commonly cited in the literature are discomfort with criticizing residents,6, 7 lack of time,4 and lack of direct observation of residents in clinical settings.810 Several studies have looked at tools to guide feedback and address the barrier of discomfort with criticism.6, 7, 11 Some showed improvements in overall feedback, though often supervisors gave only positive feedback and avoided giving feedback about weaknesses.6, 7, 11 Despite the recognition of lack of time as a barrier to feedback,4 most studies on feedback interventions thus far have not included setting aside time for the feedback to occur.6, 7, 11, 12 Finally, a number of studies utilized objective structured clinical examinations (OSCEs) coupled with immediate feedback to improve direct observation of residents, with success in improving feedback related to the encounter.9, 10, 13 To address the gaps in the current literature, the goals of our study were to address 2 specific barriers to feedback for residents: lack of time and discomfort with giving feedback.

The aim of this study was to improve Internal Medicine (IM) residents' and attendings' experiences with feedback on the wards using a pocket card and a dedicated feedback session. We developed and evaluated the pocket feedback card and session for faculty to improve the quality and frequency of their feedback to residents in the inpatient setting. We performed a randomized trial to evaluate our intervention. We hypothesized that the intervention would: 1) improve the quality and quantity of attendings' feedback given to IM ward residents; and 2) improve attendings' comfort with feedback delivery on the wards.

PARTICIPANTS AND METHODS

Intervention Design

We designed a pocket feedback card to guide a feedback session and assist attendings in giving useful feedback to IM residents on the wards (Figure 1).14 The individual items and categories were adapted from the Accreditation Council for Graduate Medical Education (ACGME) Common Program Requirements Core Competencies section and were revised via the expert consensus of the authors.14 We included 20 items related to resident skills, knowledge, attitudes, and behaviors important to the care of hospitalized patients, grouped under the 6 ACGME core competency domains.14 Many of these items correspond to competencies in the Society of Hospital Medicine (SHM) Core Competencies; in particular, the categories of Systems‐Based Practice and Practice‐Based Learning mirror competencies in the SHM Core Competencies Healthcare Systems chapter.15 Each item utilized a 5‐point Likert scale (1 = very poor, 3 = at expected level, 5 = superior) to evaluate resident performance (Figure 1). We created this card to serve as a directive and specific guide for attendings to provide feedback about specific domains and to give more constructive feedback. The card was to be used during a specific dedicated feedback session in order to overcome the commonly cited barrier of lack of time.

Figure 1

RESULTS

During the 6‐month study period, 34 teaching attendings (over 36 attending inpatient blocks) and 93 IM residents (over 111 resident inpatient blocks) participated in the study. Thirty‐four of 36 attending surveys and 96 of 111 resident surveys were completed. The overall survey response rates for residents and attendings were 85% and 94%, respectively. Two attendings participated during 2 separate blocks, first in the control group and then in the intervention group, and 18 residents participated during 2 separate blocks. No attendings or residents participated more than twice.

Resident survey response rate was 81.2% in the intervention group and 87.3% in the control group (Table 1). Residents in the intervention group reported receiving more feedback regarding skills they did well (89.7% vs 63.6%, P = 0.004) and skills needing improvement (51.3% vs 25.5%, P = 0.02) than those in the control group. In addition, more intervention residents reported receiving useful information regarding how to improve their skills (53.8% vs 27.3%, P = 0.01), and reported actually improving both their clinical skills (61.5% vs 27.8%, P = 0.001) and their professionalism/communication skills (51.3% vs 29.1%, P = 0.03) based on feedback received from attendings.

The attending survey response rates for the intervention and control groups were 100% and 90%, respectively. In general, both groups of attendings reported that they were comfortable giving feedback and that they did, in fact, give feedback in each area during their ward block (Table 2). More intervention attendings felt that at least 1 of their residents improved their professionalism/communication skills based on the feedback given (76.9% vs 31.1%, P = 0.02). There were no other significant differences between the groups of attendings.

Intervention attendings also shared their attitudes toward the feedback card and session. A majority felt that using 1 attending rounds as a feedback session helped create a dedicated time for giving feedback (68.8%), and that the feedback card helped them to give specific, constructive feedback (62.5%). Most attendings reported they would use the feedback card and session again during future inpatient blocks (81%), and would recommend them to other attendings (75%).

Qualitative data from intervention attending interviews revealed further thoughts about the feedback card and feedback session. Most attendings interviewed (7/8) felt that the card was useful for the structure and topic guidance it provided. Half felt that setting aside time for feedback was the more useful component. The other half reported that, because they usually set aside time for feedback regardless, the card was more useful. None of the attendings felt that the feedback card or session was detrimental for patient care or education, and many said that the intervention had positive effects on these areas. For example, 1 attending said that the session added to patient care because I used particular [patient] cases as examples for giving feedback.

DISCUSSION

In this randomized study, we found that a simple pocket feedback card and dedicated feedback session was acceptable to ward attendings and improved resident satisfaction with feedback. Unlike most prior studies of feedback, we demonstrated more feedback around skills needing improvement, and intervention residents felt the feedback they received helped them improve their skills. Our educational intervention was unique in that it combined a pocket card to structure feedback content and dedicated time to structure the feedback process, to address 2 of the major barriers to giving feedback: lack of time and lack of comfort.

The pocket card itself as a tool for improving feedback is innovative and valuable. As a short but directive guide, the card supports attendings' delivery of relevant and specific feedback about residents' performance, and because it is based on the ACGME competencies, it may help attendings focus feedback on areas in which they will later evaluate residents. The inclusion of a prespecified time for giving feedback was important as well, in that it allowed for face‐to‐face feedback to occur, as opposed to a passing comment after a presentation or brief notes in a written final evaluation. Both the card and the feedback session seemed equally important for the success of this intervention, with attitudes varying based on individual attending preferences. Those who usually set aside time for feedback on their own found the card more useful, whereas those who had more trouble finding time for feedback found the specific session more useful. Most attendings found the intervention as a whole helpful, and without any detrimental effects on patient care or education. The card and session may be particularly valuable for hospital attendings, given their growing presence as teachers and supervisors for residents, and their busy days on the wards.

Our study results have important implications for resident training in the hospital. Improving resident receipt of feedback about strengths and weaknesses is an ACGME training requirement, and specific guidance about how to improve skills is critical for focusing improvement efforts. Previous studies have demonstrated that directive feedback in medical training can lead to a variety of performance improvements, including better evaluations by other professionals,9, 16 and objective improvements in resident communication skills,17 chart documentation,18 and clinical management of patients.11, 15, 19 By improving the quality of feedback across several domains and facilitating the feedback process, our intervention may lead to similar improvements. Future studies should examine the global impact of guided feedback as in our study. Perhaps most importantly, attendings found the intervention acceptable and would recommend its use, implying longer term sustainability of its integration into the hospital routine.

One strength of our study was its prospective randomized design. Despite the importance of rigor in medical education research, there remains a paucity of randomized studies to evaluate educational interventions for residents in inpatient settings. Few studies of feedback interventions in particular have performed randomized trials,5, 6, 11 and only one has examined a feedback intervention in a randomized fashion in the inpatient setting.12 This evaluation of a 20‐minute intervention, and a reminder card for supervising attendings to improve written and verbal feedback to residents, modestly improved the amount of verbal feedback given to residents, but did not impact the number of residents receiving mid‐rotation feedback or feedback overall as our study did by report.12

There were several important limitations to our study. First, because this was a single institution study, we only achieved modest sample sizes, particularly in the attending groups, and were unable to assess all of the differences in attending attitudes related to feedback. Second, control and intervention participants were on service simultaneously, which may have led to contamination of the control group and an underestimation of the true impact of our intervention. Since residents were not exclusive to 1 study group on 1 occasion (18 of the 93 residents participated during 2 separate blocks), our results may be biased. In particular, those residents who had the intervention first, and were subsequently in the control group, may have rated the control experience worse than they would have otherwise, creating a bias in favor of a positive result for our intervention. Nonetheless, we believe this situation was uncommon and the potential associated bias minimal. Further, this study assessed attitudes related to feedback and self‐reported knowledge and skills, but did not directly assess resident knowledge, skills, or patient outcomes. We recognize the importance of these outcomes and hope that future interventions can determine these important downstream effects of feedback. We were also unable to assess the card and session's impact on attendings' comfort with feedback, because most attendings in both groups reported feeling comfortable giving feedback. This result may indicate that attendings actually are comfortable giving feedback, or may suggest some element of social desirability bias. Finally, in this study, we designed an intervention which combined the pocket card and dedicated feedback time. We did not quantitatively examine the effect of either component alone, and it is unclear if offering the feedback card without protected time or offering protected time without a guide would have impacted feedback on the wards. However, qualitative data from our study support the use of both components, and implementing the 2 components together is feasible in any inpatient teaching setting.

Despite these limitations, protected time for feedback guided by a pocket feedback card is a simple intervention that appears to improve feedback quantity and quality for ward residents, and guides them to improve their performance. Our low‐intensity intervention helped attendings give residents the tools to improve their clinical and communication skills. An opportunity to make a positive impact on resident education with such a small intervention is rare. The use of a feedback card with protected feedback time could be easily implemented in any training program, and is a valuable tool for busy hospitalists who are more commonly supervising residents on their inpatient rotations.

Feedback has long been recognized as pivotal to the attainment of clinical acumen and skills in medical training.1 Formative feedback can give trainees insight into their strengths and weaknesses, and provide them with clear goals and methods to attain those goals.1, 2 In fact, feedback given regularly over time by a respected figure has shown to improve physician performance.3 However, most faculty are not trained to provide effective feedback. As a result, supervisors often believe they are giving more feedback than trainees believe they are receiving, and residents receive little feedback that they perceive as useful.4 Most residents receive little to no feedback on their communications skills4 or professionalism,5 and rarely receive corrective feedback.6, 7

Faculty may fail to give feedback to residents for a number of reasons. Those barriers most commonly cited in the literature are discomfort with criticizing residents,6, 7 lack of time,4 and lack of direct observation of residents in clinical settings.810 Several studies have looked at tools to guide feedback and address the barrier of discomfort with criticism.6, 7, 11 Some showed improvements in overall feedback, though often supervisors gave only positive feedback and avoided giving feedback about weaknesses.6, 7, 11 Despite the recognition of lack of time as a barrier to feedback,4 most studies on feedback interventions thus far have not included setting aside time for the feedback to occur.6, 7, 11, 12 Finally, a number of studies utilized objective structured clinical examinations (OSCEs) coupled with immediate feedback to improve direct observation of residents, with success in improving feedback related to the encounter.9, 10, 13 To address the gaps in the current literature, the goals of our study were to address 2 specific barriers to feedback for residents: lack of time and discomfort with giving feedback.

The aim of this study was to improve Internal Medicine (IM) residents' and attendings' experiences with feedback on the wards using a pocket card and a dedicated feedback session. We developed and evaluated the pocket feedback card and session for faculty to improve the quality and frequency of their feedback to residents in the inpatient setting. We performed a randomized trial to evaluate our intervention. We hypothesized that the intervention would: 1) improve the quality and quantity of attendings' feedback given to IM ward residents; and 2) improve attendings' comfort with feedback delivery on the wards.

PARTICIPANTS AND METHODS

Intervention Design

We designed a pocket feedback card to guide a feedback session and assist attendings in giving useful feedback to IM residents on the wards (Figure 1).14 The individual items and categories were adapted from the Accreditation Council for Graduate Medical Education (ACGME) Common Program Requirements Core Competencies section and were revised via the expert consensus of the authors.14 We included 20 items related to resident skills, knowledge, attitudes, and behaviors important to the care of hospitalized patients, grouped under the 6 ACGME core competency domains.14 Many of these items correspond to competencies in the Society of Hospital Medicine (SHM) Core Competencies; in particular, the categories of Systems‐Based Practice and Practice‐Based Learning mirror competencies in the SHM Core Competencies Healthcare Systems chapter.15 Each item utilized a 5‐point Likert scale (1 = very poor, 3 = at expected level, 5 = superior) to evaluate resident performance (Figure 1). We created this card to serve as a directive and specific guide for attendings to provide feedback about specific domains and to give more constructive feedback. The card was to be used during a specific dedicated feedback session in order to overcome the commonly cited barrier of lack of time.

Figure 1

RESULTS

During the 6‐month study period, 34 teaching attendings (over 36 attending inpatient blocks) and 93 IM residents (over 111 resident inpatient blocks) participated in the study. Thirty‐four of 36 attending surveys and 96 of 111 resident surveys were completed. The overall survey response rates for residents and attendings were 85% and 94%, respectively. Two attendings participated during 2 separate blocks, first in the control group and then in the intervention group, and 18 residents participated during 2 separate blocks. No attendings or residents participated more than twice.

Resident survey response rate was 81.2% in the intervention group and 87.3% in the control group (Table 1). Residents in the intervention group reported receiving more feedback regarding skills they did well (89.7% vs 63.6%, P = 0.004) and skills needing improvement (51.3% vs 25.5%, P = 0.02) than those in the control group. In addition, more intervention residents reported receiving useful information regarding how to improve their skills (53.8% vs 27.3%, P = 0.01), and reported actually improving both their clinical skills (61.5% vs 27.8%, P = 0.001) and their professionalism/communication skills (51.3% vs 29.1%, P = 0.03) based on feedback received from attendings.

The attending survey response rates for the intervention and control groups were 100% and 90%, respectively. In general, both groups of attendings reported that they were comfortable giving feedback and that they did, in fact, give feedback in each area during their ward block (Table 2). More intervention attendings felt that at least 1 of their residents improved their professionalism/communication skills based on the feedback given (76.9% vs 31.1%, P = 0.02). There were no other significant differences between the groups of attendings.

Intervention attendings also shared their attitudes toward the feedback card and session. A majority felt that using 1 attending rounds as a feedback session helped create a dedicated time for giving feedback (68.8%), and that the feedback card helped them to give specific, constructive feedback (62.5%). Most attendings reported they would use the feedback card and session again during future inpatient blocks (81%), and would recommend them to other attendings (75%).

Qualitative data from intervention attending interviews revealed further thoughts about the feedback card and feedback session. Most attendings interviewed (7/8) felt that the card was useful for the structure and topic guidance it provided. Half felt that setting aside time for feedback was the more useful component. The other half reported that, because they usually set aside time for feedback regardless, the card was more useful. None of the attendings felt that the feedback card or session was detrimental for patient care or education, and many said that the intervention had positive effects on these areas. For example, 1 attending said that the session added to patient care because I used particular [patient] cases as examples for giving feedback.

DISCUSSION

In this randomized study, we found that a simple pocket feedback card and dedicated feedback session was acceptable to ward attendings and improved resident satisfaction with feedback. Unlike most prior studies of feedback, we demonstrated more feedback around skills needing improvement, and intervention residents felt the feedback they received helped them improve their skills. Our educational intervention was unique in that it combined a pocket card to structure feedback content and dedicated time to structure the feedback process, to address 2 of the major barriers to giving feedback: lack of time and lack of comfort.

The pocket card itself as a tool for improving feedback is innovative and valuable. As a short but directive guide, the card supports attendings' delivery of relevant and specific feedback about residents' performance, and because it is based on the ACGME competencies, it may help attendings focus feedback on areas in which they will later evaluate residents. The inclusion of a prespecified time for giving feedback was important as well, in that it allowed for face‐to‐face feedback to occur, as opposed to a passing comment after a presentation or brief notes in a written final evaluation. Both the card and the feedback session seemed equally important for the success of this intervention, with attitudes varying based on individual attending preferences. Those who usually set aside time for feedback on their own found the card more useful, whereas those who had more trouble finding time for feedback found the specific session more useful. Most attendings found the intervention as a whole helpful, and without any detrimental effects on patient care or education. The card and session may be particularly valuable for hospital attendings, given their growing presence as teachers and supervisors for residents, and their busy days on the wards.

Our study results have important implications for resident training in the hospital. Improving resident receipt of feedback about strengths and weaknesses is an ACGME training requirement, and specific guidance about how to improve skills is critical for focusing improvement efforts. Previous studies have demonstrated that directive feedback in medical training can lead to a variety of performance improvements, including better evaluations by other professionals,9, 16 and objective improvements in resident communication skills,17 chart documentation,18 and clinical management of patients.11, 15, 19 By improving the quality of feedback across several domains and facilitating the feedback process, our intervention may lead to similar improvements. Future studies should examine the global impact of guided feedback as in our study. Perhaps most importantly, attendings found the intervention acceptable and would recommend its use, implying longer term sustainability of its integration into the hospital routine.

One strength of our study was its prospective randomized design. Despite the importance of rigor in medical education research, there remains a paucity of randomized studies to evaluate educational interventions for residents in inpatient settings. Few studies of feedback interventions in particular have performed randomized trials,5, 6, 11 and only one has examined a feedback intervention in a randomized fashion in the inpatient setting.12 This evaluation of a 20‐minute intervention, and a reminder card for supervising attendings to improve written and verbal feedback to residents, modestly improved the amount of verbal feedback given to residents, but did not impact the number of residents receiving mid‐rotation feedback or feedback overall as our study did by report.12

There were several important limitations to our study. First, because this was a single institution study, we only achieved modest sample sizes, particularly in the attending groups, and were unable to assess all of the differences in attending attitudes related to feedback. Second, control and intervention participants were on service simultaneously, which may have led to contamination of the control group and an underestimation of the true impact of our intervention. Since residents were not exclusive to 1 study group on 1 occasion (18 of the 93 residents participated during 2 separate blocks), our results may be biased. In particular, those residents who had the intervention first, and were subsequently in the control group, may have rated the control experience worse than they would have otherwise, creating a bias in favor of a positive result for our intervention. Nonetheless, we believe this situation was uncommon and the potential associated bias minimal. Further, this study assessed attitudes related to feedback and self‐reported knowledge and skills, but did not directly assess resident knowledge, skills, or patient outcomes. We recognize the importance of these outcomes and hope that future interventions can determine these important downstream effects of feedback. We were also unable to assess the card and session's impact on attendings' comfort with feedback, because most attendings in both groups reported feeling comfortable giving feedback. This result may indicate that attendings actually are comfortable giving feedback, or may suggest some element of social desirability bias. Finally, in this study, we designed an intervention which combined the pocket card and dedicated feedback time. We did not quantitatively examine the effect of either component alone, and it is unclear if offering the feedback card without protected time or offering protected time without a guide would have impacted feedback on the wards. However, qualitative data from our study support the use of both components, and implementing the 2 components together is feasible in any inpatient teaching setting.

Despite these limitations, protected time for feedback guided by a pocket feedback card is a simple intervention that appears to improve feedback quantity and quality for ward residents, and guides them to improve their performance. Our low‐intensity intervention helped attendings give residents the tools to improve their clinical and communication skills. An opportunity to make a positive impact on resident education with such a small intervention is rare. The use of a feedback card with protected feedback time could be easily implemented in any training program, and is a valuable tool for busy hospitalists who are more commonly supervising residents on their inpatient rotations.

Feedback has long been recognized as pivotal to the attainment of clinical acumen and skills in medical training.1 Formative feedback can give trainees insight into their strengths and weaknesses, and provide them with clear goals and methods to attain those goals.1, 2 In fact, feedback given regularly over time by a respected figure has shown to improve physician performance.3 However, most faculty are not trained to provide effective feedback. As a result, supervisors often believe they are giving more feedback than trainees believe they are receiving, and residents receive little feedback that they perceive as useful.4 Most residents receive little to no feedback on their communications skills4 or professionalism,5 and rarely receive corrective feedback.6, 7

Faculty may fail to give feedback to residents for a number of reasons. Those barriers most commonly cited in the literature are discomfort with criticizing residents,6, 7 lack of time,4 and lack of direct observation of residents in clinical settings.810 Several studies have looked at tools to guide feedback and address the barrier of discomfort with criticism.6, 7, 11 Some showed improvements in overall feedback, though often supervisors gave only positive feedback and avoided giving feedback about weaknesses.6, 7, 11 Despite the recognition of lack of time as a barrier to feedback,4 most studies on feedback interventions thus far have not included setting aside time for the feedback to occur.6, 7, 11, 12 Finally, a number of studies utilized objective structured clinical examinations (OSCEs) coupled with immediate feedback to improve direct observation of residents, with success in improving feedback related to the encounter.9, 10, 13 To address the gaps in the current literature, the goals of our study were to address 2 specific barriers to feedback for residents: lack of time and discomfort with giving feedback.

The aim of this study was to improve Internal Medicine (IM) residents' and attendings' experiences with feedback on the wards using a pocket card and a dedicated feedback session. We developed and evaluated the pocket feedback card and session for faculty to improve the quality and frequency of their feedback to residents in the inpatient setting. We performed a randomized trial to evaluate our intervention. We hypothesized that the intervention would: 1) improve the quality and quantity of attendings' feedback given to IM ward residents; and 2) improve attendings' comfort with feedback delivery on the wards.

PARTICIPANTS AND METHODS

Intervention Design

We designed a pocket feedback card to guide a feedback session and assist attendings in giving useful feedback to IM residents on the wards (Figure 1).14 The individual items and categories were adapted from the Accreditation Council for Graduate Medical Education (ACGME) Common Program Requirements Core Competencies section and were revised via the expert consensus of the authors.14 We included 20 items related to resident skills, knowledge, attitudes, and behaviors important to the care of hospitalized patients, grouped under the 6 ACGME core competency domains.14 Many of these items correspond to competencies in the Society of Hospital Medicine (SHM) Core Competencies; in particular, the categories of Systems‐Based Practice and Practice‐Based Learning mirror competencies in the SHM Core Competencies Healthcare Systems chapter.15 Each item utilized a 5‐point Likert scale (1 = very poor, 3 = at expected level, 5 = superior) to evaluate resident performance (Figure 1). We created this card to serve as a directive and specific guide for attendings to provide feedback about specific domains and to give more constructive feedback. The card was to be used during a specific dedicated feedback session in order to overcome the commonly cited barrier of lack of time.

Figure 1

RESULTS

During the 6‐month study period, 34 teaching attendings (over 36 attending inpatient blocks) and 93 IM residents (over 111 resident inpatient blocks) participated in the study. Thirty‐four of 36 attending surveys and 96 of 111 resident surveys were completed. The overall survey response rates for residents and attendings were 85% and 94%, respectively. Two attendings participated during 2 separate blocks, first in the control group and then in the intervention group, and 18 residents participated during 2 separate blocks. No attendings or residents participated more than twice.

Resident survey response rate was 81.2% in the intervention group and 87.3% in the control group (Table 1). Residents in the intervention group reported receiving more feedback regarding skills they did well (89.7% vs 63.6%, P = 0.004) and skills needing improvement (51.3% vs 25.5%, P = 0.02) than those in the control group. In addition, more intervention residents reported receiving useful information regarding how to improve their skills (53.8% vs 27.3%, P = 0.01), and reported actually improving both their clinical skills (61.5% vs 27.8%, P = 0.001) and their professionalism/communication skills (51.3% vs 29.1%, P = 0.03) based on feedback received from attendings.

The attending survey response rates for the intervention and control groups were 100% and 90%, respectively. In general, both groups of attendings reported that they were comfortable giving feedback and that they did, in fact, give feedback in each area during their ward block (Table 2). More intervention attendings felt that at least 1 of their residents improved their professionalism/communication skills based on the feedback given (76.9% vs 31.1%, P = 0.02). There were no other significant differences between the groups of attendings.

Intervention attendings also shared their attitudes toward the feedback card and session. A majority felt that using 1 attending rounds as a feedback session helped create a dedicated time for giving feedback (68.8%), and that the feedback card helped them to give specific, constructive feedback (62.5%). Most attendings reported they would use the feedback card and session again during future inpatient blocks (81%), and would recommend them to other attendings (75%).

Qualitative data from intervention attending interviews revealed further thoughts about the feedback card and feedback session. Most attendings interviewed (7/8) felt that the card was useful for the structure and topic guidance it provided. Half felt that setting aside time for feedback was the more useful component. The other half reported that, because they usually set aside time for feedback regardless, the card was more useful. None of the attendings felt that the feedback card or session was detrimental for patient care or education, and many said that the intervention had positive effects on these areas. For example, 1 attending said that the session added to patient care because I used particular [patient] cases as examples for giving feedback.

DISCUSSION

In this randomized study, we found that a simple pocket feedback card and dedicated feedback session was acceptable to ward attendings and improved resident satisfaction with feedback. Unlike most prior studies of feedback, we demonstrated more feedback around skills needing improvement, and intervention residents felt the feedback they received helped them improve their skills. Our educational intervention was unique in that it combined a pocket card to structure feedback content and dedicated time to structure the feedback process, to address 2 of the major barriers to giving feedback: lack of time and lack of comfort.

The pocket card itself as a tool for improving feedback is innovative and valuable. As a short but directive guide, the card supports attendings' delivery of relevant and specific feedback about residents' performance, and because it is based on the ACGME competencies, it may help attendings focus feedback on areas in which they will later evaluate residents. The inclusion of a prespecified time for giving feedback was important as well, in that it allowed for face‐to‐face feedback to occur, as opposed to a passing comment after a presentation or brief notes in a written final evaluation. Both the card and the feedback session seemed equally important for the success of this intervention, with attitudes varying based on individual attending preferences. Those who usually set aside time for feedback on their own found the card more useful, whereas those who had more trouble finding time for feedback found the specific session more useful. Most attendings found the intervention as a whole helpful, and without any detrimental effects on patient care or education. The card and session may be particularly valuable for hospital attendings, given their growing presence as teachers and supervisors for residents, and their busy days on the wards.

Our study results have important implications for resident training in the hospital. Improving resident receipt of feedback about strengths and weaknesses is an ACGME training requirement, and specific guidance about how to improve skills is critical for focusing improvement efforts. Previous studies have demonstrated that directive feedback in medical training can lead to a variety of performance improvements, including better evaluations by other professionals,9, 16 and objective improvements in resident communication skills,17 chart documentation,18 and clinical management of patients.11, 15, 19 By improving the quality of feedback across several domains and facilitating the feedback process, our intervention may lead to similar improvements. Future studies should examine the global impact of guided feedback as in our study. Perhaps most importantly, attendings found the intervention acceptable and would recommend its use, implying longer term sustainability of its integration into the hospital routine.

One strength of our study was its prospective randomized design. Despite the importance of rigor in medical education research, there remains a paucity of randomized studies to evaluate educational interventions for residents in inpatient settings. Few studies of feedback interventions in particular have performed randomized trials,5, 6, 11 and only one has examined a feedback intervention in a randomized fashion in the inpatient setting.12 This evaluation of a 20‐minute intervention, and a reminder card for supervising attendings to improve written and verbal feedback to residents, modestly improved the amount of verbal feedback given to residents, but did not impact the number of residents receiving mid‐rotation feedback or feedback overall as our study did by report.12

There were several important limitations to our study. First, because this was a single institution study, we only achieved modest sample sizes, particularly in the attending groups, and were unable to assess all of the differences in attending attitudes related to feedback. Second, control and intervention participants were on service simultaneously, which may have led to contamination of the control group and an underestimation of the true impact of our intervention. Since residents were not exclusive to 1 study group on 1 occasion (18 of the 93 residents participated during 2 separate blocks), our results may be biased. In particular, those residents who had the intervention first, and were subsequently in the control group, may have rated the control experience worse than they would have otherwise, creating a bias in favor of a positive result for our intervention. Nonetheless, we believe this situation was uncommon and the potential associated bias minimal. Further, this study assessed attitudes related to feedback and self‐reported knowledge and skills, but did not directly assess resident knowledge, skills, or patient outcomes. We recognize the importance of these outcomes and hope that future interventions can determine these important downstream effects of feedback. We were also unable to assess the card and session's impact on attendings' comfort with feedback, because most attendings in both groups reported feeling comfortable giving feedback. This result may indicate that attendings actually are comfortable giving feedback, or may suggest some element of social desirability bias. Finally, in this study, we designed an intervention which combined the pocket card and dedicated feedback time. We did not quantitatively examine the effect of either component alone, and it is unclear if offering the feedback card without protected time or offering protected time without a guide would have impacted feedback on the wards. However, qualitative data from our study support the use of both components, and implementing the 2 components together is feasible in any inpatient teaching setting.

Despite these limitations, protected time for feedback guided by a pocket feedback card is a simple intervention that appears to improve feedback quantity and quality for ward residents, and guides them to improve their performance. Our low‐intensity intervention helped attendings give residents the tools to improve their clinical and communication skills. An opportunity to make a positive impact on resident education with such a small intervention is rare. The use of a feedback card with protected feedback time could be easily implemented in any training program, and is a valuable tool for busy hospitalists who are more commonly supervising residents on their inpatient rotations.

Each year in North America, over 7 million adults are hospitalized with a medical illness.1 Acute illness and decreased mobility in hospital places patients at increased risk for venous thromboembolism (VTE), which includes deep vein thrombosis (DVT) and life‐threatening pulmonary embolism (PE).2 Since VTE remains the most preventable cause of death in hospitalized patients, numerous studies have aimed at reducing the incidence of hospital‐acquired DVT. Aside from cost, the impact of VTE to the healthcare system is felt not only by those who diagnose and treat VTE, but also by those responsible for correcting the severe bleeding that can result from inappropriate use of thromboprophylaxis. Approximately 60% of symptomatic VTE occurs in medical patients, and recent hospitalization for medical illness accounts for 25% of all community‐diagnosed VTE. The Agency for Health Research and Quality ranks DVT prevention as the top priority out of 79 patient safety initiatives, and expert consensus groups provide a strong recommendation that DVT prophylaxis with a low‐dose anticoagulant should be administered to at‐risk hospitalized medical patients.2, 3

Despite the availability, efficacy, and safety of DVT prophylaxis,2 it is discouraging that only 21% to 62% of medical patients receive prophylaxis,49 and only 16% to 40% receive appropriate prophylaxis.46, 1012 However, 70% to 90% of patients in other at‐risk groups, such as surgical patients or critically ill patients, receive prophylaxis.1316 The reason why DVT prophylaxis is so underutilized in medical patients is unclear, as explanations for low rates of clinical practice guideline utilization are multifaceted,17 and few studies have investigated the barriers to optimal thromboprophylaxis.1820

To explore possible reasons for this disparity between evidence and practice, we conducted a cross‐sectional survey of 4 clinician groups involved in the care of hospitalized medical patients. Our objective was to identify barriers and potential solutions to the underutilization of DVT prophylaxis in hospitalized medical patients.

METHODS

RESULTS

Survey Responses

The overall response rate was 36.3% (563/1553), with 65.5% (211/322) of nurses, 40.4% (78/193) of physicians, 24.1% (242/1002) of pharmacists, and 88.8% (32/36) of physiotherapists completing surveys. When pharmacists were removed from the response rate calculation (since it was expected that many of those receiving the survey were not in a primarily hospital‐based practice), the overall response rate rose to 58.3% (321/551). Excluded were 9.2% (52/563) of returned surveys, as respondents indicated the topic was not relevant to their practice. Five hundred eleven surveys were included in the final analysis (Figure 1).

Figure 1

Importance, Effectiveness, Safety, and Appropriateness of DVT Prophylaxis Strategies

DVT prophylaxis was perceived across clinician groups as important (mean score 6.4; 95% CI 6.3 to 6.5), safe (mean 5.5; 95% CI 5.4 to 5.6), and effective (mean 5.5; 95% CI 5.4 to 6.6) (Figure 2). The mean score for the appropriateness of current DVT prophylaxis practices was 3.5 (95% CI 3.4 to 3.7), suggesting an overall perception of underutilization. However, by respondent groups, DVT prophylaxis was considered to be underutilized by physicians (mean 2.5; 95% CI 2.3 to 2.7) and pharmacists (mean 3.1; 95% CI 2.9 to 3.2), while nurses (mean 4.3; 95% CI 4.2 to 4.5) and physiotherapists (mean 3.8; 95% CI, 3.4 to 4.2) tended to consider current strategies as appropriate.

Figure 2

Potential Barriers to DVT Prophylaxis Utilization

Figure 3 demonstrates that no single barrier to DVT prophylaxis utilization was dominant and no barriers were considered very important. Perceived barriers carrying comparable weight were: concerns about bleeding (mean 4.8; 95% CI 4.6 to 4.9); lack of clear indications (mean 4.6; 95% CI 4.5 to 4.8) and contraindications to DVT prophylaxis (mean 4.4; 95% CI 4.3 to 4.6); lack of awareness about effectiveness of DVT prophylaxis (mean 4.5; 95% CI 4.3 to 4.7); and lack of time to consider DVT prophylaxis in every patient (mean 4.4; 95% CI 4.3 to 4.6). Patient discomfort from subcutaneous injections was perceived as the least important barrier (mean 3.8; 95% CI 3.6 to 4.0). Physicians perceived lack of awareness about the effectiveness of DVT prophylaxis as the most important barrier (mean 4.0; 95% CI 3.5 to 4.4), whereas concern about bleeding was dominant among non‐physicians (nurses' mean 5.2; 95% CI 5.0 to 5.5; pharmacists' mean 4.7; 95% CI 4.5 to 4.9; physiotherapists' mean 4.6; 95% CI 3.9 to 5.3).

Figure 3

Potential Success and Feasibility of Interventions to Optimize DVT Prophylaxis Utilization

Interventions considered across clinician groups as highly potentially successful were: preprinted order sheets (5.7; 95% CI 5.6 to 5.8); pharmacist reminders to physicians (mean 5.3; 95% CI 5.1 to 5.4); computerized reminders to physicians (mean 5.0; 95% CI 4.9 to 5.2); and use of a local opinion leader (mean 5.0; 95% CI 4.9 to 5.2). Interventions considered highly potentially feasible were: posters (mean 5.7; CI 5.6 to 5.8); preprinted order sheets (mean 5.5; 95% CI 5.4 to 5.7); laminated pocket cards (mean 5.4; 95% CI 5.2 to 5.5); multidisciplinary educational meetings (mean 5.0; 95% CI 4.9 to 5.2); and pharmacist reminders to physicians (mean 5.0; 95% CI 4.9 to 5.1). Preprinted orders and pharmacist reminders were perceived by all clinician groups as having both high potential success and feasibility (Figure 4).

Figure 4

Perceptions on Which Clinician Group Is Best Able to Assess and Implement DVT Prophylaxis

Respondents were divided between considering the attending physician and the bedside nurse as best able to perform a daily assessment of patients' need for DVT prophylaxis (43.4% [204/470] vs 44.0% [207/470], respectively). Respondents from these groups each predominantly thought this responsibility was theirs, with 68.1% (49/72) of physicians and 61.5% (123/200) of nurses perceiving this as their responsibility (Figure 5).

Figure 5

Forty‐one percent (193/471) of respondents perceived the attending physician as best able to ensure that DVT prophylaxis is ordered, while 31.2% (147/471) identified the pharmacist and 23.3% (110/471) identified the bedside nurse as best suited to this role. Among pharmacists, 66.3% (114/172) perceived that the attending pharmacist is best able to perform this task. Among respondents, 61.9% (296/478) felt the bedside nurse is best able to ensure adherence to DVT prophylaxis, with good agreement among all clinician groups.

DISCUSSION

Our survey identified several perceived barriers to optimizing DVT prophylaxis, consistent with those reported in the White Paper sponsored by the American Public Health Association.23 While no single barrier outlined in our survey was dominant, 2 novel barriers were identified: misperception of DVT prophylaxis underutilization, and confusion about roles and responsibilities in the area of DVT prophylaxis. Attention to these barriers may be helpful in developing an intervention aimed at bridging the gap between evidence and practice.

While our survey demonstrates agreement across clinician groups on the importance, efficacy, and safety of DVT prophylaxis, the discordant perceptions that exist about whether DVT prophylaxis is utilized appropriately is an important concern. Physician and pharmacist‐respondents demonstrated awareness that thromboprophylaxis is underutilized in medical patients. However, despite overwhelming published evidence to the contrary, nurses responding to our survey did not tend to recognize the problem of DVT prophylaxis underutilization in hospitalized medical patients. This knowledge deficit may be a significant barrier particularly since the pooled group of respondents indicated that nurses are among those caregivers best able to conduct a daily assessment of patients' need for DVT prophylaxis. A possible explanation for the finding that nurses and physiotherapists demonstrated a relative lack of awareness of the problem of DVT prophylaxis underutilization is ward‐specific healthcare priorities. Nursing and physiotherapy care on surgical wards is aimed at preventing postoperative complications, including DVT. However, its primary focus on medical wards is the management of acute medical problems. Prevention of hospital‐related complications, such as DVT, is often a secondary focus. Therefore, ensuring that all clinician groups are educated about the problem of DVT prophylaxis underutilization is necessary to drive quality improvement. A physician‐based survey on antithrombotic therapies demonstrated a similar need for education on guideline recommendations.20

A second important barrier identified in our survey is that both attending nurses and physicians feel that daily assessment of a patient's need for DVT prophylaxis is their responsibility. Confusion about roles and responsibilities in this area of patient care was reported by Cook et al., who identified that multidisciplinary care was perceived as a barrier to effective VTE prevention.18 Uncertainty as to which group should take ownership of DVT prophylaxis can lead to a diffusion of responsibility, a lack of accountability, and a gap in care. A resolution to whether DVT risk assessment is a nursing or a physician role could be reached through increased interdisciplinary communication and provision of clear definitions of roles to hospital staff.

Survey respondents felt that preprinted orders and pharmacist reminders to physicians were potentially successful and feasible strategies to optimize DVT prophylaxis. These components could be part of a simple tool to initiate prophylaxis. While electronic alerts have been shown to increase prophylaxis rates,24 we suspect that many respondents did not view these as highly important because of limited use of computerized order entry at their facilities. Interestingly, survey respondents did not perceive audit‐and‐feedback systems or local opinion leaders as potentially successful, though previous studies have demonstrated that they can change clinician behavior.25, 26 This may be because respondents may not be aware of the strength of technology‐based interventions (eg, electronic orders) and the role of opinion leaders, and the evidence in support of such interventions.24, 26 A systematic review of studies to improve DVT prophylaxis in hospitals reported that a combination of multiple active strategies is most effective, particularly those that link physician reminders with audit‐and‐feedback.27 For example, in the define study, a multicomponent intervention consisting of interactive educational sessions, verbal and computerized prompts, and individual performance feedback significantly improved adherence to DVT prophylaxis guidelines in critically ill patients.28 Whether a similar intervention could improve adherence to DVT prophylaxis guidelines in hospitalized medical patients merits further study. Any intervention must be paired with better education about which patients should, and should not, receive prophylaxis, as this may address many reported barriers in our survey (including concerns about bleeding). Respondents' uncertainty about these issues is not surprising, as studies of DVT prophylaxis in medical patients are not plentiful.2 However, recent guidelines do identify subgroups of medically ill patients in whom DVT prophylaxis is indicated.2 A clear and simple DVT risk assessment algorithm that identifies medical patients in whom DVT prophylaxis should (or should not) be administered may help to overcome respondents' concerns.

A limitation of our survey is the overall response rate of 36.3%, largely driven by the considerable number of nonresponding pharmacists (n = 760, reflecting 49% of the entire sample). However, the majority of the pharmacists were likely not hospital‐based, were thus not a target of this study, and their low response rate is not surprising. After excluding pharmacists, the response rate was 58.3% (321/551), which is consistent with response rates of other large‐sample surveys.29 The lower response rate for physicians and pharmacists may also reflect web‐based survey dissemination which, despite its feasibility, has lower response rates than paper‐based dissemination.3032 While the sample of physicians was relatively small compared to the other respondent groups surveyed, we aimed to identify barriers to actually implementing VTE prophylaxis, not just ordering prophylaxis, which is a multidisciplinary concern.

Although this survey was based on Canadian healthcare providers' perspectives, we believe the results are generalizable since both US and Canadian‐based studies have found that VTE prophylaxis is underutilized among hospitalized medical patients.4, 6 Furthermore, the American College of Chest Physicians (ACCP) guidelines on VTE prophylaxis, which are well‐recognized in both the United States and Canada, were developed with input from Canadian and American content experts.2 And while the US and Canadian healthcare systems are organized differently, at the patient‐care level, the roles of healthcare professionals are very similar. The generalizability of our findings is, however, limited by the institutional characteristics of respondents. We do not purport that the responses of any of the 4 clinician groups are generalizable to those groups as a whole. Although we surveyed clinicians in teaching and nonteaching, urban and rural practices, perceptions about DVT prophylaxis may be influenced by other factors, including the availability of local preprinted orders, electronic medical records, and quality improvement programs. Another potential limitation is that we did not assess all possible strategies to improve DVT prophylaxis, such as nurse practitioners and computerized decision support systems. These were purposely excluded, as they are not financially feasible in all centers, and thus not generalizable. Finally, like all self‐administered surveys, our findings reflect respondents' perceptions rather than objective observations about practice.

In conclusion, we identified novel and important barriers to optimal DVT prophylaxis utilization and potential interventions to address this important safety concern in hospitalized medical patients. To overcome some of these barriers, we propose an educational intervention prior to delivery of a top‐down, evidence‐based intervention to first increase healthcare providers' knowledge of the safety of DVT prophylaxis, system and team‐based approaches, and which interventions are most likely to be successful so as to encourage greater compliance with the intervention. A top‐down, system‐wide approach, involving the entire healthcare team and hospital administrators, can help drive this communication. As DVT prophylaxis becomes an increasingly important component in hospital accreditation, such solutions become appealing to facilitate change in practices. Results of this survey may inform future knowledge translation interventions by eliminating perceived barriers to DVT prophylaxis and by incorporating strategies that are perceived by healthcare professionals to be successful, feasible, and supported by evidence.

Each year in North America, over 7 million adults are hospitalized with a medical illness.1 Acute illness and decreased mobility in hospital places patients at increased risk for venous thromboembolism (VTE), which includes deep vein thrombosis (DVT) and life‐threatening pulmonary embolism (PE).2 Since VTE remains the most preventable cause of death in hospitalized patients, numerous studies have aimed at reducing the incidence of hospital‐acquired DVT. Aside from cost, the impact of VTE to the healthcare system is felt not only by those who diagnose and treat VTE, but also by those responsible for correcting the severe bleeding that can result from inappropriate use of thromboprophylaxis. Approximately 60% of symptomatic VTE occurs in medical patients, and recent hospitalization for medical illness accounts for 25% of all community‐diagnosed VTE. The Agency for Health Research and Quality ranks DVT prevention as the top priority out of 79 patient safety initiatives, and expert consensus groups provide a strong recommendation that DVT prophylaxis with a low‐dose anticoagulant should be administered to at‐risk hospitalized medical patients.2, 3

Despite the availability, efficacy, and safety of DVT prophylaxis,2 it is discouraging that only 21% to 62% of medical patients receive prophylaxis,49 and only 16% to 40% receive appropriate prophylaxis.46, 1012 However, 70% to 90% of patients in other at‐risk groups, such as surgical patients or critically ill patients, receive prophylaxis.1316 The reason why DVT prophylaxis is so underutilized in medical patients is unclear, as explanations for low rates of clinical practice guideline utilization are multifaceted,17 and few studies have investigated the barriers to optimal thromboprophylaxis.1820

To explore possible reasons for this disparity between evidence and practice, we conducted a cross‐sectional survey of 4 clinician groups involved in the care of hospitalized medical patients. Our objective was to identify barriers and potential solutions to the underutilization of DVT prophylaxis in hospitalized medical patients.

METHODS

RESULTS

Survey Responses

The overall response rate was 36.3% (563/1553), with 65.5% (211/322) of nurses, 40.4% (78/193) of physicians, 24.1% (242/1002) of pharmacists, and 88.8% (32/36) of physiotherapists completing surveys. When pharmacists were removed from the response rate calculation (since it was expected that many of those receiving the survey were not in a primarily hospital‐based practice), the overall response rate rose to 58.3% (321/551). Excluded were 9.2% (52/563) of returned surveys, as respondents indicated the topic was not relevant to their practice. Five hundred eleven surveys were included in the final analysis (Figure 1).

Figure 1

Importance, Effectiveness, Safety, and Appropriateness of DVT Prophylaxis Strategies

DVT prophylaxis was perceived across clinician groups as important (mean score 6.4; 95% CI 6.3 to 6.5), safe (mean 5.5; 95% CI 5.4 to 5.6), and effective (mean 5.5; 95% CI 5.4 to 6.6) (Figure 2). The mean score for the appropriateness of current DVT prophylaxis practices was 3.5 (95% CI 3.4 to 3.7), suggesting an overall perception of underutilization. However, by respondent groups, DVT prophylaxis was considered to be underutilized by physicians (mean 2.5; 95% CI 2.3 to 2.7) and pharmacists (mean 3.1; 95% CI 2.9 to 3.2), while nurses (mean 4.3; 95% CI 4.2 to 4.5) and physiotherapists (mean 3.8; 95% CI, 3.4 to 4.2) tended to consider current strategies as appropriate.

Figure 2

Potential Barriers to DVT Prophylaxis Utilization

Figure 3 demonstrates that no single barrier to DVT prophylaxis utilization was dominant and no barriers were considered very important. Perceived barriers carrying comparable weight were: concerns about bleeding (mean 4.8; 95% CI 4.6 to 4.9); lack of clear indications (mean 4.6; 95% CI 4.5 to 4.8) and contraindications to DVT prophylaxis (mean 4.4; 95% CI 4.3 to 4.6); lack of awareness about effectiveness of DVT prophylaxis (mean 4.5; 95% CI 4.3 to 4.7); and lack of time to consider DVT prophylaxis in every patient (mean 4.4; 95% CI 4.3 to 4.6). Patient discomfort from subcutaneous injections was perceived as the least important barrier (mean 3.8; 95% CI 3.6 to 4.0). Physicians perceived lack of awareness about the effectiveness of DVT prophylaxis as the most important barrier (mean 4.0; 95% CI 3.5 to 4.4), whereas concern about bleeding was dominant among non‐physicians (nurses' mean 5.2; 95% CI 5.0 to 5.5; pharmacists' mean 4.7; 95% CI 4.5 to 4.9; physiotherapists' mean 4.6; 95% CI 3.9 to 5.3).

Figure 3

Potential Success and Feasibility of Interventions to Optimize DVT Prophylaxis Utilization

Interventions considered across clinician groups as highly potentially successful were: preprinted order sheets (5.7; 95% CI 5.6 to 5.8); pharmacist reminders to physicians (mean 5.3; 95% CI 5.1 to 5.4); computerized reminders to physicians (mean 5.0; 95% CI 4.9 to 5.2); and use of a local opinion leader (mean 5.0; 95% CI 4.9 to 5.2). Interventions considered highly potentially feasible were: posters (mean 5.7; CI 5.6 to 5.8); preprinted order sheets (mean 5.5; 95% CI 5.4 to 5.7); laminated pocket cards (mean 5.4; 95% CI 5.2 to 5.5); multidisciplinary educational meetings (mean 5.0; 95% CI 4.9 to 5.2); and pharmacist reminders to physicians (mean 5.0; 95% CI 4.9 to 5.1). Preprinted orders and pharmacist reminders were perceived by all clinician groups as having both high potential success and feasibility (Figure 4).

Figure 4

Perceptions on Which Clinician Group Is Best Able to Assess and Implement DVT Prophylaxis

Respondents were divided between considering the attending physician and the bedside nurse as best able to perform a daily assessment of patients' need for DVT prophylaxis (43.4% [204/470] vs 44.0% [207/470], respectively). Respondents from these groups each predominantly thought this responsibility was theirs, with 68.1% (49/72) of physicians and 61.5% (123/200) of nurses perceiving this as their responsibility (Figure 5).

Figure 5

Forty‐one percent (193/471) of respondents perceived the attending physician as best able to ensure that DVT prophylaxis is ordered, while 31.2% (147/471) identified the pharmacist and 23.3% (110/471) identified the bedside nurse as best suited to this role. Among pharmacists, 66.3% (114/172) perceived that the attending pharmacist is best able to perform this task. Among respondents, 61.9% (296/478) felt the bedside nurse is best able to ensure adherence to DVT prophylaxis, with good agreement among all clinician groups.

DISCUSSION

Our survey identified several perceived barriers to optimizing DVT prophylaxis, consistent with those reported in the White Paper sponsored by the American Public Health Association.23 While no single barrier outlined in our survey was dominant, 2 novel barriers were identified: misperception of DVT prophylaxis underutilization, and confusion about roles and responsibilities in the area of DVT prophylaxis. Attention to these barriers may be helpful in developing an intervention aimed at bridging the gap between evidence and practice.

While our survey demonstrates agreement across clinician groups on the importance, efficacy, and safety of DVT prophylaxis, the discordant perceptions that exist about whether DVT prophylaxis is utilized appropriately is an important concern. Physician and pharmacist‐respondents demonstrated awareness that thromboprophylaxis is underutilized in medical patients. However, despite overwhelming published evidence to the contrary, nurses responding to our survey did not tend to recognize the problem of DVT prophylaxis underutilization in hospitalized medical patients. This knowledge deficit may be a significant barrier particularly since the pooled group of respondents indicated that nurses are among those caregivers best able to conduct a daily assessment of patients' need for DVT prophylaxis. A possible explanation for the finding that nurses and physiotherapists demonstrated a relative lack of awareness of the problem of DVT prophylaxis underutilization is ward‐specific healthcare priorities. Nursing and physiotherapy care on surgical wards is aimed at preventing postoperative complications, including DVT. However, its primary focus on medical wards is the management of acute medical problems. Prevention of hospital‐related complications, such as DVT, is often a secondary focus. Therefore, ensuring that all clinician groups are educated about the problem of DVT prophylaxis underutilization is necessary to drive quality improvement. A physician‐based survey on antithrombotic therapies demonstrated a similar need for education on guideline recommendations.20

A second important barrier identified in our survey is that both attending nurses and physicians feel that daily assessment of a patient's need for DVT prophylaxis is their responsibility. Confusion about roles and responsibilities in this area of patient care was reported by Cook et al., who identified that multidisciplinary care was perceived as a barrier to effective VTE prevention.18 Uncertainty as to which group should take ownership of DVT prophylaxis can lead to a diffusion of responsibility, a lack of accountability, and a gap in care. A resolution to whether DVT risk assessment is a nursing or a physician role could be reached through increased interdisciplinary communication and provision of clear definitions of roles to hospital staff.

Survey respondents felt that preprinted orders and pharmacist reminders to physicians were potentially successful and feasible strategies to optimize DVT prophylaxis. These components could be part of a simple tool to initiate prophylaxis. While electronic alerts have been shown to increase prophylaxis rates,24 we suspect that many respondents did not view these as highly important because of limited use of computerized order entry at their facilities. Interestingly, survey respondents did not perceive audit‐and‐feedback systems or local opinion leaders as potentially successful, though previous studies have demonstrated that they can change clinician behavior.25, 26 This may be because respondents may not be aware of the strength of technology‐based interventions (eg, electronic orders) and the role of opinion leaders, and the evidence in support of such interventions.24, 26 A systematic review of studies to improve DVT prophylaxis in hospitals reported that a combination of multiple active strategies is most effective, particularly those that link physician reminders with audit‐and‐feedback.27 For example, in the define study, a multicomponent intervention consisting of interactive educational sessions, verbal and computerized prompts, and individual performance feedback significantly improved adherence to DVT prophylaxis guidelines in critically ill patients.28 Whether a similar intervention could improve adherence to DVT prophylaxis guidelines in hospitalized medical patients merits further study. Any intervention must be paired with better education about which patients should, and should not, receive prophylaxis, as this may address many reported barriers in our survey (including concerns about bleeding). Respondents' uncertainty about these issues is not surprising, as studies of DVT prophylaxis in medical patients are not plentiful.2 However, recent guidelines do identify subgroups of medically ill patients in whom DVT prophylaxis is indicated.2 A clear and simple DVT risk assessment algorithm that identifies medical patients in whom DVT prophylaxis should (or should not) be administered may help to overcome respondents' concerns.

A limitation of our survey is the overall response rate of 36.3%, largely driven by the considerable number of nonresponding pharmacists (n = 760, reflecting 49% of the entire sample). However, the majority of the pharmacists were likely not hospital‐based, were thus not a target of this study, and their low response rate is not surprising. After excluding pharmacists, the response rate was 58.3% (321/551), which is consistent with response rates of other large‐sample surveys.29 The lower response rate for physicians and pharmacists may also reflect web‐based survey dissemination which, despite its feasibility, has lower response rates than paper‐based dissemination.3032 While the sample of physicians was relatively small compared to the other respondent groups surveyed, we aimed to identify barriers to actually implementing VTE prophylaxis, not just ordering prophylaxis, which is a multidisciplinary concern.

Although this survey was based on Canadian healthcare providers' perspectives, we believe the results are generalizable since both US and Canadian‐based studies have found that VTE prophylaxis is underutilized among hospitalized medical patients.4, 6 Furthermore, the American College of Chest Physicians (ACCP) guidelines on VTE prophylaxis, which are well‐recognized in both the United States and Canada, were developed with input from Canadian and American content experts.2 And while the US and Canadian healthcare systems are organized differently, at the patient‐care level, the roles of healthcare professionals are very similar. The generalizability of our findings is, however, limited by the institutional characteristics of respondents. We do not purport that the responses of any of the 4 clinician groups are generalizable to those groups as a whole. Although we surveyed clinicians in teaching and nonteaching, urban and rural practices, perceptions about DVT prophylaxis may be influenced by other factors, including the availability of local preprinted orders, electronic medical records, and quality improvement programs. Another potential limitation is that we did not assess all possible strategies to improve DVT prophylaxis, such as nurse practitioners and computerized decision support systems. These were purposely excluded, as they are not financially feasible in all centers, and thus not generalizable. Finally, like all self‐administered surveys, our findings reflect respondents' perceptions rather than objective observations about practice.

In conclusion, we identified novel and important barriers to optimal DVT prophylaxis utilization and potential interventions to address this important safety concern in hospitalized medical patients. To overcome some of these barriers, we propose an educational intervention prior to delivery of a top‐down, evidence‐based intervention to first increase healthcare providers' knowledge of the safety of DVT prophylaxis, system and team‐based approaches, and which interventions are most likely to be successful so as to encourage greater compliance with the intervention. A top‐down, system‐wide approach, involving the entire healthcare team and hospital administrators, can help drive this communication. As DVT prophylaxis becomes an increasingly important component in hospital accreditation, such solutions become appealing to facilitate change in practices. Results of this survey may inform future knowledge translation interventions by eliminating perceived barriers to DVT prophylaxis and by incorporating strategies that are perceived by healthcare professionals to be successful, feasible, and supported by evidence.

Each year in North America, over 7 million adults are hospitalized with a medical illness.1 Acute illness and decreased mobility in hospital places patients at increased risk for venous thromboembolism (VTE), which includes deep vein thrombosis (DVT) and life‐threatening pulmonary embolism (PE).2 Since VTE remains the most preventable cause of death in hospitalized patients, numerous studies have aimed at reducing the incidence of hospital‐acquired DVT. Aside from cost, the impact of VTE to the healthcare system is felt not only by those who diagnose and treat VTE, but also by those responsible for correcting the severe bleeding that can result from inappropriate use of thromboprophylaxis. Approximately 60% of symptomatic VTE occurs in medical patients, and recent hospitalization for medical illness accounts for 25% of all community‐diagnosed VTE. The Agency for Health Research and Quality ranks DVT prevention as the top priority out of 79 patient safety initiatives, and expert consensus groups provide a strong recommendation that DVT prophylaxis with a low‐dose anticoagulant should be administered to at‐risk hospitalized medical patients.2, 3

Despite the availability, efficacy, and safety of DVT prophylaxis,2 it is discouraging that only 21% to 62% of medical patients receive prophylaxis,49 and only 16% to 40% receive appropriate prophylaxis.46, 1012 However, 70% to 90% of patients in other at‐risk groups, such as surgical patients or critically ill patients, receive prophylaxis.1316 The reason why DVT prophylaxis is so underutilized in medical patients is unclear, as explanations for low rates of clinical practice guideline utilization are multifaceted,17 and few studies have investigated the barriers to optimal thromboprophylaxis.1820

To explore possible reasons for this disparity between evidence and practice, we conducted a cross‐sectional survey of 4 clinician groups involved in the care of hospitalized medical patients. Our objective was to identify barriers and potential solutions to the underutilization of DVT prophylaxis in hospitalized medical patients.

METHODS

RESULTS

Survey Responses

The overall response rate was 36.3% (563/1553), with 65.5% (211/322) of nurses, 40.4% (78/193) of physicians, 24.1% (242/1002) of pharmacists, and 88.8% (32/36) of physiotherapists completing surveys. When pharmacists were removed from the response rate calculation (since it was expected that many of those receiving the survey were not in a primarily hospital‐based practice), the overall response rate rose to 58.3% (321/551). Excluded were 9.2% (52/563) of returned surveys, as respondents indicated the topic was not relevant to their practice. Five hundred eleven surveys were included in the final analysis (Figure 1).

Figure 1

Importance, Effectiveness, Safety, and Appropriateness of DVT Prophylaxis Strategies

DVT prophylaxis was perceived across clinician groups as important (mean score 6.4; 95% CI 6.3 to 6.5), safe (mean 5.5; 95% CI 5.4 to 5.6), and effective (mean 5.5; 95% CI 5.4 to 6.6) (Figure 2). The mean score for the appropriateness of current DVT prophylaxis practices was 3.5 (95% CI 3.4 to 3.7), suggesting an overall perception of underutilization. However, by respondent groups, DVT prophylaxis was considered to be underutilized by physicians (mean 2.5; 95% CI 2.3 to 2.7) and pharmacists (mean 3.1; 95% CI 2.9 to 3.2), while nurses (mean 4.3; 95% CI 4.2 to 4.5) and physiotherapists (mean 3.8; 95% CI, 3.4 to 4.2) tended to consider current strategies as appropriate.

Figure 2

Potential Barriers to DVT Prophylaxis Utilization

Figure 3 demonstrates that no single barrier to DVT prophylaxis utilization was dominant and no barriers were considered very important. Perceived barriers carrying comparable weight were: concerns about bleeding (mean 4.8; 95% CI 4.6 to 4.9); lack of clear indications (mean 4.6; 95% CI 4.5 to 4.8) and contraindications to DVT prophylaxis (mean 4.4; 95% CI 4.3 to 4.6); lack of awareness about effectiveness of DVT prophylaxis (mean 4.5; 95% CI 4.3 to 4.7); and lack of time to consider DVT prophylaxis in every patient (mean 4.4; 95% CI 4.3 to 4.6). Patient discomfort from subcutaneous injections was perceived as the least important barrier (mean 3.8; 95% CI 3.6 to 4.0). Physicians perceived lack of awareness about the effectiveness of DVT prophylaxis as the most important barrier (mean 4.0; 95% CI 3.5 to 4.4), whereas concern about bleeding was dominant among non‐physicians (nurses' mean 5.2; 95% CI 5.0 to 5.5; pharmacists' mean 4.7; 95% CI 4.5 to 4.9; physiotherapists' mean 4.6; 95% CI 3.9 to 5.3).

Figure 3

Potential Success and Feasibility of Interventions to Optimize DVT Prophylaxis Utilization

Interventions considered across clinician groups as highly potentially successful were: preprinted order sheets (5.7; 95% CI 5.6 to 5.8); pharmacist reminders to physicians (mean 5.3; 95% CI 5.1 to 5.4); computerized reminders to physicians (mean 5.0; 95% CI 4.9 to 5.2); and use of a local opinion leader (mean 5.0; 95% CI 4.9 to 5.2). Interventions considered highly potentially feasible were: posters (mean 5.7; CI 5.6 to 5.8); preprinted order sheets (mean 5.5; 95% CI 5.4 to 5.7); laminated pocket cards (mean 5.4; 95% CI 5.2 to 5.5); multidisciplinary educational meetings (mean 5.0; 95% CI 4.9 to 5.2); and pharmacist reminders to physicians (mean 5.0; 95% CI 4.9 to 5.1). Preprinted orders and pharmacist reminders were perceived by all clinician groups as having both high potential success and feasibility (Figure 4).

Figure 4

Perceptions on Which Clinician Group Is Best Able to Assess and Implement DVT Prophylaxis

Respondents were divided between considering the attending physician and the bedside nurse as best able to perform a daily assessment of patients' need for DVT prophylaxis (43.4% [204/470] vs 44.0% [207/470], respectively). Respondents from these groups each predominantly thought this responsibility was theirs, with 68.1% (49/72) of physicians and 61.5% (123/200) of nurses perceiving this as their responsibility (Figure 5).

Figure 5

Forty‐one percent (193/471) of respondents perceived the attending physician as best able to ensure that DVT prophylaxis is ordered, while 31.2% (147/471) identified the pharmacist and 23.3% (110/471) identified the bedside nurse as best suited to this role. Among pharmacists, 66.3% (114/172) perceived that the attending pharmacist is best able to perform this task. Among respondents, 61.9% (296/478) felt the bedside nurse is best able to ensure adherence to DVT prophylaxis, with good agreement among all clinician groups.

DISCUSSION

Our survey identified several perceived barriers to optimizing DVT prophylaxis, consistent with those reported in the White Paper sponsored by the American Public Health Association.23 While no single barrier outlined in our survey was dominant, 2 novel barriers were identified: misperception of DVT prophylaxis underutilization, and confusion about roles and responsibilities in the area of DVT prophylaxis. Attention to these barriers may be helpful in developing an intervention aimed at bridging the gap between evidence and practice.

While our survey demonstrates agreement across clinician groups on the importance, efficacy, and safety of DVT prophylaxis, the discordant perceptions that exist about whether DVT prophylaxis is utilized appropriately is an important concern. Physician and pharmacist‐respondents demonstrated awareness that thromboprophylaxis is underutilized in medical patients. However, despite overwhelming published evidence to the contrary, nurses responding to our survey did not tend to recognize the problem of DVT prophylaxis underutilization in hospitalized medical patients. This knowledge deficit may be a significant barrier particularly since the pooled group of respondents indicated that nurses are among those caregivers best able to conduct a daily assessment of patients' need for DVT prophylaxis. A possible explanation for the finding that nurses and physiotherapists demonstrated a relative lack of awareness of the problem of DVT prophylaxis underutilization is ward‐specific healthcare priorities. Nursing and physiotherapy care on surgical wards is aimed at preventing postoperative complications, including DVT. However, its primary focus on medical wards is the management of acute medical problems. Prevention of hospital‐related complications, such as DVT, is often a secondary focus. Therefore, ensuring that all clinician groups are educated about the problem of DVT prophylaxis underutilization is necessary to drive quality improvement. A physician‐based survey on antithrombotic therapies demonstrated a similar need for education on guideline recommendations.20

A second important barrier identified in our survey is that both attending nurses and physicians feel that daily assessment of a patient's need for DVT prophylaxis is their responsibility. Confusion about roles and responsibilities in this area of patient care was reported by Cook et al., who identified that multidisciplinary care was perceived as a barrier to effective VTE prevention.18 Uncertainty as to which group should take ownership of DVT prophylaxis can lead to a diffusion of responsibility, a lack of accountability, and a gap in care. A resolution to whether DVT risk assessment is a nursing or a physician role could be reached through increased interdisciplinary communication and provision of clear definitions of roles to hospital staff.

Survey respondents felt that preprinted orders and pharmacist reminders to physicians were potentially successful and feasible strategies to optimize DVT prophylaxis. These components could be part of a simple tool to initiate prophylaxis. While electronic alerts have been shown to increase prophylaxis rates,24 we suspect that many respondents did not view these as highly important because of limited use of computerized order entry at their facilities. Interestingly, survey respondents did not perceive audit‐and‐feedback systems or local opinion leaders as potentially successful, though previous studies have demonstrated that they can change clinician behavior.25, 26 This may be because respondents may not be aware of the strength of technology‐based interventions (eg, electronic orders) and the role of opinion leaders, and the evidence in support of such interventions.24, 26 A systematic review of studies to improve DVT prophylaxis in hospitals reported that a combination of multiple active strategies is most effective, particularly those that link physician reminders with audit‐and‐feedback.27 For example, in the define study, a multicomponent intervention consisting of interactive educational sessions, verbal and computerized prompts, and individual performance feedback significantly improved adherence to DVT prophylaxis guidelines in critically ill patients.28 Whether a similar intervention could improve adherence to DVT prophylaxis guidelines in hospitalized medical patients merits further study. Any intervention must be paired with better education about which patients should, and should not, receive prophylaxis, as this may address many reported barriers in our survey (including concerns about bleeding). Respondents' uncertainty about these issues is not surprising, as studies of DVT prophylaxis in medical patients are not plentiful.2 However, recent guidelines do identify subgroups of medically ill patients in whom DVT prophylaxis is indicated.2 A clear and simple DVT risk assessment algorithm that identifies medical patients in whom DVT prophylaxis should (or should not) be administered may help to overcome respondents' concerns.

A limitation of our survey is the overall response rate of 36.3%, largely driven by the considerable number of nonresponding pharmacists (n = 760, reflecting 49% of the entire sample). However, the majority of the pharmacists were likely not hospital‐based, were thus not a target of this study, and their low response rate is not surprising. After excluding pharmacists, the response rate was 58.3% (321/551), which is consistent with response rates of other large‐sample surveys.29 The lower response rate for physicians and pharmacists may also reflect web‐based survey dissemination which, despite its feasibility, has lower response rates than paper‐based dissemination.3032 While the sample of physicians was relatively small compared to the other respondent groups surveyed, we aimed to identify barriers to actually implementing VTE prophylaxis, not just ordering prophylaxis, which is a multidisciplinary concern.

Although this survey was based on Canadian healthcare providers' perspectives, we believe the results are generalizable since both US and Canadian‐based studies have found that VTE prophylaxis is underutilized among hospitalized medical patients.4, 6 Furthermore, the American College of Chest Physicians (ACCP) guidelines on VTE prophylaxis, which are well‐recognized in both the United States and Canada, were developed with input from Canadian and American content experts.2 And while the US and Canadian healthcare systems are organized differently, at the patient‐care level, the roles of healthcare professionals are very similar. The generalizability of our findings is, however, limited by the institutional characteristics of respondents. We do not purport that the responses of any of the 4 clinician groups are generalizable to those groups as a whole. Although we surveyed clinicians in teaching and nonteaching, urban and rural practices, perceptions about DVT prophylaxis may be influenced by other factors, including the availability of local preprinted orders, electronic medical records, and quality improvement programs. Another potential limitation is that we did not assess all possible strategies to improve DVT prophylaxis, such as nurse practitioners and computerized decision support systems. These were purposely excluded, as they are not financially feasible in all centers, and thus not generalizable. Finally, like all self‐administered surveys, our findings reflect respondents' perceptions rather than objective observations about practice.

In conclusion, we identified novel and important barriers to optimal DVT prophylaxis utilization and potential interventions to address this important safety concern in hospitalized medical patients. To overcome some of these barriers, we propose an educational intervention prior to delivery of a top‐down, evidence‐based intervention to first increase healthcare providers' knowledge of the safety of DVT prophylaxis, system and team‐based approaches, and which interventions are most likely to be successful so as to encourage greater compliance with the intervention. A top‐down, system‐wide approach, involving the entire healthcare team and hospital administrators, can help drive this communication. As DVT prophylaxis becomes an increasingly important component in hospital accreditation, such solutions become appealing to facilitate change in practices. Results of this survey may inform future knowledge translation interventions by eliminating perceived barriers to DVT prophylaxis and by incorporating strategies that are perceived by healthcare professionals to be successful, feasible, and supported by evidence.

The population of patients in the US with limited English proficiency (LEP)those who speak English less than very well1is substantial and continues to grow.1, 2 Patients with LEP are at risk for lower quality health care overall than their English‐speaking counterparts.38 Professional in‐person interpreters greatly improve spoken communication and quality of care for these patients,4, 9 but their assistance is typically based on the clinical encounter. Particularly if interpreting by phone, interpreters are unlikely to be able to help with materials such as discharge instructions or information sheets meant for family members. Professional written translations of patient educational material help to bridge this gap, allowing clinicians to convey detailed written instructions to patients. However, professional translations must be prepared well in advance of any encounter and can only be used for easily anticipated problems.

The need to translate less common, patient‐specific instructions arises spontaneously in clinical practice, and formally prepared written translations are not useful in these situations. Online translation tools such as GoogleTranslate (available at http://translate.google.com/#) and Babelfish (available at http://babelfish.yahoo.com), a subset of machine translation technology, may help supplement professional in‐person interpretation and formal written translations in that they are ubiquitous, inexpensive, and increasingly well‐known and easy to use.10, 11 Machine translation has already been used in situations where in‐person interpretation is limited. For example, after the earthquake in Haiti, Creole interpreters were not widely available and a hand‐held translation application was quickly developed to meet the needs of relief workers and the population.11 However, data on the accuracy of these tools for critical clinical applications such as patient education are limited. A recent study of computer‐translated pharmacy labels suggested computer‐generated translations were frequently erratic, nonsensical, and even dangerous.12

We conducted a pilot evaluation of an online translation tool as it relates to detailed, complex patient educational material. Our primary goal was to compare the accuracy of a Spanish translation generated by the online tool to that done by a professional agency. Our secondary goals were: 1) to assess whether sentence word length or complexity mediated the accuracy of GT; and 2) to lay the foundation for a more comprehensive study of the accuracy of online translation tools, with respect to patient educational material.

Methods

Manual Evaluation: Evaluators, Domains, Scoring

We recruited three nonclinician, bilingual, nativeSpanish‐speaking research assistants as evaluators. The evaluators were all college educated with a Bachelor's degree or higher and were of Mexican, Nicaraguan, and Guatemalan ancestry. Each evaluator received a brief orientation regarding the project, as well as an explanation of the scores, and then proceeded to the blinded evaluation independently.

We asked evaluators to score sentences on Likert scales along five primary domains: fluency, adequacy, meaning, severity, and preference. Fluency and adequacy are well accepted components of machine translation evaluation,18 with fluency being an assessment of grammar and readability ranging from 5 (Perfect fluency; like reading a newspaper) to 1 (No fluency; no appreciable grammar, not understandable) and adequacy being an assessment of information preservation ranging from 5 (100% of information conveyed from the original) to 1 (0% of information conveyed from the original). Given that a sentence can be highly adequate but drastically change the connotation and intent of the sentence (eg, a sentence that contains 75% of the correct words but changes a sentence from take this medication twice a day to take this medication once every two days), we asked evaluators to assess meaning, a measure of connotation and intent maintenance, with scores ranging from 5 (Same meaning as original) to 1 (Totally different meaning from the original).19 Evaluators also assessed severity, a new measure of potential harm if a given sentence was assessed as having errors of any kind, ranging from 5 (Error, no effect on patient care) to 1 (Error, dangerous to patient) with an additional option of N/A (Sentence basically accurate). Finally, evaluators rated a blinded preference (also a new measure) for either of two translated sentences, ranging from Strongly prefer translation #1 to Strongly prefer translation #2. The order of the sentences was random (eg, sometimes the professional translation was first and sometimes the GT translation was). We subsequently converted this to preference for the professional translation, ranging from 5 (Strongly prefer the professional translation) to 1 (Strongly prefer the GT translation) in order to standardize the responses (Figures 1 and 2).

Figure 1

Figure 2

The overall flow of the study is given in Figure 3. Each evaluator initially scored 20 sentences translated by GT and 10 sentences translated professionally along the first four domains. All 30 of these sentences were randomly selected from the original, 263‐sentence pamphlet. For fluency, evaluators had access only to the translated sentence to be scored; for adequacy, meaning, and severity, they had access to both the translated sentence and the original English sentence. Ten of the 30 sentences were further selected randomly for scoring on the preference domain. For these 10 sentences, evaluators compared the GT and professional translations of the same sentence (with the original English sentence available as a reference) and indicated a preference, for any reason, for one translation or the other. Evaluators were blinded to the technique of translation (GT or professional) for all scored sentences and domains. We chose twice as many sentences from the GT preparations for the first four domains to maximize measurements for the translation technology we were evaluating, with the smaller number of professional translations serving as controls.

Figure 3

After scoring the first 30 sentences, evaluators met with one of the authors (R.R.K.) to discuss and consolidate their approach to scoring. They then scored an additional 10 GT‐translated sentences and 5 professionally translated sentences for the first four domains, and 9 of these 15 sentences for preference, to see if the meeting changed their scoring approach. These sentences were selected randomly from the original, 263‐sentence pamphlet, excluding the 30 evaluated in the previous step.

Results

Discussion

In this preliminary study comparing the accuracy of GT to professional translation for patient educational material, we found that GT was inferior to the professional translation in grammatical fluency but generally preserved the content and sense of the original text. Out of 30 GT sentences assessed, there was one substantially erroneous translation that was considered potentially dangerous. Evaluators preferred the professionally translated sentences for complex sentences, but when the English source sentence was simplecontaining a single clausethis preference disappeared.

Like Sharif and Tse,12 we found that for information not arranged in sentences, automated translation sometimes produced nonsensical sentences. In our study, these resulted from an English sentence fragment followed by a bulleted list; in their study, the nonsensical translations resulted from pharmacy labels. The difference in frequency of these errors between our studies may have resulted partly from the translation tool evaluated (GT vs programs used by pharmacies in the Bronx), but may have also been due to our use of machine translation for complete sentencesthe purpose for which it is optimally designed. The hypothesis that machine translations of clinical information are most understandable when used for simple, complete sentences concurs with the methodology used by these tools and requires further study.

GT has the potential to be very useful to clinicians, particularly for those instances when the communication required is both spontaneous and routine or noncritical. For example, in the inpatient setting, patients could communicate diet and other nonclinical requests, as well as ask or answer simple, short questions when the interpreter is not available. In such situations, the low cost and ease of using online translations and machine translation more generally may help to circumvent the tendency of clinicians to get by with inadequate language skills or to avoid communication altogether.25 If used wisely, GT and other online tools could supplement the use of standardized translations and professional interpreters in helping clinicians to overcome language barriers and linguistic inertia, though this will require further assessment.

Ours is a pilot study, and while it suggests a more promising way to use online translation tools, significant further evaluation is required regarding accuracy and applicability prior to widespread use of any machine translation tools for patient care. The document we utilized for evaluation was a professionally translated patient educational brochure provided to individuals starting a complex medication. As online translation tools would most likely not be used in this setting, but rather for spontaneous and less critical patient‐specific instructions, further testing of GT as applied to such scenarios should be considered. Second, we only evaluated GT for English translated into Spanish; its usefulness in other languages will need to be evaluated. It also remains to be seen how easily GT translations will be understood by patients, who may have variable medical understanding and educational attainment as compared to our evaluators. Finally, in this evaluation, we only assessed automated written translation, not automated spoken translation services such as those now available on cellular phones and other mobile devices.11 The latter are based upon translation software with an additional speech recognition interface. These applications may prove to be even more useful than online translation, but the speech recognition component will add an additional layer of potential error and these applications will need to be evaluated on their own merits.

The domains chosen for this study had only moderate interrater reliability as assessed by intraclass correlation and repeatability, with meaning and preference scoring particularly poorly. The latter domains in particular will require more thorough assessment before routine use in online translation assessment. The variability in all domains may have resulted partly from the choice of nonclinicians of different ancestral backgrounds as evaluators. However, this variability is likely better representative of the wide range of patient backgrounds. Because our evaluators were not professional translators, we asked a professional interpreter to grade all sentences to assess the quality of their evaluation. While the interpreter noted slightly fewer errors among the professionally translated sentences (13% vs 22%) and slightly more errors among the GT‐translated sentences (50% vs 39%), and preferred the professional translation slightly more (3.8 vs 3.2), his scores for all of the other measures were almost identical, increasing our confidence in our primary findings (Appendix A). Additionally, since statistical translation is conducted sentence by sentence, in our study evaluators only scored translations at the sentence level. The accuracy of GT for whole paragraphs or entire documents will need to be assessed separately. The correlation between METEOR and the manual evaluation scores was less than in prior studies; while inexpensive to assess, METEOR will have to be recalibrated in optimal circumstanceswith several reference translations available rather than just onebefore it can be used to supplement the assessment of new languages, new materials, other translation technologies, and improvements in a given technology over time for patient educational material.

In summary, GT scored worse in grammar but similarly in content and sense to the professional translation, committing one critical error in translating a complex, fragmented sentence as nonsense. We believe that, with further study and judicious use, GT has the potential to substantially improve clinicians' communication with patients with limited English proficiency in the area of brief spontaneous patient‐specific information, supplementing well the role that professional spoken interpretation and standardized written translations already play.

The population of patients in the US with limited English proficiency (LEP)those who speak English less than very well1is substantial and continues to grow.1, 2 Patients with LEP are at risk for lower quality health care overall than their English‐speaking counterparts.38 Professional in‐person interpreters greatly improve spoken communication and quality of care for these patients,4, 9 but their assistance is typically based on the clinical encounter. Particularly if interpreting by phone, interpreters are unlikely to be able to help with materials such as discharge instructions or information sheets meant for family members. Professional written translations of patient educational material help to bridge this gap, allowing clinicians to convey detailed written instructions to patients. However, professional translations must be prepared well in advance of any encounter and can only be used for easily anticipated problems.

The need to translate less common, patient‐specific instructions arises spontaneously in clinical practice, and formally prepared written translations are not useful in these situations. Online translation tools such as GoogleTranslate (available at http://translate.google.com/#) and Babelfish (available at http://babelfish.yahoo.com), a subset of machine translation technology, may help supplement professional in‐person interpretation and formal written translations in that they are ubiquitous, inexpensive, and increasingly well‐known and easy to use.10, 11 Machine translation has already been used in situations where in‐person interpretation is limited. For example, after the earthquake in Haiti, Creole interpreters were not widely available and a hand‐held translation application was quickly developed to meet the needs of relief workers and the population.11 However, data on the accuracy of these tools for critical clinical applications such as patient education are limited. A recent study of computer‐translated pharmacy labels suggested computer‐generated translations were frequently erratic, nonsensical, and even dangerous.12

We conducted a pilot evaluation of an online translation tool as it relates to detailed, complex patient educational material. Our primary goal was to compare the accuracy of a Spanish translation generated by the online tool to that done by a professional agency. Our secondary goals were: 1) to assess whether sentence word length or complexity mediated the accuracy of GT; and 2) to lay the foundation for a more comprehensive study of the accuracy of online translation tools, with respect to patient educational material.

Methods

Manual Evaluation: Evaluators, Domains, Scoring

We recruited three nonclinician, bilingual, nativeSpanish‐speaking research assistants as evaluators. The evaluators were all college educated with a Bachelor's degree or higher and were of Mexican, Nicaraguan, and Guatemalan ancestry. Each evaluator received a brief orientation regarding the project, as well as an explanation of the scores, and then proceeded to the blinded evaluation independently.

We asked evaluators to score sentences on Likert scales along five primary domains: fluency, adequacy, meaning, severity, and preference. Fluency and adequacy are well accepted components of machine translation evaluation,18 with fluency being an assessment of grammar and readability ranging from 5 (Perfect fluency; like reading a newspaper) to 1 (No fluency; no appreciable grammar, not understandable) and adequacy being an assessment of information preservation ranging from 5 (100% of information conveyed from the original) to 1 (0% of information conveyed from the original). Given that a sentence can be highly adequate but drastically change the connotation and intent of the sentence (eg, a sentence that contains 75% of the correct words but changes a sentence from take this medication twice a day to take this medication once every two days), we asked evaluators to assess meaning, a measure of connotation and intent maintenance, with scores ranging from 5 (Same meaning as original) to 1 (Totally different meaning from the original).19 Evaluators also assessed severity, a new measure of potential harm if a given sentence was assessed as having errors of any kind, ranging from 5 (Error, no effect on patient care) to 1 (Error, dangerous to patient) with an additional option of N/A (Sentence basically accurate). Finally, evaluators rated a blinded preference (also a new measure) for either of two translated sentences, ranging from Strongly prefer translation #1 to Strongly prefer translation #2. The order of the sentences was random (eg, sometimes the professional translation was first and sometimes the GT translation was). We subsequently converted this to preference for the professional translation, ranging from 5 (Strongly prefer the professional translation) to 1 (Strongly prefer the GT translation) in order to standardize the responses (Figures 1 and 2).

Figure 1

Figure 2

The overall flow of the study is given in Figure 3. Each evaluator initially scored 20 sentences translated by GT and 10 sentences translated professionally along the first four domains. All 30 of these sentences were randomly selected from the original, 263‐sentence pamphlet. For fluency, evaluators had access only to the translated sentence to be scored; for adequacy, meaning, and severity, they had access to both the translated sentence and the original English sentence. Ten of the 30 sentences were further selected randomly for scoring on the preference domain. For these 10 sentences, evaluators compared the GT and professional translations of the same sentence (with the original English sentence available as a reference) and indicated a preference, for any reason, for one translation or the other. Evaluators were blinded to the technique of translation (GT or professional) for all scored sentences and domains. We chose twice as many sentences from the GT preparations for the first four domains to maximize measurements for the translation technology we were evaluating, with the smaller number of professional translations serving as controls.

Figure 3

After scoring the first 30 sentences, evaluators met with one of the authors (R.R.K.) to discuss and consolidate their approach to scoring. They then scored an additional 10 GT‐translated sentences and 5 professionally translated sentences for the first four domains, and 9 of these 15 sentences for preference, to see if the meeting changed their scoring approach. These sentences were selected randomly from the original, 263‐sentence pamphlet, excluding the 30 evaluated in the previous step.

Results

Discussion

In this preliminary study comparing the accuracy of GT to professional translation for patient educational material, we found that GT was inferior to the professional translation in grammatical fluency but generally preserved the content and sense of the original text. Out of 30 GT sentences assessed, there was one substantially erroneous translation that was considered potentially dangerous. Evaluators preferred the professionally translated sentences for complex sentences, but when the English source sentence was simplecontaining a single clausethis preference disappeared.

Like Sharif and Tse,12 we found that for information not arranged in sentences, automated translation sometimes produced nonsensical sentences. In our study, these resulted from an English sentence fragment followed by a bulleted list; in their study, the nonsensical translations resulted from pharmacy labels. The difference in frequency of these errors between our studies may have resulted partly from the translation tool evaluated (GT vs programs used by pharmacies in the Bronx), but may have also been due to our use of machine translation for complete sentencesthe purpose for which it is optimally designed. The hypothesis that machine translations of clinical information are most understandable when used for simple, complete sentences concurs with the methodology used by these tools and requires further study.

GT has the potential to be very useful to clinicians, particularly for those instances when the communication required is both spontaneous and routine or noncritical. For example, in the inpatient setting, patients could communicate diet and other nonclinical requests, as well as ask or answer simple, short questions when the interpreter is not available. In such situations, the low cost and ease of using online translations and machine translation more generally may help to circumvent the tendency of clinicians to get by with inadequate language skills or to avoid communication altogether.25 If used wisely, GT and other online tools could supplement the use of standardized translations and professional interpreters in helping clinicians to overcome language barriers and linguistic inertia, though this will require further assessment.

Ours is a pilot study, and while it suggests a more promising way to use online translation tools, significant further evaluation is required regarding accuracy and applicability prior to widespread use of any machine translation tools for patient care. The document we utilized for evaluation was a professionally translated patient educational brochure provided to individuals starting a complex medication. As online translation tools would most likely not be used in this setting, but rather for spontaneous and less critical patient‐specific instructions, further testing of GT as applied to such scenarios should be considered. Second, we only evaluated GT for English translated into Spanish; its usefulness in other languages will need to be evaluated. It also remains to be seen how easily GT translations will be understood by patients, who may have variable medical understanding and educational attainment as compared to our evaluators. Finally, in this evaluation, we only assessed automated written translation, not automated spoken translation services such as those now available on cellular phones and other mobile devices.11 The latter are based upon translation software with an additional speech recognition interface. These applications may prove to be even more useful than online translation, but the speech recognition component will add an additional layer of potential error and these applications will need to be evaluated on their own merits.

The domains chosen for this study had only moderate interrater reliability as assessed by intraclass correlation and repeatability, with meaning and preference scoring particularly poorly. The latter domains in particular will require more thorough assessment before routine use in online translation assessment. The variability in all domains may have resulted partly from the choice of nonclinicians of different ancestral backgrounds as evaluators. However, this variability is likely better representative of the wide range of patient backgrounds. Because our evaluators were not professional translators, we asked a professional interpreter to grade all sentences to assess the quality of their evaluation. While the interpreter noted slightly fewer errors among the professionally translated sentences (13% vs 22%) and slightly more errors among the GT‐translated sentences (50% vs 39%), and preferred the professional translation slightly more (3.8 vs 3.2), his scores for all of the other measures were almost identical, increasing our confidence in our primary findings (Appendix A). Additionally, since statistical translation is conducted sentence by sentence, in our study evaluators only scored translations at the sentence level. The accuracy of GT for whole paragraphs or entire documents will need to be assessed separately. The correlation between METEOR and the manual evaluation scores was less than in prior studies; while inexpensive to assess, METEOR will have to be recalibrated in optimal circumstanceswith several reference translations available rather than just onebefore it can be used to supplement the assessment of new languages, new materials, other translation technologies, and improvements in a given technology over time for patient educational material.

In summary, GT scored worse in grammar but similarly in content and sense to the professional translation, committing one critical error in translating a complex, fragmented sentence as nonsense. We believe that, with further study and judicious use, GT has the potential to substantially improve clinicians' communication with patients with limited English proficiency in the area of brief spontaneous patient‐specific information, supplementing well the role that professional spoken interpretation and standardized written translations already play.

The population of patients in the US with limited English proficiency (LEP)those who speak English less than very well1is substantial and continues to grow.1, 2 Patients with LEP are at risk for lower quality health care overall than their English‐speaking counterparts.38 Professional in‐person interpreters greatly improve spoken communication and quality of care for these patients,4, 9 but their assistance is typically based on the clinical encounter. Particularly if interpreting by phone, interpreters are unlikely to be able to help with materials such as discharge instructions or information sheets meant for family members. Professional written translations of patient educational material help to bridge this gap, allowing clinicians to convey detailed written instructions to patients. However, professional translations must be prepared well in advance of any encounter and can only be used for easily anticipated problems.

The need to translate less common, patient‐specific instructions arises spontaneously in clinical practice, and formally prepared written translations are not useful in these situations. Online translation tools such as GoogleTranslate (available at http://translate.google.com/#) and Babelfish (available at http://babelfish.yahoo.com), a subset of machine translation technology, may help supplement professional in‐person interpretation and formal written translations in that they are ubiquitous, inexpensive, and increasingly well‐known and easy to use.10, 11 Machine translation has already been used in situations where in‐person interpretation is limited. For example, after the earthquake in Haiti, Creole interpreters were not widely available and a hand‐held translation application was quickly developed to meet the needs of relief workers and the population.11 However, data on the accuracy of these tools for critical clinical applications such as patient education are limited. A recent study of computer‐translated pharmacy labels suggested computer‐generated translations were frequently erratic, nonsensical, and even dangerous.12

We conducted a pilot evaluation of an online translation tool as it relates to detailed, complex patient educational material. Our primary goal was to compare the accuracy of a Spanish translation generated by the online tool to that done by a professional agency. Our secondary goals were: 1) to assess whether sentence word length or complexity mediated the accuracy of GT; and 2) to lay the foundation for a more comprehensive study of the accuracy of online translation tools, with respect to patient educational material.

Methods

Manual Evaluation: Evaluators, Domains, Scoring

We recruited three nonclinician, bilingual, nativeSpanish‐speaking research assistants as evaluators. The evaluators were all college educated with a Bachelor's degree or higher and were of Mexican, Nicaraguan, and Guatemalan ancestry. Each evaluator received a brief orientation regarding the project, as well as an explanation of the scores, and then proceeded to the blinded evaluation independently.

We asked evaluators to score sentences on Likert scales along five primary domains: fluency, adequacy, meaning, severity, and preference. Fluency and adequacy are well accepted components of machine translation evaluation,18 with fluency being an assessment of grammar and readability ranging from 5 (Perfect fluency; like reading a newspaper) to 1 (No fluency; no appreciable grammar, not understandable) and adequacy being an assessment of information preservation ranging from 5 (100% of information conveyed from the original) to 1 (0% of information conveyed from the original). Given that a sentence can be highly adequate but drastically change the connotation and intent of the sentence (eg, a sentence that contains 75% of the correct words but changes a sentence from take this medication twice a day to take this medication once every two days), we asked evaluators to assess meaning, a measure of connotation and intent maintenance, with scores ranging from 5 (Same meaning as original) to 1 (Totally different meaning from the original).19 Evaluators also assessed severity, a new measure of potential harm if a given sentence was assessed as having errors of any kind, ranging from 5 (Error, no effect on patient care) to 1 (Error, dangerous to patient) with an additional option of N/A (Sentence basically accurate). Finally, evaluators rated a blinded preference (also a new measure) for either of two translated sentences, ranging from Strongly prefer translation #1 to Strongly prefer translation #2. The order of the sentences was random (eg, sometimes the professional translation was first and sometimes the GT translation was). We subsequently converted this to preference for the professional translation, ranging from 5 (Strongly prefer the professional translation) to 1 (Strongly prefer the GT translation) in order to standardize the responses (Figures 1 and 2).

Figure 1

Figure 2

The overall flow of the study is given in Figure 3. Each evaluator initially scored 20 sentences translated by GT and 10 sentences translated professionally along the first four domains. All 30 of these sentences were randomly selected from the original, 263‐sentence pamphlet. For fluency, evaluators had access only to the translated sentence to be scored; for adequacy, meaning, and severity, they had access to both the translated sentence and the original English sentence. Ten of the 30 sentences were further selected randomly for scoring on the preference domain. For these 10 sentences, evaluators compared the GT and professional translations of the same sentence (with the original English sentence available as a reference) and indicated a preference, for any reason, for one translation or the other. Evaluators were blinded to the technique of translation (GT or professional) for all scored sentences and domains. We chose twice as many sentences from the GT preparations for the first four domains to maximize measurements for the translation technology we were evaluating, with the smaller number of professional translations serving as controls.

Figure 3

After scoring the first 30 sentences, evaluators met with one of the authors (R.R.K.) to discuss and consolidate their approach to scoring. They then scored an additional 10 GT‐translated sentences and 5 professionally translated sentences for the first four domains, and 9 of these 15 sentences for preference, to see if the meeting changed their scoring approach. These sentences were selected randomly from the original, 263‐sentence pamphlet, excluding the 30 evaluated in the previous step.

Results

Discussion

In this preliminary study comparing the accuracy of GT to professional translation for patient educational material, we found that GT was inferior to the professional translation in grammatical fluency but generally preserved the content and sense of the original text. Out of 30 GT sentences assessed, there was one substantially erroneous translation that was considered potentially dangerous. Evaluators preferred the professionally translated sentences for complex sentences, but when the English source sentence was simplecontaining a single clausethis preference disappeared.

Like Sharif and Tse,12 we found that for information not arranged in sentences, automated translation sometimes produced nonsensical sentences. In our study, these resulted from an English sentence fragment followed by a bulleted list; in their study, the nonsensical translations resulted from pharmacy labels. The difference in frequency of these errors between our studies may have resulted partly from the translation tool evaluated (GT vs programs used by pharmacies in the Bronx), but may have also been due to our use of machine translation for complete sentencesthe purpose for which it is optimally designed. The hypothesis that machine translations of clinical information are most understandable when used for simple, complete sentences concurs with the methodology used by these tools and requires further study.

GT has the potential to be very useful to clinicians, particularly for those instances when the communication required is both spontaneous and routine or noncritical. For example, in the inpatient setting, patients could communicate diet and other nonclinical requests, as well as ask or answer simple, short questions when the interpreter is not available. In such situations, the low cost and ease of using online translations and machine translation more generally may help to circumvent the tendency of clinicians to get by with inadequate language skills or to avoid communication altogether.25 If used wisely, GT and other online tools could supplement the use of standardized translations and professional interpreters in helping clinicians to overcome language barriers and linguistic inertia, though this will require further assessment.

Ours is a pilot study, and while it suggests a more promising way to use online translation tools, significant further evaluation is required regarding accuracy and applicability prior to widespread use of any machine translation tools for patient care. The document we utilized for evaluation was a professionally translated patient educational brochure provided to individuals starting a complex medication. As online translation tools would most likely not be used in this setting, but rather for spontaneous and less critical patient‐specific instructions, further testing of GT as applied to such scenarios should be considered. Second, we only evaluated GT for English translated into Spanish; its usefulness in other languages will need to be evaluated. It also remains to be seen how easily GT translations will be understood by patients, who may have variable medical understanding and educational attainment as compared to our evaluators. Finally, in this evaluation, we only assessed automated written translation, not automated spoken translation services such as those now available on cellular phones and other mobile devices.11 The latter are based upon translation software with an additional speech recognition interface. These applications may prove to be even more useful than online translation, but the speech recognition component will add an additional layer of potential error and these applications will need to be evaluated on their own merits.

The domains chosen for this study had only moderate interrater reliability as assessed by intraclass correlation and repeatability, with meaning and preference scoring particularly poorly. The latter domains in particular will require more thorough assessment before routine use in online translation assessment. The variability in all domains may have resulted partly from the choice of nonclinicians of different ancestral backgrounds as evaluators. However, this variability is likely better representative of the wide range of patient backgrounds. Because our evaluators were not professional translators, we asked a professional interpreter to grade all sentences to assess the quality of their evaluation. While the interpreter noted slightly fewer errors among the professionally translated sentences (13% vs 22%) and slightly more errors among the GT‐translated sentences (50% vs 39%), and preferred the professional translation slightly more (3.8 vs 3.2), his scores for all of the other measures were almost identical, increasing our confidence in our primary findings (Appendix A). Additionally, since statistical translation is conducted sentence by sentence, in our study evaluators only scored translations at the sentence level. The accuracy of GT for whole paragraphs or entire documents will need to be assessed separately. The correlation between METEOR and the manual evaluation scores was less than in prior studies; while inexpensive to assess, METEOR will have to be recalibrated in optimal circumstanceswith several reference translations available rather than just onebefore it can be used to supplement the assessment of new languages, new materials, other translation technologies, and improvements in a given technology over time for patient educational material.

In summary, GT scored worse in grammar but similarly in content and sense to the professional translation, committing one critical error in translating a complex, fragmented sentence as nonsense. We believe that, with further study and judicious use, GT has the potential to substantially improve clinicians' communication with patients with limited English proficiency in the area of brief spontaneous patient‐specific information, supplementing well the role that professional spoken interpretation and standardized written translations already play.

Pneumonia occurs more commonly among older persons.1 With advancing age, the frequency of hospitalizations and mortality for pneumonia are higher.2 Among the tools developed to predict short‐term mortality is the pneumonia severity index (PSI), which is the best known among severity of illness indices for pneumonia.3 Its ability to predict short‐term mortality for CAP, particularly in identifying those at low risk was previously demonstrated.4 More recently, the extension of its utility in predicting 30‐day mortality for healthcare‐associated pneumonia (HCAP) was demonstrated.5

Severity of illness is one of several risk factors for adverse outcomes among older persons with acute illness. Besides comorbidity, other factors include functional impairment and atypical presentation. Information on physical functioning had equal importance as laboratory data in prognostication of in‐hospital mortality.6 In addition, walking impairment was 1 of 5 components of a risk adjustment index developed to predict 1‐year mortality for hospitalized older persons.7 Atypical presentations of illness, such as delirium and falls, independently predicted poor outcomes among hospitalized older patients.8

Specifically for pneumonia, functional status has also been shown to be an independent predictor of short‐term mortality among older patients hospitalized with CAP.913 Among atypical presentations, only absence of chills was an independent prognostic factor for CAP.9 Bacteremia was an independent factor related to death among adults with CAP, albeit for severe disease resulting in intensive care unit admission.14 It was also included in a severity assessment score; its higher scores were associated with early mortality.15 However, blood culture results are only available 2 to 3 days into the hospital episode. Therefore, bacteremia is a potential risk factor for mortality that is not identifiable at the start of hospitalization.

While PSI is a comprehensive collection of demographic, clinical, and investigative measures, it does not include items on functional status or atypical presentation. Neither does it account for recent hospitalization or comorbid conditions of significance to older persons, such as dementia and depression. It is plausible that at least some of these factors hold added prognostic value.

With all these in mind, we conducted a study with the following objectives: 1) to determine whether functional impairment, recent hospitalization, comorbid conditions of particular significance with advancing age, and atypical presentation are significantly associated with short‐term mortality among older patients hospitalized for CAP and HCAP, after taking into account PSI; and 2) if so, to estimate the magnitude of increased mortality risk with these factors. We tested our null hypotheses that, after adjustment for PSI class, 1) recent hospitalization, 2) pre‐morbid functional impairment, 3) dementia and depression, and 4) atypical presentation of illness have no association with 30‐day mortality for older persons hospitalized for CAP and HCAP, both combined and alone.

PATIENTS AND METHODS

Patient Population

We included first hospital episodes of adults aged 65 years or older with the principal diagnosis of pneumonia in 2007. These episodes were identified by their primary International Classification of Diseases, 9th revision, Clinical Modification (ICD‐9‐CM) codes of 480 to 486 in the administrative data. Next, we applied our study definition of pneumonia, which required the presence of acute symptoms or signs of pneumonia at the point of hospital admission, and a chest radiograph with features consistent with pneumonia that was obtained during the period from 24 hours before, to 48 hours after, hospital admission. In doing so, we included patients with community‐acquired pneumonia (CAP)16 and healthcare‐associated pneumonia (HCAP),17 but not hospital‐acquired pneumonia (HAP). We excluded patients whose charts were not accessible for review because of human immunodeficiency virus/acquired immune deficiency syndrome (HIV/AIDS) and those whose charts were unavailable for other reasons. The study flow diagram is shown in Figure 1.

Figure 1

We assigned the diagnosis of HCAP to patients who were admitted to an acute care hospital for 2 or more days in the prior 90 days, resided in a nursing home or long‐term care facility, or received of intravenous antibiotic therapy, chemotherapy, wound care, or hemodialysis in the prior 30 days.18 Remaining patients were assigned CAP.

RESULTS

Among 1607 patients included, 890 (55.4%) had CAP and 717 (44.6%) had HCAP. Baseline patient characteristics of patients with CAP and HCAP are shown in Table 1. The 30‐day mortality rate was 28.1% for the entire group, and 20.6% and 37.4% for patients with CAP and HCAP, respectively. When stratified according to PSI classes 2, 3, 4, and 5, this rate was 0%, 8.2%, 24.4%, and 56.0%, respectively. Because there were no deaths among those with PSI class 2, this category was merged with class 3 for the regression analyses. Missing data on pre‐morbid ambulation impairment and feeding impairment occurred for 39 (2.4%) and 69 (4.6%) patients, respectively.

For CAP and HCAP together, pre‐morbid ambulation impairment was associated with increased 30‐day mortality (339/798 [42.5%] vs 112/809 [13.8%], unadjusted OR 4.60, 95% CI 3.60 to 5.87, P < 0.01), as was hospitalization in the prior 30 days (94/209 [45.0%] vs 357/1398 [25.5%], unadjusted OR 2.38, 95% CI 1.77 to 3.21, P = 0.02). This was also the case for dementia (118/307 [38.4%] vs 333/1300 [25.6%], unadjusted OR 1.81, 95% CI 1.40 to 2.35, P < 0.01), acute geriatric syndromes (163/442 [36.9%] vs 288/1165 [24.7%], unadjusted OR 1.78, 95% CI 1.41 to 2.25, P < 0.01), and absence of cough and purulent sputum (226/559 [40.4%] vs 225/1048 [21.5%], unadjusted OR 2.48, 95% CI 1.98 to 3.11, P < 0.01). However, depression was not significantly associated with 30‐day mortality (57/165 [34.6%] vs 394/1442 [27.3%], unadjusted OR 1.40, 95% CI 1.00 to 1.97, P = 0.05).

Table 2 summarizes the results of logistic regression. It shows that pre‐morbid ambulation impairment, hospitalization in the prior 30 days, and absence of cough and purulent sputum were all independently associated with 30‐day mortality after adjustment for PSI score for the entire group. These associations remained statistically significant when CAP and HCAP were examined separately. Because none of those with CAP could have hospitalization in the prior 30 days, this factor was not included in the CAP model. The strength of association for the same patient factor varied across the pneumonia sub‐type. This was markedly so for pre‐morbid ambulation impairment, with the OR estimate being almost 3‐fold higher for CAP than for HCAP. Dementia, depression, and acute geriatric syndromes were not associated with 30‐day mortality. When the analyses were repeated after excluding hospital episodes with missing values for pre‐morbid ambulation impairment, the same 3 variables were significantly associated with 30‐day mortality, with trivial differences in strength of association compared to when imputation was performed. The OR estimates for pre‐morbid ambulation impairment, hospitalization in the prior 30 days, and absence of cough and purulent sputum were 2.82 (95% CI 2.12 to 3.76), 1.83 (95% CI 1.42 to 2.83), and 1.47 (95% CI 1.14 to 1.91).

Two‐level hierarchical modeling to account for clustering at the hospital level obtained negligible change in OR estimates of the patient factors and their 95% CI. There were no statistically significant interactions between PSI class and the 3 patient factors (results not shown).

The model‐predicted increase in mortality risk with presence of individual patient factors for the entire group is shown in Table 3. Across the 3 factors, 30‐day mortality increased by 1.9% to 6.1% for those with PSI class 2 and 3, and by 9.0% to 23.2% for those with PSI class 5. The upper end of these ranges represented the effect of pre‐morbid ambulation impairment, while the lower end was that for absence of cough and purulent sputum. With reference to the predicted mortality rates for PSI class which are listed in the footnotes of Table 3, the adverse prognosis conferred by individual patient factors amounted to relative risk inflation of 27% to 145% depending on the specific factor and PSI class.

DISCUSSION

After accounting for PSI class, we found 3 additional patient factors that were independently associated with 30‐day mortality among older persons hospitalized for pneumonia. Firstly, our study confirms that impaired physical function reflected by pre‐morbid ambulation impairment increases mortality risk, as previously demonstrated by Torres et al.10 It is likely that impaired function reflects an underlying vulnerability for adverse outcomes that is seen across primary diagnoses.7 Secondly, recent hospitalization often indicates clinical, functional, and social complexities, as well as increased likelihood of infection by more virulent organisms commonly associated with healthcare‐related infections. Together, these 2 factors could increase mortality risk. Thirdly, atypical presentations may be associated with increased mortality, because these often occur in frail older persons who are vulnerable to adverse outcomes8 due to diseases suffered and treatment received. Atypical presentations may also result in delayed diagnosis and treatment of pneumonia.

Pilotto et al. found that a multidimensional index comprising functional status, comorbidity burden, mental status, and nutritional assessment, among others, had a higher predictive accuracy for 30‐day mortality than did PSI.20 While there was a previous attempt to combine PSI with independent predictors to identify low‐risk older patients with CAP,21 we could not find similar work on the range of patient factors examined in this study. Indeed, the most important contribution that our study brings to the growing body of literature on short‐term mortality, among older persons hospitalized for pneumonia, is the prognostic importance of these 3 additional patient factors over and above severity of illness measured by PSI. With reference to the baseline predicted risk for different PSI class categories shown in Table 3, we have demonstrated that the predicted increase in mortality risk with the presence of these 3 factors is often not trivial, particularly for those with more severe pneumonia.

These 3 patient factors retained prognostic significance after accounting for PSI class for HCAP. However, only 2 factors were associated with mortality for CAP, because by definition recent hospitalization does not occur. A relevant discussion point is whether CAP and HCAP should be grouped together or classified separately. It is pertinent to reflect that the utility of making a distinction between CAP and HCAP appears to lie largely in the domain of therapeutics regarding the initial choice of antibiotics,18, 2225 although there has been some debate on this point.26 Moreover, the major features of HCAP, namely recent hospitalization (albeit in the prior 30 days, rather than 90 days) and nursing home residence (an item in PSI) were included in our regression analyses. Therefore, it seems reasonable to consider CAP and HCAP as a single group for risk stratification at the clinical frontline. We also argue that combining CAP and HCAP for risk adjustment will result in larger sample sizes that can minimize uncertainty around treatment effect estimates, when comparing across different interventions or providers. The same approach of analyzing CAP and HCAP together was adopted in a recent study that compared US hospitals on their risk‐adjusted performance for pneumonia among Medicare beneficiaries.27

The 30‐day mortality rates in this study are higher than those in the original PSI studies, even when stratified according to PSI class. However, more recent studies also registered relatively high mortality rates ranging from 18% to 19%.12, 28 There are a number of possible reasons for the higher mortality rates observed in our study. Firstly, we included both CAP and HCAP, whereas some other studies focused only on CAP. Secondly, the original PSI studies excluded patients with previous hospitalization within 7 days of admission, while we included them. Thirdly, our study population was relatively old (median age: 80 years) and had a higher proportion from nursing homes (22%). Although age and nursing home residence are variables in the PSI, the weights assigned to these 2 items may not adequately reflect the magnitude of mortality risk they confer. Finally, our understanding is that the study population comprises a relatively high proportion of patients who have do‐not‐resuscitate (DNR) instructions, though this was not measured. All these patient characteristics are likely to be associated with higher mortality risk.

The major strength of this study relates to its real world setting, where there were no major exclusion criteria except for HIV/AIDS. In addition, the clinical data at our disposal allowed selection from a relatively wide range of patient factors, beyond that commonly available in administrative data alone.

However, a few important limitations need to be acknowledged. Firstly, the retrospective nature of the study restricted data to those routinely collected, rather than that specifically acquired for research. Important unmeasured factors include inflammatory markers such as C‐reactive protein (CRP) or procalcitonin levels which have been shown to have prognostic value.29 Others include frailty, socioeconomic status, and social support.20 Secondly, increased likelihood of measurement error associated with retrospectively collected data could result in bias with uncertain direction. Thirdly, our strategy of assuming no functional impairment in the absence of documentation raises the possibility of underidentification and consequent bias in the direction of underestimation of the strength of association between pre‐morbid ambulation impairment and mortality. If so, the true association could even be stronger. Finally, we did not capture do‐not‐resuscitate (DNR) decisions because these were not consistently documented in the charts. We concede that DNR status is expected to be associated with short‐term mortality30 and therefore remains an unobserved factor that may explain a proportion of the mortality risk attributed to other factors in our study, such as pre‐morbid ambulation impairment.

Where do we proceed from here? Given our findings, further work that examines the unmeasured factors mentioned should be done. CRP and procalcitonin levels can be extracted from the laboratory results database when they are measured. However, specification of the other 3 factors is more challenging, given that these represent clinical or social constructs wherein optimal measurement is less certain. It would be important to estimate how much these factors improve the prediction of short‐term mortality beyond that achieved by PSI and the patient factors we have identified.

Nonetheless, the clinical implications of our work are clear. While PSI class is a time‐tested tool, addition of pre‐morbid ambulation impairment, hospitalization in the prior 30 days, and absence of cough and purulent sputum can further improve risk stratification for short‐term mortality, when older persons present initially with clinical and radiological features of pneumonia. Information on these factors should be available in routine clinical care and, therefore, their use in risk stratification should be considered. For more valid and credible risk adjustment, these 3 factors could be considered in addition to severity of illness indices where data availability permits.

CONCLUSION

Recent hospitalization, pre‐morbid ambulation impairment, and atypical clinical presentation were independently associated with higher 30‐day mortality among older persons hospitalized for pneumonia, after adjusting for severity of illness with PSI class. These factors could be considered in addition to PSI, when performing risk stratification and adjustment in this setting.

Pneumonia occurs more commonly among older persons.1 With advancing age, the frequency of hospitalizations and mortality for pneumonia are higher.2 Among the tools developed to predict short‐term mortality is the pneumonia severity index (PSI), which is the best known among severity of illness indices for pneumonia.3 Its ability to predict short‐term mortality for CAP, particularly in identifying those at low risk was previously demonstrated.4 More recently, the extension of its utility in predicting 30‐day mortality for healthcare‐associated pneumonia (HCAP) was demonstrated.5

Severity of illness is one of several risk factors for adverse outcomes among older persons with acute illness. Besides comorbidity, other factors include functional impairment and atypical presentation. Information on physical functioning had equal importance as laboratory data in prognostication of in‐hospital mortality.6 In addition, walking impairment was 1 of 5 components of a risk adjustment index developed to predict 1‐year mortality for hospitalized older persons.7 Atypical presentations of illness, such as delirium and falls, independently predicted poor outcomes among hospitalized older patients.8

Specifically for pneumonia, functional status has also been shown to be an independent predictor of short‐term mortality among older patients hospitalized with CAP.913 Among atypical presentations, only absence of chills was an independent prognostic factor for CAP.9 Bacteremia was an independent factor related to death among adults with CAP, albeit for severe disease resulting in intensive care unit admission.14 It was also included in a severity assessment score; its higher scores were associated with early mortality.15 However, blood culture results are only available 2 to 3 days into the hospital episode. Therefore, bacteremia is a potential risk factor for mortality that is not identifiable at the start of hospitalization.

While PSI is a comprehensive collection of demographic, clinical, and investigative measures, it does not include items on functional status or atypical presentation. Neither does it account for recent hospitalization or comorbid conditions of significance to older persons, such as dementia and depression. It is plausible that at least some of these factors hold added prognostic value.

With all these in mind, we conducted a study with the following objectives: 1) to determine whether functional impairment, recent hospitalization, comorbid conditions of particular significance with advancing age, and atypical presentation are significantly associated with short‐term mortality among older patients hospitalized for CAP and HCAP, after taking into account PSI; and 2) if so, to estimate the magnitude of increased mortality risk with these factors. We tested our null hypotheses that, after adjustment for PSI class, 1) recent hospitalization, 2) pre‐morbid functional impairment, 3) dementia and depression, and 4) atypical presentation of illness have no association with 30‐day mortality for older persons hospitalized for CAP and HCAP, both combined and alone.

PATIENTS AND METHODS

Patient Population

We included first hospital episodes of adults aged 65 years or older with the principal diagnosis of pneumonia in 2007. These episodes were identified by their primary International Classification of Diseases, 9th revision, Clinical Modification (ICD‐9‐CM) codes of 480 to 486 in the administrative data. Next, we applied our study definition of pneumonia, which required the presence of acute symptoms or signs of pneumonia at the point of hospital admission, and a chest radiograph with features consistent with pneumonia that was obtained during the period from 24 hours before, to 48 hours after, hospital admission. In doing so, we included patients with community‐acquired pneumonia (CAP)16 and healthcare‐associated pneumonia (HCAP),17 but not hospital‐acquired pneumonia (HAP). We excluded patients whose charts were not accessible for review because of human immunodeficiency virus/acquired immune deficiency syndrome (HIV/AIDS) and those whose charts were unavailable for other reasons. The study flow diagram is shown in Figure 1.

Figure 1

We assigned the diagnosis of HCAP to patients who were admitted to an acute care hospital for 2 or more days in the prior 90 days, resided in a nursing home or long‐term care facility, or received of intravenous antibiotic therapy, chemotherapy, wound care, or hemodialysis in the prior 30 days.18 Remaining patients were assigned CAP.

RESULTS

Among 1607 patients included, 890 (55.4%) had CAP and 717 (44.6%) had HCAP. Baseline patient characteristics of patients with CAP and HCAP are shown in Table 1. The 30‐day mortality rate was 28.1% for the entire group, and 20.6% and 37.4% for patients with CAP and HCAP, respectively. When stratified according to PSI classes 2, 3, 4, and 5, this rate was 0%, 8.2%, 24.4%, and 56.0%, respectively. Because there were no deaths among those with PSI class 2, this category was merged with class 3 for the regression analyses. Missing data on pre‐morbid ambulation impairment and feeding impairment occurred for 39 (2.4%) and 69 (4.6%) patients, respectively.

For CAP and HCAP together, pre‐morbid ambulation impairment was associated with increased 30‐day mortality (339/798 [42.5%] vs 112/809 [13.8%], unadjusted OR 4.60, 95% CI 3.60 to 5.87, P < 0.01), as was hospitalization in the prior 30 days (94/209 [45.0%] vs 357/1398 [25.5%], unadjusted OR 2.38, 95% CI 1.77 to 3.21, P = 0.02). This was also the case for dementia (118/307 [38.4%] vs 333/1300 [25.6%], unadjusted OR 1.81, 95% CI 1.40 to 2.35, P < 0.01), acute geriatric syndromes (163/442 [36.9%] vs 288/1165 [24.7%], unadjusted OR 1.78, 95% CI 1.41 to 2.25, P < 0.01), and absence of cough and purulent sputum (226/559 [40.4%] vs 225/1048 [21.5%], unadjusted OR 2.48, 95% CI 1.98 to 3.11, P < 0.01). However, depression was not significantly associated with 30‐day mortality (57/165 [34.6%] vs 394/1442 [27.3%], unadjusted OR 1.40, 95% CI 1.00 to 1.97, P = 0.05).

Table 2 summarizes the results of logistic regression. It shows that pre‐morbid ambulation impairment, hospitalization in the prior 30 days, and absence of cough and purulent sputum were all independently associated with 30‐day mortality after adjustment for PSI score for the entire group. These associations remained statistically significant when CAP and HCAP were examined separately. Because none of those with CAP could have hospitalization in the prior 30 days, this factor was not included in the CAP model. The strength of association for the same patient factor varied across the pneumonia sub‐type. This was markedly so for pre‐morbid ambulation impairment, with the OR estimate being almost 3‐fold higher for CAP than for HCAP. Dementia, depression, and acute geriatric syndromes were not associated with 30‐day mortality. When the analyses were repeated after excluding hospital episodes with missing values for pre‐morbid ambulation impairment, the same 3 variables were significantly associated with 30‐day mortality, with trivial differences in strength of association compared to when imputation was performed. The OR estimates for pre‐morbid ambulation impairment, hospitalization in the prior 30 days, and absence of cough and purulent sputum were 2.82 (95% CI 2.12 to 3.76), 1.83 (95% CI 1.42 to 2.83), and 1.47 (95% CI 1.14 to 1.91).

Two‐level hierarchical modeling to account for clustering at the hospital level obtained negligible change in OR estimates of the patient factors and their 95% CI. There were no statistically significant interactions between PSI class and the 3 patient factors (results not shown).

The model‐predicted increase in mortality risk with presence of individual patient factors for the entire group is shown in Table 3. Across the 3 factors, 30‐day mortality increased by 1.9% to 6.1% for those with PSI class 2 and 3, and by 9.0% to 23.2% for those with PSI class 5. The upper end of these ranges represented the effect of pre‐morbid ambulation impairment, while the lower end was that for absence of cough and purulent sputum. With reference to the predicted mortality rates for PSI class which are listed in the footnotes of Table 3, the adverse prognosis conferred by individual patient factors amounted to relative risk inflation of 27% to 145% depending on the specific factor and PSI class.

DISCUSSION

After accounting for PSI class, we found 3 additional patient factors that were independently associated with 30‐day mortality among older persons hospitalized for pneumonia. Firstly, our study confirms that impaired physical function reflected by pre‐morbid ambulation impairment increases mortality risk, as previously demonstrated by Torres et al.10 It is likely that impaired function reflects an underlying vulnerability for adverse outcomes that is seen across primary diagnoses.7 Secondly, recent hospitalization often indicates clinical, functional, and social complexities, as well as increased likelihood of infection by more virulent organisms commonly associated with healthcare‐related infections. Together, these 2 factors could increase mortality risk. Thirdly, atypical presentations may be associated with increased mortality, because these often occur in frail older persons who are vulnerable to adverse outcomes8 due to diseases suffered and treatment received. Atypical presentations may also result in delayed diagnosis and treatment of pneumonia.

Pilotto et al. found that a multidimensional index comprising functional status, comorbidity burden, mental status, and nutritional assessment, among others, had a higher predictive accuracy for 30‐day mortality than did PSI.20 While there was a previous attempt to combine PSI with independent predictors to identify low‐risk older patients with CAP,21 we could not find similar work on the range of patient factors examined in this study. Indeed, the most important contribution that our study brings to the growing body of literature on short‐term mortality, among older persons hospitalized for pneumonia, is the prognostic importance of these 3 additional patient factors over and above severity of illness measured by PSI. With reference to the baseline predicted risk for different PSI class categories shown in Table 3, we have demonstrated that the predicted increase in mortality risk with the presence of these 3 factors is often not trivial, particularly for those with more severe pneumonia.

These 3 patient factors retained prognostic significance after accounting for PSI class for HCAP. However, only 2 factors were associated with mortality for CAP, because by definition recent hospitalization does not occur. A relevant discussion point is whether CAP and HCAP should be grouped together or classified separately. It is pertinent to reflect that the utility of making a distinction between CAP and HCAP appears to lie largely in the domain of therapeutics regarding the initial choice of antibiotics,18, 2225 although there has been some debate on this point.26 Moreover, the major features of HCAP, namely recent hospitalization (albeit in the prior 30 days, rather than 90 days) and nursing home residence (an item in PSI) were included in our regression analyses. Therefore, it seems reasonable to consider CAP and HCAP as a single group for risk stratification at the clinical frontline. We also argue that combining CAP and HCAP for risk adjustment will result in larger sample sizes that can minimize uncertainty around treatment effect estimates, when comparing across different interventions or providers. The same approach of analyzing CAP and HCAP together was adopted in a recent study that compared US hospitals on their risk‐adjusted performance for pneumonia among Medicare beneficiaries.27

The 30‐day mortality rates in this study are higher than those in the original PSI studies, even when stratified according to PSI class. However, more recent studies also registered relatively high mortality rates ranging from 18% to 19%.12, 28 There are a number of possible reasons for the higher mortality rates observed in our study. Firstly, we included both CAP and HCAP, whereas some other studies focused only on CAP. Secondly, the original PSI studies excluded patients with previous hospitalization within 7 days of admission, while we included them. Thirdly, our study population was relatively old (median age: 80 years) and had a higher proportion from nursing homes (22%). Although age and nursing home residence are variables in the PSI, the weights assigned to these 2 items may not adequately reflect the magnitude of mortality risk they confer. Finally, our understanding is that the study population comprises a relatively high proportion of patients who have do‐not‐resuscitate (DNR) instructions, though this was not measured. All these patient characteristics are likely to be associated with higher mortality risk.

The major strength of this study relates to its real world setting, where there were no major exclusion criteria except for HIV/AIDS. In addition, the clinical data at our disposal allowed selection from a relatively wide range of patient factors, beyond that commonly available in administrative data alone.