User login
An important role of the internist is that of inpatient medical consultant.13 As consultants, internists make recommendations regarding the patient's medical care and help the primary team to care for the patient. This requires familiarity with the body of knowledge of consultative medicine, as well as process skills that relate to working with teams of providers.1, 4, 5 For some physicians, the knowledge and skills of medical consultation are acquired during residency; however, many internists feel inadequately prepared for their roles of consultants.68 Because no specific requirements for medical consultation curricula during graduate medical education have been set forth, internists and other physicians do not receive uniform or comprehensive training in this area.3, 57, 9 Although internal medicine residents may gain experience while performing consultations on subspecialty rotations (eg, cardiology), the teaching on these blocks tends to be focused on the specialty content and less so on consultative principles.1, 4
As inpatient care is increasingly being taken over by hospitalists, the role of the hospitalist has expanded to include medical consultation. It is estimated that 92% of hospitalists care for patients on medical consultation services.8 The Society of Hospital Medicine (SHM) has also included medical consultation as one of the core competencies of the hospitalist.2 Therefore, it is essential that hospitalists master the knowledge and skills that are required to serve as effective consultants.10, 11
An educational strategy that has been shown to be effective in improving medical practice is audit and feedback.1215 Providing physicians with feedback on their clinical practice has been shown to improve performance more so than other educational methods.12 Practice‐based learning and improvement (PBLI) utilizes this strategy and it has become one of the core competencies stressed by the Accreditation Council for Graduate Medical Education (ACGME). It involves analyzing one's patient care practices in order to identify areas for improvement. In this study, we tested the impact of a newly developed one‐on‐one medical consultation educational module that was combined with audit and feedback in an attempt to improve the quality of the consultations being performed by our hospitalists.
Materials and Methods
Study Design and Setting
This single group pre‐post educational intervention study took place at Johns Hopkins Bayview Medical Center (JHBMC), a 353‐bed university‐affiliated tertiary care medical center in Baltimore, MD, during the 2006‐2007 academic year.
Study Subjects
All 7 members of the hospitalist group at JHBMC who were serving on the medical consultation service during the study period participated. The internal medicine residents who elected to rotate on the consultation service during the study period were also exposed to the case‐based module component of the intervention.
Intervention
The educational intervention was delivered as a one‐on‐one session and lasted approximately 1 hour. The time was spent on the following activities:
A true‐false pretest to assess knowledge based on clinical scenarios (Appendix 1).
A case‐based module emphasizing the core principles of consultative medicine.16 The module was purposively designed to teach and stimulate thought around 3 complex general medical consultations. These cases are followed by questions about scenarios. The cases specifically address the role of medical consultant and the ways to be most effective in this role based on the recommendations of experts in the field.1, 10 Additional details about the content and format can be viewed at
http://www.jhcme.com/site .16 As the physician was working through the teaching cases, the teacher would facilitate discussion around wrong answers and issues that the learner wanted to discuss.The true‐false test to assess knowledge was once again administered (the posttest was identical to the pretest).
For the hospitalist faculty members only (and not the residents), audit and feedback was utilized. The physician was shown 2 of his/her most recent consults and was asked to reflect upon the strengths and weaknesses of the consult. The hospitalist was explicitly asked to critique them in light of the knowledge they gained from the consultation module. The teacher also gave specific feedback, both positive and negative, about the written consultations with attention directed specifically toward: the number of recommendations, the specificity of the guidance (eg, exact dosing of medications), clear documentation of their name and contact information, and documentation that the suggestions were verbally passed on to the primary team.
Evaluation Data
Learner knowledge, both at baseline and after the case‐based module, was assessed using a written test.
Consultations performed before and after the intervention were compared. Copies of up to 5 consults done by each hospitalist during the year before or after the educational intervention were collected. Identifiers and dates were removed from the consults so that scorers did not know whether the consults were preintervention or postintervention. Consults were scored out of a possible total of 4 to 6 pointsdepending on whether specific elements were applicable. One point was given for each of the following: (1) number of recommendations 5; (2) specific details for all drugs listed [if applicable]; (3) specific details for imaging studies suggested [if applicable]; (4) specific follow‐up documented; (5) consultant's name being clearly written; and (6) verbal contact with the referring team documented. These 6 elements were included based on expert recommendation.10 All consults were scored by 2 hospitalists independently. Disagreements in scores were infrequent (on <10% of the 48 consults scored) and these were only off by 1 point for the overall score. The disagreements were settled by discussion and consensus. All consult scores were converted to a score out of 5, to allow comparisons to be made.
Following the intervention, each participant completed an overall assessment of the educational experience.
Data Analysis
We examined the frequency of responses for each variable and reviewed the distributions. The knowledge scores on the written pretests were not normally distributed and therefore when making comparisons to the posttest, we used the Wilcoxon rank signed test. In comparing the performance scores on the consults across the 2 time periods, we compared the results with both Wilcoxon rank signed test and paired t tests. Because the results were equivalent with both tests, the means from the t tests are shown. Data were analyzed using STATA version 8 (Stata Corp., College Station, TX).
Results
Study Subjects
Among the 14 hospitalist faculty members who were on staff during the study period, 7 were performing medical consults and therefore participated in the study. The 7 faculty members had a mean age of 35 years; 5 (71%) were female, and 5 (71%) were board‐certified in Internal Medicine. The average elapsed time since completion of residency was 5.1 years and average number of years practicing as a hospitalist was 3.8 years (Table 1).
| Faculty (n = 7) | |
| Age in years, mean (SD) | 35.57 (5.1) |
| Female, n (%) | 5 (71%) |
| Board certified, n (%) | 5 (71%) |
| Years since completion of residency, mean (SD) | 5.1 (4.4) |
| Number of years in practice, mean (SD) | 3.8 (2.9) |
| Weeks spent in medical consult rotation, mean (SD) | 3.7 (0.8) |
| Have read consultation books, n (%) | 5 (71%) |
| Housestaff (n = 11) | |
| Age in years, mean (SD) | 29.1 (1.8) |
| Female, n (%) | 7 (64%) |
| Residency year, n (%) | |
| PGY1 | 0 (0%) |
| PGY2 | 2 (20%) |
| PGY3 | 7 (70%) |
| PGY4 | 1 (10%) |
| Weeks spent in medical consult rotation, mean (SD) | 1.5 (0.85) |
| Have read consultation books, n (%) | 5 (50%) |
There were 12 house‐staff members who were on their medical consultation rotation during the study period and were exposed to the intervention. Of the 12 house‐staff members, 11 provided demographic information. Characteristics of the 11 house‐staff participants are also shown in Table 1.
Premodule vs. Postmodule Knowledge Assessment
Both faculty and house‐staff performed very well on the true/false pretest. The small changes in the median scores from pretest to posttest did not change significantly for the faculty (pretest: 11/14, posttest: 12/14; P = 0.08), but did reach statistical significance for the house‐staff (pretest: 10/14, posttest: 12/14; P = 0.03).
Audit and Feedback
Of the 7 faculty who participated in the study, 6 performed consults both before and after the intervention. Using the consult scoring system, the scores for all 6 physicians' consults improved after the intervention compared to their earlier consults (Table 2). For 1 faculty member, the consult scores were statistically significantly higher after the intervention (P = 0.017). When all consults completed by the hospitalists were compared before and after the training, there was statistically significant improvement in consult scores (P < 0.001) (Table 2).
| Preintervention (n =27) | Postintervention (n = 21) | ||||
|---|---|---|---|---|---|
| Consultant | Scores* | Mean | Scores* | Mean | P value |
| |||||
| A | 2, 3, 3.75, 3, 2.5 | 2.8 | 3, 3, 3, 4, 4 | 3.4 | 0.093 |
| B | 3, 3, 3, 3, 1 | 2.6 | 4, 3, 3, 2.5 | 3.1 | 0.18 |
| C | 2, 1.67 | 1.8 | 4, 2, 3 | 3.0 | 0.11 |
| D | 4, 2.5, 3.75, 2.5, 3.75 | 3.3 | 3.75, 3 | 3.4 | 0.45 |
| E | 2, 3, 1, 2, 2 | 2.0 | 3, 3, 3.75 | 3.3 | 0.017 |
| F | 3, 3.75, 2.5, 4, 2 | 3.1 | 2, 3.75, 4, 4 | 3.3 | 0.27 |
| All | 2.7 | 3.3 | 0.0006 | ||
Satisfaction with Consultation Curricula
All faculty and house‐staff participants felt that the intervention had an impact on them (19/19, 100%). Eighteen out of 19 participants (95%) would recommend the educational session to colleagues. After participating, 82% of learners felt confident in performing medical consultations. With respect to the audit and feedback process of reviewing their previously performed consultations, all physicians claimed that their written consultation notes would change in the future.
Discussion
This curricular intervention using a case‐based module combined with audit and feedback appears to have resulted not only in improved knowledge, but also changed physician behavior in the form of higher‐quality written consultations. The teaching sessions were also well received and valued by busy hospitalists.
A review of randomized trials of audit and feedback12 revealed that this strategy is effective in improving professional practice in a variety of areas, including laboratory overutilization,13, 14 clinical practice guideline adherence,15, 17 and antibiotic utilization.13 In 1 study, internal medicine specialists audited their consultation letters and most believed that there had been lasting improvements to their notes.18 However, this study did not objectively compare the consultation letters from before audit and feedback to those written afterward but instead relied solely on the respondents' self‐assessment. It is known that many residents and recent graduates of internal medicine programs feel inadequately prepared in the role of consultant.6, 8 This work describes a curricular intervention that served to augment confidence, knowledge, and actual performance in consultation medicine of physicians. Goldman et al.'s10 Ten Commandments for Effective Consultations, which were later modified by Salerno et al.,11 were highlighted in our case‐based teachings: determine the question being asked or how you can help the requesting physician, establish the urgency of the consultation, gather primary data, be as brief as appropriate in your report, provide specific recommendations, provide contingency plans and discuss their execution, define your role in conjunction with the requesting physician, offer educational information, communicate recommendations directly to the requesting physician, and provide daily follow‐up. These tenets informed the development of the consultation scoring system that was used to assess the quality of the written consultations produced by our consultant hospitalists.
Audit and feedback is similar to PBLI, one of the ACGME core competencies for residency training. Both attempt to engage individuals by having them analyze their patient care practices, looking critically to: (1) identify areas needing improvement, and (2) consider strategies that can be implemented to enhance clinical performance. We now show that consultative medicine is an area that appears to be responsive to a mixed methodological educational intervention that includes audit and feedback.
Faculty and house‐staff knowledge of consultative medicine was assessed both before and after the case‐based educational module. Both groups scored very highly on the true/false pretest, suggesting either that their knowledge was excellent at baseline or the test was not sufficiently challenging. If their knowledge was truly very high, then the intervention need not have focused on improving knowledge. It is our interpretation that the true/false knowledge assessment was not challenging enough and therefore failed to comprehensively characterize their knowledge of consultative medicine.
Several limitations of this study should be considered. First, the sample size was small, including only 7 faculty and 12 house‐staff members. However, these numbers were sufficient to show statistically significant overall improvements in both knowledge and on the consultation scores. Second, few consultations were performed by each faculty member, ranging from 2 to 5, before and after the intervention. This may explain why only 1 out of 6 faculty members showed statistically significant improvement in the quality of consults after the intervention. Third, the true/false format of the knowledge tests allowed the subjects to score very high on the pretest, thereby making it difficult to detect knowledge gained after the intervention. Fourth, the scale used to evaluate consults has not been previously validated. The elements assessed by this scale were decided upon based on guidance from the literature10 and the authors' expertise, thereby affording it content validity evidence.19 The recommendations that guided the scale's development have been shown to improve compliance with the recommendations put forth by the consultant.1, 11 Internal structure validity evidence was conferred by the high level of agreement in scores between the independent raters. Relation to other variables validity evidence may be considered because doctors D and F scored highest on this scale and they are the 2 physicians most experienced in consult medicine. Finally, the educational intervention was time‐intensive for both learners and teacher. It consisted of a 1 hour‐long one‐on‐one session. This can be difficult to incorporate into a busy hospitalist program. The intervention can be made more efficient by having learners take the web‐based module online independently, and then meeting with the teacher for the audit and feedback component.
This consult medicine curricular intervention involving audit and feedback was beneficial to hospitalists and resulted in improved consultation notes. While resource intensive, the one‐on‐one teaching session appears to have worked and resulted in outcomes that are meaningful with respect to patient care.
- , .Kammerer and Gross' Medical Consultation: the Internist on Surgical, Obstetric, and Psychiatric Services.3rd ed.Baltimore:Williams and Wilkins;1998.
- Society of Hospital Medicine.Hospitalist as consultant.J Hosp Med.2006;1(S1):70.
- .The internist as consultant.Arch Intern Med.1980;140:137–138.
- , , .Development of an academic section of general internal medicine.Am J Med.1977;63(4):493–498.
- , , , , .Consultations in internal medicine: a training program resource.J Med Educ.1977;52(4):323–327.
- , , .Practice patterns and the adequacy of residency training in consultation medicine.J Gen Intern Med.1993;8(10):554–560.
- , , .General internal medicine consultation: the last bridge.Arch Intern Med.1983;143:875–876.
- ,, , .Hospitalists' perceptions of their residency training needs: results of a national survey.Am J Med.2001;111(3):247–254.
- .The service and educational contributions of a general medicine consultation service.J Gen Intern Med.1986;1:225–227.
- , , .Ten commandments for effective consultations.Arch Intern Med.1983;143:1753–1755.
- , , , .Principles of effective consultation, an update for the 21st‐century consultant.Arch Intern Med.2007;167:271–275.
- , , , , .Does telling people what they have been doing change what they do? A systematic review of the effects of audit and feedback.Qual Saf Health Care.2006;15:433–436.
- , , , .Factors contributing to inappropriate ordering of tests in an academic medical department and the effect of an educational feedback strategy.Postgrad Med J.2006;82:823–829.
- , , , et al.Effects of routine individual feedback over nine years on general practitioners' requests for tests.BMJ.1996;312:490.
- , , , .Improving antibiotic utilization among hospitalists: a pilot academic detailing project with a public health approach.J Hosp Med.2008;3(1):64–70.
- , . The role of the medical consultant. Johns Hopkins Consultative Medicine Essentials for Hospitalists. Available at:http://www.jhcme.com/site/article.cfm?ID=8. Accessed April2009.
- , , .Audit and feedback and clinical practice guideline adherence: making feedback actionable.Implement Sci.2006;1:9.
- , , , .Peer assessment of outpatient consultation letters—feasibility and satisfaction.BMC Med Educ.2007;7:13.
- , , .What is the validity evidence for assessment of clinical teaching?J Gen Intern Med.2005;20:1159–1164.
An important role of the internist is that of inpatient medical consultant.13 As consultants, internists make recommendations regarding the patient's medical care and help the primary team to care for the patient. This requires familiarity with the body of knowledge of consultative medicine, as well as process skills that relate to working with teams of providers.1, 4, 5 For some physicians, the knowledge and skills of medical consultation are acquired during residency; however, many internists feel inadequately prepared for their roles of consultants.68 Because no specific requirements for medical consultation curricula during graduate medical education have been set forth, internists and other physicians do not receive uniform or comprehensive training in this area.3, 57, 9 Although internal medicine residents may gain experience while performing consultations on subspecialty rotations (eg, cardiology), the teaching on these blocks tends to be focused on the specialty content and less so on consultative principles.1, 4
As inpatient care is increasingly being taken over by hospitalists, the role of the hospitalist has expanded to include medical consultation. It is estimated that 92% of hospitalists care for patients on medical consultation services.8 The Society of Hospital Medicine (SHM) has also included medical consultation as one of the core competencies of the hospitalist.2 Therefore, it is essential that hospitalists master the knowledge and skills that are required to serve as effective consultants.10, 11
An educational strategy that has been shown to be effective in improving medical practice is audit and feedback.1215 Providing physicians with feedback on their clinical practice has been shown to improve performance more so than other educational methods.12 Practice‐based learning and improvement (PBLI) utilizes this strategy and it has become one of the core competencies stressed by the Accreditation Council for Graduate Medical Education (ACGME). It involves analyzing one's patient care practices in order to identify areas for improvement. In this study, we tested the impact of a newly developed one‐on‐one medical consultation educational module that was combined with audit and feedback in an attempt to improve the quality of the consultations being performed by our hospitalists.
Materials and Methods
Study Design and Setting
This single group pre‐post educational intervention study took place at Johns Hopkins Bayview Medical Center (JHBMC), a 353‐bed university‐affiliated tertiary care medical center in Baltimore, MD, during the 2006‐2007 academic year.
Study Subjects
All 7 members of the hospitalist group at JHBMC who were serving on the medical consultation service during the study period participated. The internal medicine residents who elected to rotate on the consultation service during the study period were also exposed to the case‐based module component of the intervention.
Intervention
The educational intervention was delivered as a one‐on‐one session and lasted approximately 1 hour. The time was spent on the following activities:
A true‐false pretest to assess knowledge based on clinical scenarios (Appendix 1).
A case‐based module emphasizing the core principles of consultative medicine.16 The module was purposively designed to teach and stimulate thought around 3 complex general medical consultations. These cases are followed by questions about scenarios. The cases specifically address the role of medical consultant and the ways to be most effective in this role based on the recommendations of experts in the field.1, 10 Additional details about the content and format can be viewed at
http://www.jhcme.com/site .16 As the physician was working through the teaching cases, the teacher would facilitate discussion around wrong answers and issues that the learner wanted to discuss.The true‐false test to assess knowledge was once again administered (the posttest was identical to the pretest).
For the hospitalist faculty members only (and not the residents), audit and feedback was utilized. The physician was shown 2 of his/her most recent consults and was asked to reflect upon the strengths and weaknesses of the consult. The hospitalist was explicitly asked to critique them in light of the knowledge they gained from the consultation module. The teacher also gave specific feedback, both positive and negative, about the written consultations with attention directed specifically toward: the number of recommendations, the specificity of the guidance (eg, exact dosing of medications), clear documentation of their name and contact information, and documentation that the suggestions were verbally passed on to the primary team.
Evaluation Data
Learner knowledge, both at baseline and after the case‐based module, was assessed using a written test.
Consultations performed before and after the intervention were compared. Copies of up to 5 consults done by each hospitalist during the year before or after the educational intervention were collected. Identifiers and dates were removed from the consults so that scorers did not know whether the consults were preintervention or postintervention. Consults were scored out of a possible total of 4 to 6 pointsdepending on whether specific elements were applicable. One point was given for each of the following: (1) number of recommendations 5; (2) specific details for all drugs listed [if applicable]; (3) specific details for imaging studies suggested [if applicable]; (4) specific follow‐up documented; (5) consultant's name being clearly written; and (6) verbal contact with the referring team documented. These 6 elements were included based on expert recommendation.10 All consults were scored by 2 hospitalists independently. Disagreements in scores were infrequent (on <10% of the 48 consults scored) and these were only off by 1 point for the overall score. The disagreements were settled by discussion and consensus. All consult scores were converted to a score out of 5, to allow comparisons to be made.
Following the intervention, each participant completed an overall assessment of the educational experience.
Data Analysis
We examined the frequency of responses for each variable and reviewed the distributions. The knowledge scores on the written pretests were not normally distributed and therefore when making comparisons to the posttest, we used the Wilcoxon rank signed test. In comparing the performance scores on the consults across the 2 time periods, we compared the results with both Wilcoxon rank signed test and paired t tests. Because the results were equivalent with both tests, the means from the t tests are shown. Data were analyzed using STATA version 8 (Stata Corp., College Station, TX).
Results
Study Subjects
Among the 14 hospitalist faculty members who were on staff during the study period, 7 were performing medical consults and therefore participated in the study. The 7 faculty members had a mean age of 35 years; 5 (71%) were female, and 5 (71%) were board‐certified in Internal Medicine. The average elapsed time since completion of residency was 5.1 years and average number of years practicing as a hospitalist was 3.8 years (Table 1).
| Faculty (n = 7) | |
| Age in years, mean (SD) | 35.57 (5.1) |
| Female, n (%) | 5 (71%) |
| Board certified, n (%) | 5 (71%) |
| Years since completion of residency, mean (SD) | 5.1 (4.4) |
| Number of years in practice, mean (SD) | 3.8 (2.9) |
| Weeks spent in medical consult rotation, mean (SD) | 3.7 (0.8) |
| Have read consultation books, n (%) | 5 (71%) |
| Housestaff (n = 11) | |
| Age in years, mean (SD) | 29.1 (1.8) |
| Female, n (%) | 7 (64%) |
| Residency year, n (%) | |
| PGY1 | 0 (0%) |
| PGY2 | 2 (20%) |
| PGY3 | 7 (70%) |
| PGY4 | 1 (10%) |
| Weeks spent in medical consult rotation, mean (SD) | 1.5 (0.85) |
| Have read consultation books, n (%) | 5 (50%) |
There were 12 house‐staff members who were on their medical consultation rotation during the study period and were exposed to the intervention. Of the 12 house‐staff members, 11 provided demographic information. Characteristics of the 11 house‐staff participants are also shown in Table 1.
Premodule vs. Postmodule Knowledge Assessment
Both faculty and house‐staff performed very well on the true/false pretest. The small changes in the median scores from pretest to posttest did not change significantly for the faculty (pretest: 11/14, posttest: 12/14; P = 0.08), but did reach statistical significance for the house‐staff (pretest: 10/14, posttest: 12/14; P = 0.03).
Audit and Feedback
Of the 7 faculty who participated in the study, 6 performed consults both before and after the intervention. Using the consult scoring system, the scores for all 6 physicians' consults improved after the intervention compared to their earlier consults (Table 2). For 1 faculty member, the consult scores were statistically significantly higher after the intervention (P = 0.017). When all consults completed by the hospitalists were compared before and after the training, there was statistically significant improvement in consult scores (P < 0.001) (Table 2).
| Preintervention (n =27) | Postintervention (n = 21) | ||||
|---|---|---|---|---|---|
| Consultant | Scores* | Mean | Scores* | Mean | P value |
| |||||
| A | 2, 3, 3.75, 3, 2.5 | 2.8 | 3, 3, 3, 4, 4 | 3.4 | 0.093 |
| B | 3, 3, 3, 3, 1 | 2.6 | 4, 3, 3, 2.5 | 3.1 | 0.18 |
| C | 2, 1.67 | 1.8 | 4, 2, 3 | 3.0 | 0.11 |
| D | 4, 2.5, 3.75, 2.5, 3.75 | 3.3 | 3.75, 3 | 3.4 | 0.45 |
| E | 2, 3, 1, 2, 2 | 2.0 | 3, 3, 3.75 | 3.3 | 0.017 |
| F | 3, 3.75, 2.5, 4, 2 | 3.1 | 2, 3.75, 4, 4 | 3.3 | 0.27 |
| All | 2.7 | 3.3 | 0.0006 | ||
Satisfaction with Consultation Curricula
All faculty and house‐staff participants felt that the intervention had an impact on them (19/19, 100%). Eighteen out of 19 participants (95%) would recommend the educational session to colleagues. After participating, 82% of learners felt confident in performing medical consultations. With respect to the audit and feedback process of reviewing their previously performed consultations, all physicians claimed that their written consultation notes would change in the future.
Discussion
This curricular intervention using a case‐based module combined with audit and feedback appears to have resulted not only in improved knowledge, but also changed physician behavior in the form of higher‐quality written consultations. The teaching sessions were also well received and valued by busy hospitalists.
A review of randomized trials of audit and feedback12 revealed that this strategy is effective in improving professional practice in a variety of areas, including laboratory overutilization,13, 14 clinical practice guideline adherence,15, 17 and antibiotic utilization.13 In 1 study, internal medicine specialists audited their consultation letters and most believed that there had been lasting improvements to their notes.18 However, this study did not objectively compare the consultation letters from before audit and feedback to those written afterward but instead relied solely on the respondents' self‐assessment. It is known that many residents and recent graduates of internal medicine programs feel inadequately prepared in the role of consultant.6, 8 This work describes a curricular intervention that served to augment confidence, knowledge, and actual performance in consultation medicine of physicians. Goldman et al.'s10 Ten Commandments for Effective Consultations, which were later modified by Salerno et al.,11 were highlighted in our case‐based teachings: determine the question being asked or how you can help the requesting physician, establish the urgency of the consultation, gather primary data, be as brief as appropriate in your report, provide specific recommendations, provide contingency plans and discuss their execution, define your role in conjunction with the requesting physician, offer educational information, communicate recommendations directly to the requesting physician, and provide daily follow‐up. These tenets informed the development of the consultation scoring system that was used to assess the quality of the written consultations produced by our consultant hospitalists.
Audit and feedback is similar to PBLI, one of the ACGME core competencies for residency training. Both attempt to engage individuals by having them analyze their patient care practices, looking critically to: (1) identify areas needing improvement, and (2) consider strategies that can be implemented to enhance clinical performance. We now show that consultative medicine is an area that appears to be responsive to a mixed methodological educational intervention that includes audit and feedback.
Faculty and house‐staff knowledge of consultative medicine was assessed both before and after the case‐based educational module. Both groups scored very highly on the true/false pretest, suggesting either that their knowledge was excellent at baseline or the test was not sufficiently challenging. If their knowledge was truly very high, then the intervention need not have focused on improving knowledge. It is our interpretation that the true/false knowledge assessment was not challenging enough and therefore failed to comprehensively characterize their knowledge of consultative medicine.
Several limitations of this study should be considered. First, the sample size was small, including only 7 faculty and 12 house‐staff members. However, these numbers were sufficient to show statistically significant overall improvements in both knowledge and on the consultation scores. Second, few consultations were performed by each faculty member, ranging from 2 to 5, before and after the intervention. This may explain why only 1 out of 6 faculty members showed statistically significant improvement in the quality of consults after the intervention. Third, the true/false format of the knowledge tests allowed the subjects to score very high on the pretest, thereby making it difficult to detect knowledge gained after the intervention. Fourth, the scale used to evaluate consults has not been previously validated. The elements assessed by this scale were decided upon based on guidance from the literature10 and the authors' expertise, thereby affording it content validity evidence.19 The recommendations that guided the scale's development have been shown to improve compliance with the recommendations put forth by the consultant.1, 11 Internal structure validity evidence was conferred by the high level of agreement in scores between the independent raters. Relation to other variables validity evidence may be considered because doctors D and F scored highest on this scale and they are the 2 physicians most experienced in consult medicine. Finally, the educational intervention was time‐intensive for both learners and teacher. It consisted of a 1 hour‐long one‐on‐one session. This can be difficult to incorporate into a busy hospitalist program. The intervention can be made more efficient by having learners take the web‐based module online independently, and then meeting with the teacher for the audit and feedback component.
This consult medicine curricular intervention involving audit and feedback was beneficial to hospitalists and resulted in improved consultation notes. While resource intensive, the one‐on‐one teaching session appears to have worked and resulted in outcomes that are meaningful with respect to patient care.
An important role of the internist is that of inpatient medical consultant.13 As consultants, internists make recommendations regarding the patient's medical care and help the primary team to care for the patient. This requires familiarity with the body of knowledge of consultative medicine, as well as process skills that relate to working with teams of providers.1, 4, 5 For some physicians, the knowledge and skills of medical consultation are acquired during residency; however, many internists feel inadequately prepared for their roles of consultants.68 Because no specific requirements for medical consultation curricula during graduate medical education have been set forth, internists and other physicians do not receive uniform or comprehensive training in this area.3, 57, 9 Although internal medicine residents may gain experience while performing consultations on subspecialty rotations (eg, cardiology), the teaching on these blocks tends to be focused on the specialty content and less so on consultative principles.1, 4
As inpatient care is increasingly being taken over by hospitalists, the role of the hospitalist has expanded to include medical consultation. It is estimated that 92% of hospitalists care for patients on medical consultation services.8 The Society of Hospital Medicine (SHM) has also included medical consultation as one of the core competencies of the hospitalist.2 Therefore, it is essential that hospitalists master the knowledge and skills that are required to serve as effective consultants.10, 11
An educational strategy that has been shown to be effective in improving medical practice is audit and feedback.1215 Providing physicians with feedback on their clinical practice has been shown to improve performance more so than other educational methods.12 Practice‐based learning and improvement (PBLI) utilizes this strategy and it has become one of the core competencies stressed by the Accreditation Council for Graduate Medical Education (ACGME). It involves analyzing one's patient care practices in order to identify areas for improvement. In this study, we tested the impact of a newly developed one‐on‐one medical consultation educational module that was combined with audit and feedback in an attempt to improve the quality of the consultations being performed by our hospitalists.
Materials and Methods
Study Design and Setting
This single group pre‐post educational intervention study took place at Johns Hopkins Bayview Medical Center (JHBMC), a 353‐bed university‐affiliated tertiary care medical center in Baltimore, MD, during the 2006‐2007 academic year.
Study Subjects
All 7 members of the hospitalist group at JHBMC who were serving on the medical consultation service during the study period participated. The internal medicine residents who elected to rotate on the consultation service during the study period were also exposed to the case‐based module component of the intervention.
Intervention
The educational intervention was delivered as a one‐on‐one session and lasted approximately 1 hour. The time was spent on the following activities:
A true‐false pretest to assess knowledge based on clinical scenarios (Appendix 1).
A case‐based module emphasizing the core principles of consultative medicine.16 The module was purposively designed to teach and stimulate thought around 3 complex general medical consultations. These cases are followed by questions about scenarios. The cases specifically address the role of medical consultant and the ways to be most effective in this role based on the recommendations of experts in the field.1, 10 Additional details about the content and format can be viewed at
http://www.jhcme.com/site .16 As the physician was working through the teaching cases, the teacher would facilitate discussion around wrong answers and issues that the learner wanted to discuss.The true‐false test to assess knowledge was once again administered (the posttest was identical to the pretest).
For the hospitalist faculty members only (and not the residents), audit and feedback was utilized. The physician was shown 2 of his/her most recent consults and was asked to reflect upon the strengths and weaknesses of the consult. The hospitalist was explicitly asked to critique them in light of the knowledge they gained from the consultation module. The teacher also gave specific feedback, both positive and negative, about the written consultations with attention directed specifically toward: the number of recommendations, the specificity of the guidance (eg, exact dosing of medications), clear documentation of their name and contact information, and documentation that the suggestions were verbally passed on to the primary team.
Evaluation Data
Learner knowledge, both at baseline and after the case‐based module, was assessed using a written test.
Consultations performed before and after the intervention were compared. Copies of up to 5 consults done by each hospitalist during the year before or after the educational intervention were collected. Identifiers and dates were removed from the consults so that scorers did not know whether the consults were preintervention or postintervention. Consults were scored out of a possible total of 4 to 6 pointsdepending on whether specific elements were applicable. One point was given for each of the following: (1) number of recommendations 5; (2) specific details for all drugs listed [if applicable]; (3) specific details for imaging studies suggested [if applicable]; (4) specific follow‐up documented; (5) consultant's name being clearly written; and (6) verbal contact with the referring team documented. These 6 elements were included based on expert recommendation.10 All consults were scored by 2 hospitalists independently. Disagreements in scores were infrequent (on <10% of the 48 consults scored) and these were only off by 1 point for the overall score. The disagreements were settled by discussion and consensus. All consult scores were converted to a score out of 5, to allow comparisons to be made.
Following the intervention, each participant completed an overall assessment of the educational experience.
Data Analysis
We examined the frequency of responses for each variable and reviewed the distributions. The knowledge scores on the written pretests were not normally distributed and therefore when making comparisons to the posttest, we used the Wilcoxon rank signed test. In comparing the performance scores on the consults across the 2 time periods, we compared the results with both Wilcoxon rank signed test and paired t tests. Because the results were equivalent with both tests, the means from the t tests are shown. Data were analyzed using STATA version 8 (Stata Corp., College Station, TX).
Results
Study Subjects
Among the 14 hospitalist faculty members who were on staff during the study period, 7 were performing medical consults and therefore participated in the study. The 7 faculty members had a mean age of 35 years; 5 (71%) were female, and 5 (71%) were board‐certified in Internal Medicine. The average elapsed time since completion of residency was 5.1 years and average number of years practicing as a hospitalist was 3.8 years (Table 1).
| Faculty (n = 7) | |
| Age in years, mean (SD) | 35.57 (5.1) |
| Female, n (%) | 5 (71%) |
| Board certified, n (%) | 5 (71%) |
| Years since completion of residency, mean (SD) | 5.1 (4.4) |
| Number of years in practice, mean (SD) | 3.8 (2.9) |
| Weeks spent in medical consult rotation, mean (SD) | 3.7 (0.8) |
| Have read consultation books, n (%) | 5 (71%) |
| Housestaff (n = 11) | |
| Age in years, mean (SD) | 29.1 (1.8) |
| Female, n (%) | 7 (64%) |
| Residency year, n (%) | |
| PGY1 | 0 (0%) |
| PGY2 | 2 (20%) |
| PGY3 | 7 (70%) |
| PGY4 | 1 (10%) |
| Weeks spent in medical consult rotation, mean (SD) | 1.5 (0.85) |
| Have read consultation books, n (%) | 5 (50%) |
There were 12 house‐staff members who were on their medical consultation rotation during the study period and were exposed to the intervention. Of the 12 house‐staff members, 11 provided demographic information. Characteristics of the 11 house‐staff participants are also shown in Table 1.
Premodule vs. Postmodule Knowledge Assessment
Both faculty and house‐staff performed very well on the true/false pretest. The small changes in the median scores from pretest to posttest did not change significantly for the faculty (pretest: 11/14, posttest: 12/14; P = 0.08), but did reach statistical significance for the house‐staff (pretest: 10/14, posttest: 12/14; P = 0.03).
Audit and Feedback
Of the 7 faculty who participated in the study, 6 performed consults both before and after the intervention. Using the consult scoring system, the scores for all 6 physicians' consults improved after the intervention compared to their earlier consults (Table 2). For 1 faculty member, the consult scores were statistically significantly higher after the intervention (P = 0.017). When all consults completed by the hospitalists were compared before and after the training, there was statistically significant improvement in consult scores (P < 0.001) (Table 2).
| Preintervention (n =27) | Postintervention (n = 21) | ||||
|---|---|---|---|---|---|
| Consultant | Scores* | Mean | Scores* | Mean | P value |
| |||||
| A | 2, 3, 3.75, 3, 2.5 | 2.8 | 3, 3, 3, 4, 4 | 3.4 | 0.093 |
| B | 3, 3, 3, 3, 1 | 2.6 | 4, 3, 3, 2.5 | 3.1 | 0.18 |
| C | 2, 1.67 | 1.8 | 4, 2, 3 | 3.0 | 0.11 |
| D | 4, 2.5, 3.75, 2.5, 3.75 | 3.3 | 3.75, 3 | 3.4 | 0.45 |
| E | 2, 3, 1, 2, 2 | 2.0 | 3, 3, 3.75 | 3.3 | 0.017 |
| F | 3, 3.75, 2.5, 4, 2 | 3.1 | 2, 3.75, 4, 4 | 3.3 | 0.27 |
| All | 2.7 | 3.3 | 0.0006 | ||
Satisfaction with Consultation Curricula
All faculty and house‐staff participants felt that the intervention had an impact on them (19/19, 100%). Eighteen out of 19 participants (95%) would recommend the educational session to colleagues. After participating, 82% of learners felt confident in performing medical consultations. With respect to the audit and feedback process of reviewing their previously performed consultations, all physicians claimed that their written consultation notes would change in the future.
Discussion
This curricular intervention using a case‐based module combined with audit and feedback appears to have resulted not only in improved knowledge, but also changed physician behavior in the form of higher‐quality written consultations. The teaching sessions were also well received and valued by busy hospitalists.
A review of randomized trials of audit and feedback12 revealed that this strategy is effective in improving professional practice in a variety of areas, including laboratory overutilization,13, 14 clinical practice guideline adherence,15, 17 and antibiotic utilization.13 In 1 study, internal medicine specialists audited their consultation letters and most believed that there had been lasting improvements to their notes.18 However, this study did not objectively compare the consultation letters from before audit and feedback to those written afterward but instead relied solely on the respondents' self‐assessment. It is known that many residents and recent graduates of internal medicine programs feel inadequately prepared in the role of consultant.6, 8 This work describes a curricular intervention that served to augment confidence, knowledge, and actual performance in consultation medicine of physicians. Goldman et al.'s10 Ten Commandments for Effective Consultations, which were later modified by Salerno et al.,11 were highlighted in our case‐based teachings: determine the question being asked or how you can help the requesting physician, establish the urgency of the consultation, gather primary data, be as brief as appropriate in your report, provide specific recommendations, provide contingency plans and discuss their execution, define your role in conjunction with the requesting physician, offer educational information, communicate recommendations directly to the requesting physician, and provide daily follow‐up. These tenets informed the development of the consultation scoring system that was used to assess the quality of the written consultations produced by our consultant hospitalists.
Audit and feedback is similar to PBLI, one of the ACGME core competencies for residency training. Both attempt to engage individuals by having them analyze their patient care practices, looking critically to: (1) identify areas needing improvement, and (2) consider strategies that can be implemented to enhance clinical performance. We now show that consultative medicine is an area that appears to be responsive to a mixed methodological educational intervention that includes audit and feedback.
Faculty and house‐staff knowledge of consultative medicine was assessed both before and after the case‐based educational module. Both groups scored very highly on the true/false pretest, suggesting either that their knowledge was excellent at baseline or the test was not sufficiently challenging. If their knowledge was truly very high, then the intervention need not have focused on improving knowledge. It is our interpretation that the true/false knowledge assessment was not challenging enough and therefore failed to comprehensively characterize their knowledge of consultative medicine.
Several limitations of this study should be considered. First, the sample size was small, including only 7 faculty and 12 house‐staff members. However, these numbers were sufficient to show statistically significant overall improvements in both knowledge and on the consultation scores. Second, few consultations were performed by each faculty member, ranging from 2 to 5, before and after the intervention. This may explain why only 1 out of 6 faculty members showed statistically significant improvement in the quality of consults after the intervention. Third, the true/false format of the knowledge tests allowed the subjects to score very high on the pretest, thereby making it difficult to detect knowledge gained after the intervention. Fourth, the scale used to evaluate consults has not been previously validated. The elements assessed by this scale were decided upon based on guidance from the literature10 and the authors' expertise, thereby affording it content validity evidence.19 The recommendations that guided the scale's development have been shown to improve compliance with the recommendations put forth by the consultant.1, 11 Internal structure validity evidence was conferred by the high level of agreement in scores between the independent raters. Relation to other variables validity evidence may be considered because doctors D and F scored highest on this scale and they are the 2 physicians most experienced in consult medicine. Finally, the educational intervention was time‐intensive for both learners and teacher. It consisted of a 1 hour‐long one‐on‐one session. This can be difficult to incorporate into a busy hospitalist program. The intervention can be made more efficient by having learners take the web‐based module online independently, and then meeting with the teacher for the audit and feedback component.
This consult medicine curricular intervention involving audit and feedback was beneficial to hospitalists and resulted in improved consultation notes. While resource intensive, the one‐on‐one teaching session appears to have worked and resulted in outcomes that are meaningful with respect to patient care.
- , .Kammerer and Gross' Medical Consultation: the Internist on Surgical, Obstetric, and Psychiatric Services.3rd ed.Baltimore:Williams and Wilkins;1998.
- Society of Hospital Medicine.Hospitalist as consultant.J Hosp Med.2006;1(S1):70.
- .The internist as consultant.Arch Intern Med.1980;140:137–138.
- , , .Development of an academic section of general internal medicine.Am J Med.1977;63(4):493–498.
- , , , , .Consultations in internal medicine: a training program resource.J Med Educ.1977;52(4):323–327.
- , , .Practice patterns and the adequacy of residency training in consultation medicine.J Gen Intern Med.1993;8(10):554–560.
- , , .General internal medicine consultation: the last bridge.Arch Intern Med.1983;143:875–876.
- ,, , .Hospitalists' perceptions of their residency training needs: results of a national survey.Am J Med.2001;111(3):247–254.
- .The service and educational contributions of a general medicine consultation service.J Gen Intern Med.1986;1:225–227.
- , , .Ten commandments for effective consultations.Arch Intern Med.1983;143:1753–1755.
- , , , .Principles of effective consultation, an update for the 21st‐century consultant.Arch Intern Med.2007;167:271–275.
- , , , , .Does telling people what they have been doing change what they do? A systematic review of the effects of audit and feedback.Qual Saf Health Care.2006;15:433–436.
- , , , .Factors contributing to inappropriate ordering of tests in an academic medical department and the effect of an educational feedback strategy.Postgrad Med J.2006;82:823–829.
- , , , et al.Effects of routine individual feedback over nine years on general practitioners' requests for tests.BMJ.1996;312:490.
- , , , .Improving antibiotic utilization among hospitalists: a pilot academic detailing project with a public health approach.J Hosp Med.2008;3(1):64–70.
- , . The role of the medical consultant. Johns Hopkins Consultative Medicine Essentials for Hospitalists. Available at:http://www.jhcme.com/site/article.cfm?ID=8. Accessed April2009.
- , , .Audit and feedback and clinical practice guideline adherence: making feedback actionable.Implement Sci.2006;1:9.
- , , , .Peer assessment of outpatient consultation letters—feasibility and satisfaction.BMC Med Educ.2007;7:13.
- , , .What is the validity evidence for assessment of clinical teaching?J Gen Intern Med.2005;20:1159–1164.
- , .Kammerer and Gross' Medical Consultation: the Internist on Surgical, Obstetric, and Psychiatric Services.3rd ed.Baltimore:Williams and Wilkins;1998.
- Society of Hospital Medicine.Hospitalist as consultant.J Hosp Med.2006;1(S1):70.
- .The internist as consultant.Arch Intern Med.1980;140:137–138.
- , , .Development of an academic section of general internal medicine.Am J Med.1977;63(4):493–498.
- , , , , .Consultations in internal medicine: a training program resource.J Med Educ.1977;52(4):323–327.
- , , .Practice patterns and the adequacy of residency training in consultation medicine.J Gen Intern Med.1993;8(10):554–560.
- , , .General internal medicine consultation: the last bridge.Arch Intern Med.1983;143:875–876.
- ,, , .Hospitalists' perceptions of their residency training needs: results of a national survey.Am J Med.2001;111(3):247–254.
- .The service and educational contributions of a general medicine consultation service.J Gen Intern Med.1986;1:225–227.
- , , .Ten commandments for effective consultations.Arch Intern Med.1983;143:1753–1755.
- , , , .Principles of effective consultation, an update for the 21st‐century consultant.Arch Intern Med.2007;167:271–275.
- , , , , .Does telling people what they have been doing change what they do? A systematic review of the effects of audit and feedback.Qual Saf Health Care.2006;15:433–436.
- , , , .Factors contributing to inappropriate ordering of tests in an academic medical department and the effect of an educational feedback strategy.Postgrad Med J.2006;82:823–829.
- , , , et al.Effects of routine individual feedback over nine years on general practitioners' requests for tests.BMJ.1996;312:490.
- , , , .Improving antibiotic utilization among hospitalists: a pilot academic detailing project with a public health approach.J Hosp Med.2008;3(1):64–70.
- , . The role of the medical consultant. Johns Hopkins Consultative Medicine Essentials for Hospitalists. Available at:http://www.jhcme.com/site/article.cfm?ID=8. Accessed April2009.
- , , .Audit and feedback and clinical practice guideline adherence: making feedback actionable.Implement Sci.2006;1:9.
- , , , .Peer assessment of outpatient consultation letters—feasibility and satisfaction.BMC Med Educ.2007;7:13.
- , , .What is the validity evidence for assessment of clinical teaching?J Gen Intern Med.2005;20:1159–1164.
Copyright © 2009 Society of Hospital Medicine