User login
STUDY DESIGN: Using a randomized pretest/posttest control group design, we assigned physicians to the CART group or the control group, followed up prospectively, and evaluated for appropriate adherence to guidelines. The 3 age-specific versions of the CART annotated history and physical examination form contained up to 49 preventive services recommendations.
POPULATION: All resident physicians in a large family practice residency program were studied over the course of 1 academic year.
OUTCOMES MEASURED: We performed blinded chart reviews to assess the appropriateness of preventive services ordered by the physicians before the introduction of the CART, during its use, and after its removal. A multiple-choice test completed before and after the use of the CART forms assessed knowledge.
RESULTS: When the CART was used, the appropriateness of physician preventive behavior increased by 21% overall. The appropriateness of history, physical examination, and laboratory interventions increased by 33%. When the CART was removed, physician behavior returned to baseline (P≤.0025 for 16 of 20 interventions). No significant differences were observed in the control group over time. Knowledge increased during the study period for all physicians (P <.005) but did not differ significantly between the treatment and control groups (P=.608).
CONCLUSIONS: Use of the CART significantly improved physician performance in the appropriate delivery of preventive care.
Frame and Carlson1 published a critical review of periodic health screenings in The Journal of Family Practice in 1975. Such efforts, currently manifested in the United States Preventive Services Task Force (USPSTF), have had a profound impact on the practice of family physicians. Increased oversight of family physicians’ preventive practices by third parties and the increased medical sophistication of the public have fueled demand for preventive health services. Family physicians are experiencing increased accountability to their patients and communities for the effective delivery of preventive medical care.2-4
A number of systems have been developed to assist physicians with the implementation of evidence-based preventive services guidelines. Over the past 15 years, numerous researchers have studied the effects of interventions using flow charts,5-7 checklists,8 computer-based reminders,9-14 and educational programs15-19 for delivery of preventive services. Ornstein and colleagues10 found that the use of a computer-based tracking and reminder system for preventive services, together with a physician education program, modestly increased the frequency of counseling and screening services for patients. Similarly, Cheney and Ramsdell8 found that physicians who received an age- and sex-specific checklist in their patients’ medical charts performed significantly more age- and sex-appropriate preventive services than control physicians. But physician compliance with screening tools is often a problem.6 In the Cheney and Ramsdell study, only 39% of physicians used the available checklist to screen patients. Another study14 found a greater percentage of appropriate health screens by physicians using computerized reminders. However, many physicians did not make use of the available resource. Some studies found only minimal increases in screening rates with interventions such as educational seminars.19 One study5 used a simple flow sheet to analyze appropriate health service needs and noted that their results suggest that a flow sheet, although helpful, is not enough. Indeed, accomplishing sustained improvement in preventive services clinical outcomes is a complex process.20
We investigated the effect of combining a complete set of annotated recommendations with a history and physical examination form on the rate of appropriate application of preventive service recommendations. We suspected that busy practitioners would need preventive services guidelines available at the point of care to be able to adhere closely, so we created a form that is very user-friendly. We also studied the effect of removing the screening tool after the intervention, to evaluate whether the behavioral effect was due to education or to the presence of the chart tool, a question that has not been resolved in several previous studies.
Methods
Study Setting and Instrument
Our study was conducted between June 1997 and June 1998 at a community hospital continuity clinic. Age-appropriate screening history questions, age-specific reminders, and test frequency recommendations drawn from the 1996 USPSTF were integrated into a form for documenting history and physical examinations. This Comprehensive Annotated Reminder Tool (CART) contained 8 sections: history, screening history, physical examination, screening mnemonics, laboratory screens, prophylaxis, counseling, and the assessment/plan section. CART forms were developed for patients aged 19-39, 40-64, and 65 or older. Table w1,Table w2, Table w3 3* Each of these 3-page forms contained guidelines for up to 49 preventive interventions. Twenty recommendations were evaluated in this study. Because nursing has been primarily responsible for ensuring immunization compliance in our setting, we did not expect an effect and only evaluated one immunization.
Intervention
Resident physicians were randomly assigned to a treatment group that was exposed to the CART (n=15) and a control group that used existing blank history and physical examination forms (n=16). Resident physicians from both groups precepted patients with the same faculty physicians on a regular basis. Physicians in the control group used a different area of the clinic facility from the treatment group to avoid cross-use of the CART form. The 3 time periods of the study were: pre-intervention phase (3 months), intervention phase (6 months), and post-intervention phase (3 months). The pre-intervention phase allowed for the establishment of a baseline measurement of preventive medicine practiced by physicians in both groups. In the intervention phase, the CART forms were placed on all charts of new patients seen by physicians in the experimental group. The forms were not introduced by any formal instruction. No preventive medicine lectures were given to physicians in either group during the course of the study. Finally, in the post-intervention phase the CART was no longer placed on charts.
Measures
Blinded chart reviews were performed by the principal investigator and 2 other independent reviewers on randomly selected new patients. The physicians were expected to complete a history and physical examination on each new clinic patient within the first 3 visits. Chart reviews occurred before use, during use, and after removal of the CART. They evaluated charts for screening appropriateness (not the absolute frequency of interventions). No credit was given when a preventive intervention was performed that was not indicated. A total of 300 charts were reviewed for the treatment group and 308 for the control group. The inter-rater agreement between the principal investigator and 2 other independent reviewers ranged between 93% and 98%. The median kappa statistic among 16 screening recommendations revealed very good agreement at 0.81 (all P <.042). Kappa could not be calculated for 4 of the screening recommendations, because the observed agreement of the 3 raters (100%) was equal to the expected agreement for those recommendations. All physicians were given a pretest measuring knowledge of the USPSTF recommendations at the beginning of the pre-intervention phase and a posttest near the end of the post-intervention phase. Finally, as physician learning and behavior were the focus of this study, patient adherence was not measured.
Data Analysis
Recommendations were organized into 4 categories: history, examination and laboratory, counseling, and prophylaxis. Tests of significance were calculated using a chi-square test that is designed to compare proportions among many independent samples.21 To control for statistical error, we used a Bonferroni adjusted P value. Because 20 screening recommendations were analyzed, a P value of .0025 (.05/20) was used as the cutoff for statistical significance.
Scores from the knowledge test were analyzed using a mixed factorial analysis of variance. The between-groups factor was group (treatment or control) and the within-subjects factor was test period (pretest or posttest). This analysis was conducted to see if the tool would lead to a group-by-test period interaction. Given the repeated use of the CART, it was thought that the treatment group might show a greater improvement in test scores than the control group.
Results
Impact of the CART
The Table 1 shows the percentages of patients who were appropriately screened during each period for the treatment and control groups. Significant increases were observed for all recommendations in the treatment group, except for occult blood and tetanus and diptheria booster. The mean absolute increase in the percentage of appropriately screened patients was 45%, 21%, and 15% for the recommendations in the history, examination and laboratory, and counseling categories, respectively. Removal of the annotated reminder during the post-intervention period brought the percentages of appropriate screenings down to baseline levels. Increases during the intervention phase were statistically significant for 16 of the 20 recommendations.
In the control group, the percentages of appropriately screened patients remained relatively stable. Although some variation is visible across the 3 study periods, there are no consistent trends among the recommendations, and none of the chi-square tests reached the Bonferroni level of significance. The Figure 1 shows the median percentage of patients screened during the baseline, intervention, and post-intervention periods. The average percentage of appropriate screenings for the clinicians using the CART increased during the intervention period and then returned to baseline levels after the tool was removed from use. The inverted V pattern repeatedly observed for the treatment group is not seen with the control group.
Screening Knowledge Scores
Only the main effect for the test period reached statistical significance. The mean test scores were 54.66 and 62.35 for clinicians at the pretest and posttest periods, respectively (F [1,27] = 23.89; P <.0005). The main effect for group was not statistically significant. The mean test scores for the treatment and control groups were 58.59 and 58.13, respectively (F [1,27] = 0.27; P <.608). The group-by-test period interaction also was not statistically significant (F [1,27] = 1.42, P=.244). Thus, although physicians had higher posttest knowledge scores than pretest scores, this increase was similar for treatment and control groups.
Discussion
CART use increased performance both in degree (percentage change) and in breadth (number of preventive services ordered appropriately). The primary outcome was the appropriateness of physician performance of preventive interventions. For example, if the patient is low-risk, then gonorrhea/chlamydia screening should not be performed, and appropriate physician behavior is to do no screening test. We did not give lectures or provide any other didactic preventive medicine training during the CART study period. But knowledge increased to a similar extent in both groups over that academic year. Nevertheless, the behavioral change seen in the treatment group was temporary; it returned to baseline in the post-intervention period. Therefore, it was the presence of the CART that improved physician behavior. The CART is comprehensive: It has prompts for up to 49 preventive recommendations. Physicians adhered significantly better to history and examination and laboratory recommendations when using the CART. The improvement in counseling was less impressive; physician behavior improved on only 4 of 6 counseling recommendations. It is likely that “form fatigue” was a factor, because the counseling recommendations were at the bottom of a page full of preventive recommendations. The difficulty of counseling, time constraints, and physician perception of the relative ineffectiveness of counseling may have also had an effect.
We believe that the primary reason for the success of the CART is immediate visual access to essential information. Computer reminders promote some behavioral change, but in other studies it was necessary to go to another computer screen for further information about a given test. Time pressure in the busy office makes this difficult. The CART gives physicians succinct information in a box adjacent to each preventive recommendation Table 1, including the recommended frequency, strength of recommendation, and appropriate groups to be screened for each intervention. Most other studies did not answer the question of whether increased knowledge over time was responsible for improved provider behavior. By removing the CART and reassessing physician behavior, we demonstrated that it was presence of the information in the CART format that was responsible for improving physician behavior.
Limitations
Our study has several limitations. We did not address the important issue of reminder systems or checklists. We only assessed the preventive services recommended by physicians during a single periodic health examination. We did not solve the problem of time constraint. It is difficult to cover every single preventive service recommendation in one visit. Our residents are less experienced and have more visit time than the average primary care provider. Finally, we only studied physician knowledge and behavior. We did not measure other important outcomes, such as patient satisfaction, adherence, morbidity, or mortality.
Conclusions
The nonspecific designation “tool” in the CART acronym implies potential future application of this concept for different subject matter and different formats. As we move inexorably toward universal electronic medical records, the CART can be adapted for electronic use. But whether on desktops, laptops, or hand-held computers, immediate visual access to information is essential. Combining the CART with computer reminder systems could yield significant improvement in patient outcomes when followed over time. Also, it would be interesting to incorporate the CART information into a computer-gathered patient history to create a partially completed history and physical examination form for provider use during the periodic health examination. This would save valuable office visit time. Many previous studies have focused on a small number of preventive services. We chose a comprehensive set of recommendations to more closely mimic the experience of primary care physicians who must cover many recommendations with patients. It is important that more studies use this approach. To maximize impact on the public health in an era of cost containment and to minimize physician error, we must enable physicians to efficiently apply a comprehensive set of preventive services in an appropriately focused manner. The CART is an effective step in the right direction.
Acknowledgments
We thank the Advocate Christ Hospital Med Fund for grant support. We would also like to thank Charles Warnell, MD, for his chart review work, and Carolyn Barsano for the literature review.
1. Frame PS, Carlson SJ. A critical review of periodic health screening using specific screening criteria: part 1. Selected diseases of respiratory, cardiovascular, and central nervous system. J Fam Pract 1975;2:29-36.
2. Sox HC, Jr. Preventive health services in adults. New Engl J Med 1994;330:1589-95.
3. Bergman-Evans B, Walker SN. The prevalence of clinical preventive services utilization by older women. Nurse Pract 1996;21:88,90,99-100.
4. Woo B, Woo B, Cook EF, Weidberg M, Goldman L. Screening procedures in the asymptomatic adult. J Am Med Assoc 1985;254:1480-84.
5. Keim DB, Gomez CF, Wolf AMD. The level of preventive health care in an internal medicine residency clinic: still only and ounce of prevention? South Med J 1998;91:550-54.
6. Frame PS, Kowulich BA, Llewellyn AM. Improving physician compliance with a health maintenance protocol. J Fam Pract 1984;19:341-44.
7. Madlon-Kay DJ. Use of a structured encounter form to improve well-child care documentation. Arch Fam Med 1998;7:480-83.
8. Cheney C, Ramsdell JW. Effect of medical records checklists on implementation of periodic health measures. Am J Med 1987;83:129-36.
10. Ornstein SM, Garr DR, Jenkins RG, Musham CM, Hamadeh G, Lancaster C. Implementation and evaluation of a computer-based preventive services system. Fam Med 1995;27:260-66.
11. Ornstein SM, Garr DR, Jenkins RG, Rust PF, Arnon A. Computer-generated physician and patient reminders: tools to improve population adherence to selected preventive services. J Fam Pract 1991;32:82-90.
12. Litzelman DK, Dittus RS, Miller ME, Tierney WM. Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols. J Gen Intern Med 1993;8:311-17.
13. Turner BJ, Day SC, Borenstein B. A controlled trial to improve delivery of preventive health care: physician or patient reminders? J Gen Intern Med 1989;4:403-09.
14. Overhage JM, Tierney WM, McDonald CJ. Computer reminders to implement preventive care guidelines for hospitalized patients. Arch Intern Med 1996;156:1551-56.
15. Tape TG, Campbell JR. Computerized medical records and preventive health care: success depends on many factors. Am J Med 1993;94:619-25.
16. Borum ML. Medical residents colorectal cancer screening may be dependent on ambulatory care education. Dig Dis Sci 1997;42:1176-78.
17. Comninellis NB, Harper DM. Does comprehensive preventive medicine training enhance clinical prevention? Fam Med 1997;29:112-14.
18. Geiger WJ, Neuberger MJ, Bell GC. Implementing the US preventive services guidelines in a family practice residency. Fam Med 1993;25:447-51.
19. Leshan LA, Fitzsimmons M, Marbella A, Gottlieb M. Increasing clinical prevention efforts in a family practice residency program through CQI methods. J Qual Improv 1997;23:391-400.
20. Ward J, Sanson-Fisher R. Does a 3-day workshop for family medicine trainees improve preventive care? A randomized control trial. Prev Med 1996;25:741-47.
21. McBride P, Underbakke G, Plane MB, et al. Improving prevention systems in primary care practices: the health education and research trial (HEART). J Fam Pract 2000;49:115-25.
22. Fleiss JL. Statistical methods for rates and proportions. New York, NY: John Wiley & Sons; 1973.
STUDY DESIGN: Using a randomized pretest/posttest control group design, we assigned physicians to the CART group or the control group, followed up prospectively, and evaluated for appropriate adherence to guidelines. The 3 age-specific versions of the CART annotated history and physical examination form contained up to 49 preventive services recommendations.
POPULATION: All resident physicians in a large family practice residency program were studied over the course of 1 academic year.
OUTCOMES MEASURED: We performed blinded chart reviews to assess the appropriateness of preventive services ordered by the physicians before the introduction of the CART, during its use, and after its removal. A multiple-choice test completed before and after the use of the CART forms assessed knowledge.
RESULTS: When the CART was used, the appropriateness of physician preventive behavior increased by 21% overall. The appropriateness of history, physical examination, and laboratory interventions increased by 33%. When the CART was removed, physician behavior returned to baseline (P≤.0025 for 16 of 20 interventions). No significant differences were observed in the control group over time. Knowledge increased during the study period for all physicians (P <.005) but did not differ significantly between the treatment and control groups (P=.608).
CONCLUSIONS: Use of the CART significantly improved physician performance in the appropriate delivery of preventive care.
Frame and Carlson1 published a critical review of periodic health screenings in The Journal of Family Practice in 1975. Such efforts, currently manifested in the United States Preventive Services Task Force (USPSTF), have had a profound impact on the practice of family physicians. Increased oversight of family physicians’ preventive practices by third parties and the increased medical sophistication of the public have fueled demand for preventive health services. Family physicians are experiencing increased accountability to their patients and communities for the effective delivery of preventive medical care.2-4
A number of systems have been developed to assist physicians with the implementation of evidence-based preventive services guidelines. Over the past 15 years, numerous researchers have studied the effects of interventions using flow charts,5-7 checklists,8 computer-based reminders,9-14 and educational programs15-19 for delivery of preventive services. Ornstein and colleagues10 found that the use of a computer-based tracking and reminder system for preventive services, together with a physician education program, modestly increased the frequency of counseling and screening services for patients. Similarly, Cheney and Ramsdell8 found that physicians who received an age- and sex-specific checklist in their patients’ medical charts performed significantly more age- and sex-appropriate preventive services than control physicians. But physician compliance with screening tools is often a problem.6 In the Cheney and Ramsdell study, only 39% of physicians used the available checklist to screen patients. Another study14 found a greater percentage of appropriate health screens by physicians using computerized reminders. However, many physicians did not make use of the available resource. Some studies found only minimal increases in screening rates with interventions such as educational seminars.19 One study5 used a simple flow sheet to analyze appropriate health service needs and noted that their results suggest that a flow sheet, although helpful, is not enough. Indeed, accomplishing sustained improvement in preventive services clinical outcomes is a complex process.20
We investigated the effect of combining a complete set of annotated recommendations with a history and physical examination form on the rate of appropriate application of preventive service recommendations. We suspected that busy practitioners would need preventive services guidelines available at the point of care to be able to adhere closely, so we created a form that is very user-friendly. We also studied the effect of removing the screening tool after the intervention, to evaluate whether the behavioral effect was due to education or to the presence of the chart tool, a question that has not been resolved in several previous studies.
Methods
Study Setting and Instrument
Our study was conducted between June 1997 and June 1998 at a community hospital continuity clinic. Age-appropriate screening history questions, age-specific reminders, and test frequency recommendations drawn from the 1996 USPSTF were integrated into a form for documenting history and physical examinations. This Comprehensive Annotated Reminder Tool (CART) contained 8 sections: history, screening history, physical examination, screening mnemonics, laboratory screens, prophylaxis, counseling, and the assessment/plan section. CART forms were developed for patients aged 19-39, 40-64, and 65 or older. Table w1,Table w2, Table w3 3* Each of these 3-page forms contained guidelines for up to 49 preventive interventions. Twenty recommendations were evaluated in this study. Because nursing has been primarily responsible for ensuring immunization compliance in our setting, we did not expect an effect and only evaluated one immunization.
Intervention
Resident physicians were randomly assigned to a treatment group that was exposed to the CART (n=15) and a control group that used existing blank history and physical examination forms (n=16). Resident physicians from both groups precepted patients with the same faculty physicians on a regular basis. Physicians in the control group used a different area of the clinic facility from the treatment group to avoid cross-use of the CART form. The 3 time periods of the study were: pre-intervention phase (3 months), intervention phase (6 months), and post-intervention phase (3 months). The pre-intervention phase allowed for the establishment of a baseline measurement of preventive medicine practiced by physicians in both groups. In the intervention phase, the CART forms were placed on all charts of new patients seen by physicians in the experimental group. The forms were not introduced by any formal instruction. No preventive medicine lectures were given to physicians in either group during the course of the study. Finally, in the post-intervention phase the CART was no longer placed on charts.
Measures
Blinded chart reviews were performed by the principal investigator and 2 other independent reviewers on randomly selected new patients. The physicians were expected to complete a history and physical examination on each new clinic patient within the first 3 visits. Chart reviews occurred before use, during use, and after removal of the CART. They evaluated charts for screening appropriateness (not the absolute frequency of interventions). No credit was given when a preventive intervention was performed that was not indicated. A total of 300 charts were reviewed for the treatment group and 308 for the control group. The inter-rater agreement between the principal investigator and 2 other independent reviewers ranged between 93% and 98%. The median kappa statistic among 16 screening recommendations revealed very good agreement at 0.81 (all P <.042). Kappa could not be calculated for 4 of the screening recommendations, because the observed agreement of the 3 raters (100%) was equal to the expected agreement for those recommendations. All physicians were given a pretest measuring knowledge of the USPSTF recommendations at the beginning of the pre-intervention phase and a posttest near the end of the post-intervention phase. Finally, as physician learning and behavior were the focus of this study, patient adherence was not measured.
Data Analysis
Recommendations were organized into 4 categories: history, examination and laboratory, counseling, and prophylaxis. Tests of significance were calculated using a chi-square test that is designed to compare proportions among many independent samples.21 To control for statistical error, we used a Bonferroni adjusted P value. Because 20 screening recommendations were analyzed, a P value of .0025 (.05/20) was used as the cutoff for statistical significance.
Scores from the knowledge test were analyzed using a mixed factorial analysis of variance. The between-groups factor was group (treatment or control) and the within-subjects factor was test period (pretest or posttest). This analysis was conducted to see if the tool would lead to a group-by-test period interaction. Given the repeated use of the CART, it was thought that the treatment group might show a greater improvement in test scores than the control group.
Results
Impact of the CART
The Table 1 shows the percentages of patients who were appropriately screened during each period for the treatment and control groups. Significant increases were observed for all recommendations in the treatment group, except for occult blood and tetanus and diptheria booster. The mean absolute increase in the percentage of appropriately screened patients was 45%, 21%, and 15% for the recommendations in the history, examination and laboratory, and counseling categories, respectively. Removal of the annotated reminder during the post-intervention period brought the percentages of appropriate screenings down to baseline levels. Increases during the intervention phase were statistically significant for 16 of the 20 recommendations.
In the control group, the percentages of appropriately screened patients remained relatively stable. Although some variation is visible across the 3 study periods, there are no consistent trends among the recommendations, and none of the chi-square tests reached the Bonferroni level of significance. The Figure 1 shows the median percentage of patients screened during the baseline, intervention, and post-intervention periods. The average percentage of appropriate screenings for the clinicians using the CART increased during the intervention period and then returned to baseline levels after the tool was removed from use. The inverted V pattern repeatedly observed for the treatment group is not seen with the control group.
Screening Knowledge Scores
Only the main effect for the test period reached statistical significance. The mean test scores were 54.66 and 62.35 for clinicians at the pretest and posttest periods, respectively (F [1,27] = 23.89; P <.0005). The main effect for group was not statistically significant. The mean test scores for the treatment and control groups were 58.59 and 58.13, respectively (F [1,27] = 0.27; P <.608). The group-by-test period interaction also was not statistically significant (F [1,27] = 1.42, P=.244). Thus, although physicians had higher posttest knowledge scores than pretest scores, this increase was similar for treatment and control groups.
Discussion
CART use increased performance both in degree (percentage change) and in breadth (number of preventive services ordered appropriately). The primary outcome was the appropriateness of physician performance of preventive interventions. For example, if the patient is low-risk, then gonorrhea/chlamydia screening should not be performed, and appropriate physician behavior is to do no screening test. We did not give lectures or provide any other didactic preventive medicine training during the CART study period. But knowledge increased to a similar extent in both groups over that academic year. Nevertheless, the behavioral change seen in the treatment group was temporary; it returned to baseline in the post-intervention period. Therefore, it was the presence of the CART that improved physician behavior. The CART is comprehensive: It has prompts for up to 49 preventive recommendations. Physicians adhered significantly better to history and examination and laboratory recommendations when using the CART. The improvement in counseling was less impressive; physician behavior improved on only 4 of 6 counseling recommendations. It is likely that “form fatigue” was a factor, because the counseling recommendations were at the bottom of a page full of preventive recommendations. The difficulty of counseling, time constraints, and physician perception of the relative ineffectiveness of counseling may have also had an effect.
We believe that the primary reason for the success of the CART is immediate visual access to essential information. Computer reminders promote some behavioral change, but in other studies it was necessary to go to another computer screen for further information about a given test. Time pressure in the busy office makes this difficult. The CART gives physicians succinct information in a box adjacent to each preventive recommendation Table 1, including the recommended frequency, strength of recommendation, and appropriate groups to be screened for each intervention. Most other studies did not answer the question of whether increased knowledge over time was responsible for improved provider behavior. By removing the CART and reassessing physician behavior, we demonstrated that it was presence of the information in the CART format that was responsible for improving physician behavior.
Limitations
Our study has several limitations. We did not address the important issue of reminder systems or checklists. We only assessed the preventive services recommended by physicians during a single periodic health examination. We did not solve the problem of time constraint. It is difficult to cover every single preventive service recommendation in one visit. Our residents are less experienced and have more visit time than the average primary care provider. Finally, we only studied physician knowledge and behavior. We did not measure other important outcomes, such as patient satisfaction, adherence, morbidity, or mortality.
Conclusions
The nonspecific designation “tool” in the CART acronym implies potential future application of this concept for different subject matter and different formats. As we move inexorably toward universal electronic medical records, the CART can be adapted for electronic use. But whether on desktops, laptops, or hand-held computers, immediate visual access to information is essential. Combining the CART with computer reminder systems could yield significant improvement in patient outcomes when followed over time. Also, it would be interesting to incorporate the CART information into a computer-gathered patient history to create a partially completed history and physical examination form for provider use during the periodic health examination. This would save valuable office visit time. Many previous studies have focused on a small number of preventive services. We chose a comprehensive set of recommendations to more closely mimic the experience of primary care physicians who must cover many recommendations with patients. It is important that more studies use this approach. To maximize impact on the public health in an era of cost containment and to minimize physician error, we must enable physicians to efficiently apply a comprehensive set of preventive services in an appropriately focused manner. The CART is an effective step in the right direction.
Acknowledgments
We thank the Advocate Christ Hospital Med Fund for grant support. We would also like to thank Charles Warnell, MD, for his chart review work, and Carolyn Barsano for the literature review.
STUDY DESIGN: Using a randomized pretest/posttest control group design, we assigned physicians to the CART group or the control group, followed up prospectively, and evaluated for appropriate adherence to guidelines. The 3 age-specific versions of the CART annotated history and physical examination form contained up to 49 preventive services recommendations.
POPULATION: All resident physicians in a large family practice residency program were studied over the course of 1 academic year.
OUTCOMES MEASURED: We performed blinded chart reviews to assess the appropriateness of preventive services ordered by the physicians before the introduction of the CART, during its use, and after its removal. A multiple-choice test completed before and after the use of the CART forms assessed knowledge.
RESULTS: When the CART was used, the appropriateness of physician preventive behavior increased by 21% overall. The appropriateness of history, physical examination, and laboratory interventions increased by 33%. When the CART was removed, physician behavior returned to baseline (P≤.0025 for 16 of 20 interventions). No significant differences were observed in the control group over time. Knowledge increased during the study period for all physicians (P <.005) but did not differ significantly between the treatment and control groups (P=.608).
CONCLUSIONS: Use of the CART significantly improved physician performance in the appropriate delivery of preventive care.
Frame and Carlson1 published a critical review of periodic health screenings in The Journal of Family Practice in 1975. Such efforts, currently manifested in the United States Preventive Services Task Force (USPSTF), have had a profound impact on the practice of family physicians. Increased oversight of family physicians’ preventive practices by third parties and the increased medical sophistication of the public have fueled demand for preventive health services. Family physicians are experiencing increased accountability to their patients and communities for the effective delivery of preventive medical care.2-4
A number of systems have been developed to assist physicians with the implementation of evidence-based preventive services guidelines. Over the past 15 years, numerous researchers have studied the effects of interventions using flow charts,5-7 checklists,8 computer-based reminders,9-14 and educational programs15-19 for delivery of preventive services. Ornstein and colleagues10 found that the use of a computer-based tracking and reminder system for preventive services, together with a physician education program, modestly increased the frequency of counseling and screening services for patients. Similarly, Cheney and Ramsdell8 found that physicians who received an age- and sex-specific checklist in their patients’ medical charts performed significantly more age- and sex-appropriate preventive services than control physicians. But physician compliance with screening tools is often a problem.6 In the Cheney and Ramsdell study, only 39% of physicians used the available checklist to screen patients. Another study14 found a greater percentage of appropriate health screens by physicians using computerized reminders. However, many physicians did not make use of the available resource. Some studies found only minimal increases in screening rates with interventions such as educational seminars.19 One study5 used a simple flow sheet to analyze appropriate health service needs and noted that their results suggest that a flow sheet, although helpful, is not enough. Indeed, accomplishing sustained improvement in preventive services clinical outcomes is a complex process.20
We investigated the effect of combining a complete set of annotated recommendations with a history and physical examination form on the rate of appropriate application of preventive service recommendations. We suspected that busy practitioners would need preventive services guidelines available at the point of care to be able to adhere closely, so we created a form that is very user-friendly. We also studied the effect of removing the screening tool after the intervention, to evaluate whether the behavioral effect was due to education or to the presence of the chart tool, a question that has not been resolved in several previous studies.
Methods
Study Setting and Instrument
Our study was conducted between June 1997 and June 1998 at a community hospital continuity clinic. Age-appropriate screening history questions, age-specific reminders, and test frequency recommendations drawn from the 1996 USPSTF were integrated into a form for documenting history and physical examinations. This Comprehensive Annotated Reminder Tool (CART) contained 8 sections: history, screening history, physical examination, screening mnemonics, laboratory screens, prophylaxis, counseling, and the assessment/plan section. CART forms were developed for patients aged 19-39, 40-64, and 65 or older. Table w1,Table w2, Table w3 3* Each of these 3-page forms contained guidelines for up to 49 preventive interventions. Twenty recommendations were evaluated in this study. Because nursing has been primarily responsible for ensuring immunization compliance in our setting, we did not expect an effect and only evaluated one immunization.
Intervention
Resident physicians were randomly assigned to a treatment group that was exposed to the CART (n=15) and a control group that used existing blank history and physical examination forms (n=16). Resident physicians from both groups precepted patients with the same faculty physicians on a regular basis. Physicians in the control group used a different area of the clinic facility from the treatment group to avoid cross-use of the CART form. The 3 time periods of the study were: pre-intervention phase (3 months), intervention phase (6 months), and post-intervention phase (3 months). The pre-intervention phase allowed for the establishment of a baseline measurement of preventive medicine practiced by physicians in both groups. In the intervention phase, the CART forms were placed on all charts of new patients seen by physicians in the experimental group. The forms were not introduced by any formal instruction. No preventive medicine lectures were given to physicians in either group during the course of the study. Finally, in the post-intervention phase the CART was no longer placed on charts.
Measures
Blinded chart reviews were performed by the principal investigator and 2 other independent reviewers on randomly selected new patients. The physicians were expected to complete a history and physical examination on each new clinic patient within the first 3 visits. Chart reviews occurred before use, during use, and after removal of the CART. They evaluated charts for screening appropriateness (not the absolute frequency of interventions). No credit was given when a preventive intervention was performed that was not indicated. A total of 300 charts were reviewed for the treatment group and 308 for the control group. The inter-rater agreement between the principal investigator and 2 other independent reviewers ranged between 93% and 98%. The median kappa statistic among 16 screening recommendations revealed very good agreement at 0.81 (all P <.042). Kappa could not be calculated for 4 of the screening recommendations, because the observed agreement of the 3 raters (100%) was equal to the expected agreement for those recommendations. All physicians were given a pretest measuring knowledge of the USPSTF recommendations at the beginning of the pre-intervention phase and a posttest near the end of the post-intervention phase. Finally, as physician learning and behavior were the focus of this study, patient adherence was not measured.
Data Analysis
Recommendations were organized into 4 categories: history, examination and laboratory, counseling, and prophylaxis. Tests of significance were calculated using a chi-square test that is designed to compare proportions among many independent samples.21 To control for statistical error, we used a Bonferroni adjusted P value. Because 20 screening recommendations were analyzed, a P value of .0025 (.05/20) was used as the cutoff for statistical significance.
Scores from the knowledge test were analyzed using a mixed factorial analysis of variance. The between-groups factor was group (treatment or control) and the within-subjects factor was test period (pretest or posttest). This analysis was conducted to see if the tool would lead to a group-by-test period interaction. Given the repeated use of the CART, it was thought that the treatment group might show a greater improvement in test scores than the control group.
Results
Impact of the CART
The Table 1 shows the percentages of patients who were appropriately screened during each period for the treatment and control groups. Significant increases were observed for all recommendations in the treatment group, except for occult blood and tetanus and diptheria booster. The mean absolute increase in the percentage of appropriately screened patients was 45%, 21%, and 15% for the recommendations in the history, examination and laboratory, and counseling categories, respectively. Removal of the annotated reminder during the post-intervention period brought the percentages of appropriate screenings down to baseline levels. Increases during the intervention phase were statistically significant for 16 of the 20 recommendations.
In the control group, the percentages of appropriately screened patients remained relatively stable. Although some variation is visible across the 3 study periods, there are no consistent trends among the recommendations, and none of the chi-square tests reached the Bonferroni level of significance. The Figure 1 shows the median percentage of patients screened during the baseline, intervention, and post-intervention periods. The average percentage of appropriate screenings for the clinicians using the CART increased during the intervention period and then returned to baseline levels after the tool was removed from use. The inverted V pattern repeatedly observed for the treatment group is not seen with the control group.
Screening Knowledge Scores
Only the main effect for the test period reached statistical significance. The mean test scores were 54.66 and 62.35 for clinicians at the pretest and posttest periods, respectively (F [1,27] = 23.89; P <.0005). The main effect for group was not statistically significant. The mean test scores for the treatment and control groups were 58.59 and 58.13, respectively (F [1,27] = 0.27; P <.608). The group-by-test period interaction also was not statistically significant (F [1,27] = 1.42, P=.244). Thus, although physicians had higher posttest knowledge scores than pretest scores, this increase was similar for treatment and control groups.
Discussion
CART use increased performance both in degree (percentage change) and in breadth (number of preventive services ordered appropriately). The primary outcome was the appropriateness of physician performance of preventive interventions. For example, if the patient is low-risk, then gonorrhea/chlamydia screening should not be performed, and appropriate physician behavior is to do no screening test. We did not give lectures or provide any other didactic preventive medicine training during the CART study period. But knowledge increased to a similar extent in both groups over that academic year. Nevertheless, the behavioral change seen in the treatment group was temporary; it returned to baseline in the post-intervention period. Therefore, it was the presence of the CART that improved physician behavior. The CART is comprehensive: It has prompts for up to 49 preventive recommendations. Physicians adhered significantly better to history and examination and laboratory recommendations when using the CART. The improvement in counseling was less impressive; physician behavior improved on only 4 of 6 counseling recommendations. It is likely that “form fatigue” was a factor, because the counseling recommendations were at the bottom of a page full of preventive recommendations. The difficulty of counseling, time constraints, and physician perception of the relative ineffectiveness of counseling may have also had an effect.
We believe that the primary reason for the success of the CART is immediate visual access to essential information. Computer reminders promote some behavioral change, but in other studies it was necessary to go to another computer screen for further information about a given test. Time pressure in the busy office makes this difficult. The CART gives physicians succinct information in a box adjacent to each preventive recommendation Table 1, including the recommended frequency, strength of recommendation, and appropriate groups to be screened for each intervention. Most other studies did not answer the question of whether increased knowledge over time was responsible for improved provider behavior. By removing the CART and reassessing physician behavior, we demonstrated that it was presence of the information in the CART format that was responsible for improving physician behavior.
Limitations
Our study has several limitations. We did not address the important issue of reminder systems or checklists. We only assessed the preventive services recommended by physicians during a single periodic health examination. We did not solve the problem of time constraint. It is difficult to cover every single preventive service recommendation in one visit. Our residents are less experienced and have more visit time than the average primary care provider. Finally, we only studied physician knowledge and behavior. We did not measure other important outcomes, such as patient satisfaction, adherence, morbidity, or mortality.
Conclusions
The nonspecific designation “tool” in the CART acronym implies potential future application of this concept for different subject matter and different formats. As we move inexorably toward universal electronic medical records, the CART can be adapted for electronic use. But whether on desktops, laptops, or hand-held computers, immediate visual access to information is essential. Combining the CART with computer reminder systems could yield significant improvement in patient outcomes when followed over time. Also, it would be interesting to incorporate the CART information into a computer-gathered patient history to create a partially completed history and physical examination form for provider use during the periodic health examination. This would save valuable office visit time. Many previous studies have focused on a small number of preventive services. We chose a comprehensive set of recommendations to more closely mimic the experience of primary care physicians who must cover many recommendations with patients. It is important that more studies use this approach. To maximize impact on the public health in an era of cost containment and to minimize physician error, we must enable physicians to efficiently apply a comprehensive set of preventive services in an appropriately focused manner. The CART is an effective step in the right direction.
Acknowledgments
We thank the Advocate Christ Hospital Med Fund for grant support. We would also like to thank Charles Warnell, MD, for his chart review work, and Carolyn Barsano for the literature review.
1. Frame PS, Carlson SJ. A critical review of periodic health screening using specific screening criteria: part 1. Selected diseases of respiratory, cardiovascular, and central nervous system. J Fam Pract 1975;2:29-36.
2. Sox HC, Jr. Preventive health services in adults. New Engl J Med 1994;330:1589-95.
3. Bergman-Evans B, Walker SN. The prevalence of clinical preventive services utilization by older women. Nurse Pract 1996;21:88,90,99-100.
4. Woo B, Woo B, Cook EF, Weidberg M, Goldman L. Screening procedures in the asymptomatic adult. J Am Med Assoc 1985;254:1480-84.
5. Keim DB, Gomez CF, Wolf AMD. The level of preventive health care in an internal medicine residency clinic: still only and ounce of prevention? South Med J 1998;91:550-54.
6. Frame PS, Kowulich BA, Llewellyn AM. Improving physician compliance with a health maintenance protocol. J Fam Pract 1984;19:341-44.
7. Madlon-Kay DJ. Use of a structured encounter form to improve well-child care documentation. Arch Fam Med 1998;7:480-83.
8. Cheney C, Ramsdell JW. Effect of medical records checklists on implementation of periodic health measures. Am J Med 1987;83:129-36.
10. Ornstein SM, Garr DR, Jenkins RG, Musham CM, Hamadeh G, Lancaster C. Implementation and evaluation of a computer-based preventive services system. Fam Med 1995;27:260-66.
11. Ornstein SM, Garr DR, Jenkins RG, Rust PF, Arnon A. Computer-generated physician and patient reminders: tools to improve population adherence to selected preventive services. J Fam Pract 1991;32:82-90.
12. Litzelman DK, Dittus RS, Miller ME, Tierney WM. Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols. J Gen Intern Med 1993;8:311-17.
13. Turner BJ, Day SC, Borenstein B. A controlled trial to improve delivery of preventive health care: physician or patient reminders? J Gen Intern Med 1989;4:403-09.
14. Overhage JM, Tierney WM, McDonald CJ. Computer reminders to implement preventive care guidelines for hospitalized patients. Arch Intern Med 1996;156:1551-56.
15. Tape TG, Campbell JR. Computerized medical records and preventive health care: success depends on many factors. Am J Med 1993;94:619-25.
16. Borum ML. Medical residents colorectal cancer screening may be dependent on ambulatory care education. Dig Dis Sci 1997;42:1176-78.
17. Comninellis NB, Harper DM. Does comprehensive preventive medicine training enhance clinical prevention? Fam Med 1997;29:112-14.
18. Geiger WJ, Neuberger MJ, Bell GC. Implementing the US preventive services guidelines in a family practice residency. Fam Med 1993;25:447-51.
19. Leshan LA, Fitzsimmons M, Marbella A, Gottlieb M. Increasing clinical prevention efforts in a family practice residency program through CQI methods. J Qual Improv 1997;23:391-400.
20. Ward J, Sanson-Fisher R. Does a 3-day workshop for family medicine trainees improve preventive care? A randomized control trial. Prev Med 1996;25:741-47.
21. McBride P, Underbakke G, Plane MB, et al. Improving prevention systems in primary care practices: the health education and research trial (HEART). J Fam Pract 2000;49:115-25.
22. Fleiss JL. Statistical methods for rates and proportions. New York, NY: John Wiley & Sons; 1973.
1. Frame PS, Carlson SJ. A critical review of periodic health screening using specific screening criteria: part 1. Selected diseases of respiratory, cardiovascular, and central nervous system. J Fam Pract 1975;2:29-36.
2. Sox HC, Jr. Preventive health services in adults. New Engl J Med 1994;330:1589-95.
3. Bergman-Evans B, Walker SN. The prevalence of clinical preventive services utilization by older women. Nurse Pract 1996;21:88,90,99-100.
4. Woo B, Woo B, Cook EF, Weidberg M, Goldman L. Screening procedures in the asymptomatic adult. J Am Med Assoc 1985;254:1480-84.
5. Keim DB, Gomez CF, Wolf AMD. The level of preventive health care in an internal medicine residency clinic: still only and ounce of prevention? South Med J 1998;91:550-54.
6. Frame PS, Kowulich BA, Llewellyn AM. Improving physician compliance with a health maintenance protocol. J Fam Pract 1984;19:341-44.
7. Madlon-Kay DJ. Use of a structured encounter form to improve well-child care documentation. Arch Fam Med 1998;7:480-83.
8. Cheney C, Ramsdell JW. Effect of medical records checklists on implementation of periodic health measures. Am J Med 1987;83:129-36.
10. Ornstein SM, Garr DR, Jenkins RG, Musham CM, Hamadeh G, Lancaster C. Implementation and evaluation of a computer-based preventive services system. Fam Med 1995;27:260-66.
11. Ornstein SM, Garr DR, Jenkins RG, Rust PF, Arnon A. Computer-generated physician and patient reminders: tools to improve population adherence to selected preventive services. J Fam Pract 1991;32:82-90.
12. Litzelman DK, Dittus RS, Miller ME, Tierney WM. Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols. J Gen Intern Med 1993;8:311-17.
13. Turner BJ, Day SC, Borenstein B. A controlled trial to improve delivery of preventive health care: physician or patient reminders? J Gen Intern Med 1989;4:403-09.
14. Overhage JM, Tierney WM, McDonald CJ. Computer reminders to implement preventive care guidelines for hospitalized patients. Arch Intern Med 1996;156:1551-56.
15. Tape TG, Campbell JR. Computerized medical records and preventive health care: success depends on many factors. Am J Med 1993;94:619-25.
16. Borum ML. Medical residents colorectal cancer screening may be dependent on ambulatory care education. Dig Dis Sci 1997;42:1176-78.
17. Comninellis NB, Harper DM. Does comprehensive preventive medicine training enhance clinical prevention? Fam Med 1997;29:112-14.
18. Geiger WJ, Neuberger MJ, Bell GC. Implementing the US preventive services guidelines in a family practice residency. Fam Med 1993;25:447-51.
19. Leshan LA, Fitzsimmons M, Marbella A, Gottlieb M. Increasing clinical prevention efforts in a family practice residency program through CQI methods. J Qual Improv 1997;23:391-400.
20. Ward J, Sanson-Fisher R. Does a 3-day workshop for family medicine trainees improve preventive care? A randomized control trial. Prev Med 1996;25:741-47.
21. McBride P, Underbakke G, Plane MB, et al. Improving prevention systems in primary care practices: the health education and research trial (HEART). J Fam Pract 2000;49:115-25.
22. Fleiss JL. Statistical methods for rates and proportions. New York, NY: John Wiley & Sons; 1973.