Bruce N. Baskerville, MHA

In conducting research with community-based primary care practices it is often not feasible to randomize individual physicians to the treatment conditions. This is due to problems of potential contamination between intervention and control subjects in the same practice setting or because the success of the intervention demands that all physicians in the practice setting adhere to a guideline. As a result, the practice itself is randomized to the conditions.

The randomization of physicians in groups, rather than each individual separately, has important consequences for sample size, interpretation, and analysis.^1-3 It is argued that groups of physicians are likely to be heterogeneous,⁴ giving rise to a component of variation that one must take into account in the analysis and that one can control only by studying many groups of physicians rather than many physicians within each group.⁴

Randomizing physicians by cluster and then analyzing the data by physician or patient has the potential to introduce possible bias in the results. It has been noted that many studies randomized groups of health professionals (cluster randomization) but analyzed the results by physician, thus resulting in a possible overestimation of the significance of the observed effects (unit of analysis error).⁵ Divine and colleagues⁶ observed that 38 out of 54 studies of physicians’ patient care practices had not appropriately accounted for the clustered nature of the study data. Similarly, Simpson and coworkers⁷ found that only 4 out of 21 primary prevention trials included sample size calculations or discussions of power that allowed for clustering, while 12 out of 21 took clustering into account in the statistical analysis. When the effect size of the intervention is small to moderate, analyzing results by individual without adjusting for the cluster phenomena can lead to false conclusions about the significance of the effectiveness of the intervention. For example, Donner and Klar⁸ show that for the data of Murray and colleagues⁹ the P value would be .03 if the effect of clustering were ignored, while it was greater than .1 after adjusting for the effect of clustering.

Using baseline data from a successful randomized controlled trial of primary care practices in Southern Ontario, Canada,¹⁰ we will explain the intracluster correlation coefficient (ICC) in determining the required sample size of physicians. The ICC is a measure of variation within and between clusters of physicians. It is a measure of the clustering effect or the lack of independence among the physicians that make up the cluster. The smaller the ICC, the more likely the physicians in the cluster behave independently, and analysis at the level of the physician can proceed without significant adjustment to sample size. The higher the ICC, the more closely the measure quantifies class or group rather than the individual physician, and the effective sample size is decreased to the number of classes rather than the number of individuals. Our objective was to provide information on the cluster effect of measuring the performance of various preventive maneuvers between groups of physicians to enable other researchers in the area of primary care prevention to avoid errors.

Methods

As part of a larger clinical trial to improve preventive practice, we conducted a cross-sectional study to provide a point estimate of preventive performance in capitation primary care practices. We chose the preventive maneuvers from the Canadian Task Force on the Periodic Health Examination.¹¹ According to their classification system, there is randomized clinical trial evidence to support “A” level (highly recommended) maneuvers and cohort and case controlled studies to support “B” (recommended) maneuvers. The task force also reviewed the quality of evidence for maneuvers that should not be done and identified these as “D” level maneuvers. Eight A and B level recommendations and 5 D level recommendations were identified by a panel of practicing family physicians. Selection criteria included the need to represent a broad spectrum of preventive interventions for both men and women patients of all ages, and the need to address diseases that were clinically important. The 8 recommended and 5 inappropriate maneuvers chosen for our study are listed in Table 1.

This study was conducted in 72 community-based health service organizations (HSOs) in Ontario located at 100 different sites primarily in the Toronto, Hamilton, and London areas in the spring of 1997. The Ottawa Civic Hospital research ethics committee approved our study.

Data Collection

Practice and physician characteristics were collected using a self-administered questionnaire to which 96% of 108 participating physicians responded (Table 2 has the questionnaire items). Preventive performance at the physician and overall practice level was determined using a chart audit.

Chart Audit. Patient charts were eligible for inclusion in the medical audit if they were for patients who were aged 17 years or older on the date of last visit and had visited the HSO at least once in the 2 years before the audit. The variables collected from the charts included demographic and patient characteristics as well as indicators of performance of preventive maneuvers.

The chart auditors obtained a list of patients within an HSO practice group of physicians and then randomly selected charts using computer-generated random numbers. The patient list was either constructed by the auditors or by using the medical office record computer system. The list included all rostered and nonrostered patients. Unique chart numbers or numeric identifiers were assigned to each patient. The required number of charts was randomly selected from the sampling frame, the chart was pulled, and eligibility for inclusion was determined. The auditors proceeded to find charts at random from the sampling frame until they obtained 100 eligible charts per practice.

To verify the quality of the data entered from the 100 randomly selected charts and to measure the inter-rater reliability between auditors, 20% of each HSO’s audited charts were independently verified by another auditor. If coding discrepancies were found in more than 5 of 20 charts, the entire 100 charts were audited and verified again.

Data Analysis. Our analysis with SPSS software version 8.0 (SPSS Inc, Chicago, Ill) focused primarily on calculating the extent to which each preventive maneuver was being performed according to the recommendations of the Canadian Task Force on the Periodic Health Exam. An index of “up-to-datedness” was computed for each physician and practice as the number of A and B preventive measures done divided by the number of eligible A and B measures. In addition, an index called “inappropriateness” was computed in the same manner to represent the D measures.

Frequencies and descriptive statistics were generated on all variables, and each variable was checked for data entry errors and inappropriate or illogical responses. Means and standard deviations were computed for continuous variables and frequency distributions were computed for categorical variables, such as sex and age group. In addition, chi-square tests were used to compare the background characteristics of participating and nonparticipating HSO physicians. Ninety-five percent confidence intervals were calculated for the mean preventive indexes. Finally, kwas computed as a measure of reliability between the 2 chart auditors.

The ICC was calculated for sample cluster means¹² of up-to-datedness and inappropriateness. The practice characteristic data revealed that the mean cluster size in terms of number of physicians per practice was 2.8 with a variance of 3.6 and a total of 106 physicians across 46 practices. To determine the between-subjects (practices) variance (s_b²) and within-subjects (practices) variance (s_w 2) for the ICC calculation, the one-way analysis of variance (ANOVA) procedure was run on both measures (up-to-datedness and inappropriateness) as well as each of the preventive maneuvers separately.¹³ ICC (D) was computed from the F statistics of the one-way ANOVA and the adjusted cluster size as follows: D = F-1/F+n₀-1 where n₀ is the mean practice size ([2.8] - [practice variance (3.6)/106 physicians]).

Finally, using the formula by Donner and coworkers¹⁴ and the ICC, the sample size for comparing 2 independent groups allowing for clustering for both up-to-datedness and inappropriateness was determined. The formula is: n = 2(Z_−/2+Z₀₂)²F²[1+(x_c-1)D]/*² where n is the per group sample size; Z_−/2= 1.96 and Z₀₂= 0.84 are the standard normal percentiles for the type I and type II error rates at 0.05 and 0.20, respectively; Fis the standard deviation in the outcome variable; * is the expected difference between the two means; x_cis the average cluster size; and D is the ICC. The sample size calculations were based on an expected difference of 0.09 between groups (with 80% power and 5% significance) and a standard deviation of 0.10. Table 3 shows the effect on per-group sample size for varying ICC values and average cluster sizes.

Results

A total of 46 HSOs were recruited out of a possible 100 sites, for a response rate of 46% at baseline. The response rate to the physician questionnaire was 98% (106 of 108). Physicians in practices that agreed to participate differed significantly from those who did not. Participating physicians were younger, having graduated in 1977 on average compared with 1971 (t=4.58 [df=191], P<.001) and were more likely to be women, 30.4% compared with 9.9% for nonparticipating physicians (c²=11.09 [df=1, N=193], P=.001). Table 2 provides descriptive information on practice and physician characteristics. Five practices of 46 needed to have the entire 100 charts re-audited. Final concordance between the 2 auditors for each practice verification was 85% (k=.71).

The mean up-to-datedness score for the practices or the mean proportion of A and B maneuvers performed was 53.5% (95% confidence interval [CI], 51.0%-56.0%) and the mean inappropriateness score was 21.5% (95% CI, 18.1%-24.9%). In other words, on average, 53.5% of patients eligible for recommended preventive maneuvers received them and 21.5% of eligible patients received inappropriate preventive maneuvers.

Table 1 gives the practice mean square, the error mean square, the ICC, and the required sample size per group for the overall measures of up-to-datedness and inappropriateness as well as for 13 preventive maneuvers individually. For inappropriateness, there was more variability between practices than within practices among physicians, resulting in a larger practice mean square and a significant F statistic (P <.05). For up-to-datedness, the variability within practices among physicians was greater than the variability between practices, although not significantly so. Table 1 shows the intraclass correlation as 0.0365 for up-to-datedness and 0.1790 for inappropriateness. Inappropriateness scores were not normally distributed, and 2 physicians had scores greater than 0.60. However, with these extreme outliers removed, the ICC for inappropriateness remained high at 0.14.

The ICC ranges from 0.005 for blood pressure measurement to 0.66 for chest x-rays of smokers. The variability between and within group clusters is the same for blood pressure measurement. For chest x-rays of smokers the variability between clusters is very significant and within clusters it is small, indicating that some practice clusters perform a larger number of chest x-rays on smokers than other practices. However, the performance of chest x-rays was not normally distributed, with 79% of physicians not performing them and one solo physician with an extreme score of 0.53. With this extreme outlier removed the ICC for chest x-rays was 0.25, with a mean square between practices of 0.0024 and a mean square within practices of 0.0012 (P<.01). Table 1 shows the effect on sample size for analysis at the level of the physician as the ICC varies.

Discussion

Statistical theory points to the consequences of cluster randomization as a reduction in effective sample size. This occurs because the individuals within a cluster cannot be regarded as independent. The precise effect of cluster randomization on sample size requirements depends on both the size of the cluster and the degree of within-cluster dependence as measured by ICC.² Cluster randomized trials are increasingly being used in health services research particularly for evaluating interventions involving organizational changes when it is not feasible to randomize at the level of the individual. Cluster randomization at the level of the practice minimizes the potential for contamination between treatment and control groups. However, the statistical power of a cluster randomized trial when the unit of randomization is the practice and the unit of analysis is the health professional can be greatly reduced in comparison to an individually randomized trial.¹⁵

To preserve power the researcher should, whenever possible, ensure that the unit of randomization and the unit of analysis are the same.¹⁶ In this manner standard statistical tests can be used. Often this is not possible given secondary research questions that may be targeted to the health professionals within the practice and not the practice as a whole. If data are analyzed at the level of the individual and not at the level of the cluster (in effect ignoring the clustering effect), then there is a strong possibility that P values will be artificially extreme and confidence intervals will be overly narrow, increasing the chances of spuriously significant findings and misleading conclusions.¹⁵ When using the individual physician as the unit of analysis, one must take into account the correlation between responses of individuals within the same cluster. For continuous outcome variables that are normally distributed, a mixed-effects analysis of variance (or covariance) is appropriate, with clusters nested within the comparison groups.¹⁷ For dichotomous variables, Donner and Klar suggest that an adjusted chi-square test be used.⁸ Although we focus on the issue of clustering for study designs using random allocation, the issue of clustering is also apparent in cross-sectional and cohort studies, where the practice-level and/or physician-level factors may have an impact on patient-level data. Researchers need to be aware of the possibility of intracluster correlation and the implications for analysis in these studies as well.¹⁸

In the example presented, the ICC for the outcome measure “up-to-datedness” was approximately 0.04 in contrast to the ICC for inappropriateness, which was 0.18. The required sample size per group for the outcome measure “up-to-datedness” would be 21 physicians compared with inappropriateness, where the sample size would be 25 per group. In contrast, if the study dealt with improving smoking cessation counseling or reducing chest x-rays in smokers, the sample size would be 27 or 42 physicians per group. Treating the unit of analysis and the unit of randomization the same would require only 19 physicians per group.

Campbell and colleagues¹⁹ looked at a number of primary and secondary care study data sets and found that ICCs for measures in primary care were generally between 0.05 and 0.15. In contrast, in this study the ICCs ranged from 0.005 to 0.66, depending on the measure. The difference in ICC between measures and across studies is interesting, and we can only speculate why some measures show more interdependence. It is possible that inappropriateness taps phenomena such as policies at the practice level which physicians can not easily influence, while up-to-datedness may help explain how physicians even when working in the same practice setting behave independently when it comes to delivering recommended preventive care. It is important to be aware and not to assume that because one measure may show independence that all measures under study show the same independence. For example, blood pressure measurement and urine proteinuria screening are different in terms of ICC. Differences between outcome measures should be taken into account when calculating required sample size and in statistical analysis when the unit of randomization and analysis are not the same.

Limitations

There are 2 limitations with this research. First, analysis of respondents and nonrespondents to the recruitment effort showed that the study participants were more likely to be younger and women. This would imply that our findings may not be generalizable to the HSO population as a whole. Second, the measures of preventive performance were based on a chart audit and as a consequence are susceptible to the potential problems associated with chart documentation. A low level of preventive performance does not necessarily mean that prevention is not being practiced or that it is being performed inconsistently within a group practice. It may indicate that a less sophisticated documentation process is being used.

Conclusion

Physicians clustered together in the same practice do not necessarily perform the delivery of preventive services equally. As demonstrated by the measure “up-to-datedness,” there is relatively little correlation among physicians working together for performance of many preventive maneuvers. For some maneuvers, most notably those that may be automatically performed as part of practice policy, there is modest correlation among physicians who work together. We hope that these findings assist other researchers in their decision making around the need to adjust sample sizes for the effect of clustering.

In conducting research with community-based primary care practices it is often not feasible to randomize individual physicians to the treatment conditions. This is due to problems of potential contamination between intervention and control subjects in the same practice setting or because the success of the intervention demands that all physicians in the practice setting adhere to a guideline. As a result, the practice itself is randomized to the conditions.

The randomization of physicians in groups, rather than each individual separately, has important consequences for sample size, interpretation, and analysis.^1-3 It is argued that groups of physicians are likely to be heterogeneous,⁴ giving rise to a component of variation that one must take into account in the analysis and that one can control only by studying many groups of physicians rather than many physicians within each group.⁴

Randomizing physicians by cluster and then analyzing the data by physician or patient has the potential to introduce possible bias in the results. It has been noted that many studies randomized groups of health professionals (cluster randomization) but analyzed the results by physician, thus resulting in a possible overestimation of the significance of the observed effects (unit of analysis error).⁵ Divine and colleagues⁶ observed that 38 out of 54 studies of physicians’ patient care practices had not appropriately accounted for the clustered nature of the study data. Similarly, Simpson and coworkers⁷ found that only 4 out of 21 primary prevention trials included sample size calculations or discussions of power that allowed for clustering, while 12 out of 21 took clustering into account in the statistical analysis. When the effect size of the intervention is small to moderate, analyzing results by individual without adjusting for the cluster phenomena can lead to false conclusions about the significance of the effectiveness of the intervention. For example, Donner and Klar⁸ show that for the data of Murray and colleagues⁹ the P value would be .03 if the effect of clustering were ignored, while it was greater than .1 after adjusting for the effect of clustering.

Using baseline data from a successful randomized controlled trial of primary care practices in Southern Ontario, Canada,¹⁰ we will explain the intracluster correlation coefficient (ICC) in determining the required sample size of physicians. The ICC is a measure of variation within and between clusters of physicians. It is a measure of the clustering effect or the lack of independence among the physicians that make up the cluster. The smaller the ICC, the more likely the physicians in the cluster behave independently, and analysis at the level of the physician can proceed without significant adjustment to sample size. The higher the ICC, the more closely the measure quantifies class or group rather than the individual physician, and the effective sample size is decreased to the number of classes rather than the number of individuals. Our objective was to provide information on the cluster effect of measuring the performance of various preventive maneuvers between groups of physicians to enable other researchers in the area of primary care prevention to avoid errors.

Methods

As part of a larger clinical trial to improve preventive practice, we conducted a cross-sectional study to provide a point estimate of preventive performance in capitation primary care practices. We chose the preventive maneuvers from the Canadian Task Force on the Periodic Health Examination.¹¹ According to their classification system, there is randomized clinical trial evidence to support “A” level (highly recommended) maneuvers and cohort and case controlled studies to support “B” (recommended) maneuvers. The task force also reviewed the quality of evidence for maneuvers that should not be done and identified these as “D” level maneuvers. Eight A and B level recommendations and 5 D level recommendations were identified by a panel of practicing family physicians. Selection criteria included the need to represent a broad spectrum of preventive interventions for both men and women patients of all ages, and the need to address diseases that were clinically important. The 8 recommended and 5 inappropriate maneuvers chosen for our study are listed in Table 1.

This study was conducted in 72 community-based health service organizations (HSOs) in Ontario located at 100 different sites primarily in the Toronto, Hamilton, and London areas in the spring of 1997. The Ottawa Civic Hospital research ethics committee approved our study.

Data Collection

Practice and physician characteristics were collected using a self-administered questionnaire to which 96% of 108 participating physicians responded (Table 2 has the questionnaire items). Preventive performance at the physician and overall practice level was determined using a chart audit.

Chart Audit. Patient charts were eligible for inclusion in the medical audit if they were for patients who were aged 17 years or older on the date of last visit and had visited the HSO at least once in the 2 years before the audit. The variables collected from the charts included demographic and patient characteristics as well as indicators of performance of preventive maneuvers.

The chart auditors obtained a list of patients within an HSO practice group of physicians and then randomly selected charts using computer-generated random numbers. The patient list was either constructed by the auditors or by using the medical office record computer system. The list included all rostered and nonrostered patients. Unique chart numbers or numeric identifiers were assigned to each patient. The required number of charts was randomly selected from the sampling frame, the chart was pulled, and eligibility for inclusion was determined. The auditors proceeded to find charts at random from the sampling frame until they obtained 100 eligible charts per practice.

To verify the quality of the data entered from the 100 randomly selected charts and to measure the inter-rater reliability between auditors, 20% of each HSO’s audited charts were independently verified by another auditor. If coding discrepancies were found in more than 5 of 20 charts, the entire 100 charts were audited and verified again.

Data Analysis. Our analysis with SPSS software version 8.0 (SPSS Inc, Chicago, Ill) focused primarily on calculating the extent to which each preventive maneuver was being performed according to the recommendations of the Canadian Task Force on the Periodic Health Exam. An index of “up-to-datedness” was computed for each physician and practice as the number of A and B preventive measures done divided by the number of eligible A and B measures. In addition, an index called “inappropriateness” was computed in the same manner to represent the D measures.

Frequencies and descriptive statistics were generated on all variables, and each variable was checked for data entry errors and inappropriate or illogical responses. Means and standard deviations were computed for continuous variables and frequency distributions were computed for categorical variables, such as sex and age group. In addition, chi-square tests were used to compare the background characteristics of participating and nonparticipating HSO physicians. Ninety-five percent confidence intervals were calculated for the mean preventive indexes. Finally, kwas computed as a measure of reliability between the 2 chart auditors.

The ICC was calculated for sample cluster means¹² of up-to-datedness and inappropriateness. The practice characteristic data revealed that the mean cluster size in terms of number of physicians per practice was 2.8 with a variance of 3.6 and a total of 106 physicians across 46 practices. To determine the between-subjects (practices) variance (s_b²) and within-subjects (practices) variance (s_w 2) for the ICC calculation, the one-way analysis of variance (ANOVA) procedure was run on both measures (up-to-datedness and inappropriateness) as well as each of the preventive maneuvers separately.¹³ ICC (D) was computed from the F statistics of the one-way ANOVA and the adjusted cluster size as follows: D = F-1/F+n₀-1 where n₀ is the mean practice size ([2.8] - [practice variance (3.6)/106 physicians]).

Finally, using the formula by Donner and coworkers¹⁴ and the ICC, the sample size for comparing 2 independent groups allowing for clustering for both up-to-datedness and inappropriateness was determined. The formula is: n = 2(Z_−/2+Z₀₂)²F²[1+(x_c-1)D]/*² where n is the per group sample size; Z_−/2= 1.96 and Z₀₂= 0.84 are the standard normal percentiles for the type I and type II error rates at 0.05 and 0.20, respectively; Fis the standard deviation in the outcome variable; * is the expected difference between the two means; x_cis the average cluster size; and D is the ICC. The sample size calculations were based on an expected difference of 0.09 between groups (with 80% power and 5% significance) and a standard deviation of 0.10. Table 3 shows the effect on per-group sample size for varying ICC values and average cluster sizes.

Results

A total of 46 HSOs were recruited out of a possible 100 sites, for a response rate of 46% at baseline. The response rate to the physician questionnaire was 98% (106 of 108). Physicians in practices that agreed to participate differed significantly from those who did not. Participating physicians were younger, having graduated in 1977 on average compared with 1971 (t=4.58 [df=191], P<.001) and were more likely to be women, 30.4% compared with 9.9% for nonparticipating physicians (c²=11.09 [df=1, N=193], P=.001). Table 2 provides descriptive information on practice and physician characteristics. Five practices of 46 needed to have the entire 100 charts re-audited. Final concordance between the 2 auditors for each practice verification was 85% (k=.71).

The mean up-to-datedness score for the practices or the mean proportion of A and B maneuvers performed was 53.5% (95% confidence interval [CI], 51.0%-56.0%) and the mean inappropriateness score was 21.5% (95% CI, 18.1%-24.9%). In other words, on average, 53.5% of patients eligible for recommended preventive maneuvers received them and 21.5% of eligible patients received inappropriate preventive maneuvers.

Table 1 gives the practice mean square, the error mean square, the ICC, and the required sample size per group for the overall measures of up-to-datedness and inappropriateness as well as for 13 preventive maneuvers individually. For inappropriateness, there was more variability between practices than within practices among physicians, resulting in a larger practice mean square and a significant F statistic (P <.05). For up-to-datedness, the variability within practices among physicians was greater than the variability between practices, although not significantly so. Table 1 shows the intraclass correlation as 0.0365 for up-to-datedness and 0.1790 for inappropriateness. Inappropriateness scores were not normally distributed, and 2 physicians had scores greater than 0.60. However, with these extreme outliers removed, the ICC for inappropriateness remained high at 0.14.

The ICC ranges from 0.005 for blood pressure measurement to 0.66 for chest x-rays of smokers. The variability between and within group clusters is the same for blood pressure measurement. For chest x-rays of smokers the variability between clusters is very significant and within clusters it is small, indicating that some practice clusters perform a larger number of chest x-rays on smokers than other practices. However, the performance of chest x-rays was not normally distributed, with 79% of physicians not performing them and one solo physician with an extreme score of 0.53. With this extreme outlier removed the ICC for chest x-rays was 0.25, with a mean square between practices of 0.0024 and a mean square within practices of 0.0012 (P<.01). Table 1 shows the effect on sample size for analysis at the level of the physician as the ICC varies.

Discussion

Statistical theory points to the consequences of cluster randomization as a reduction in effective sample size. This occurs because the individuals within a cluster cannot be regarded as independent. The precise effect of cluster randomization on sample size requirements depends on both the size of the cluster and the degree of within-cluster dependence as measured by ICC.² Cluster randomized trials are increasingly being used in health services research particularly for evaluating interventions involving organizational changes when it is not feasible to randomize at the level of the individual. Cluster randomization at the level of the practice minimizes the potential for contamination between treatment and control groups. However, the statistical power of a cluster randomized trial when the unit of randomization is the practice and the unit of analysis is the health professional can be greatly reduced in comparison to an individually randomized trial.¹⁵

To preserve power the researcher should, whenever possible, ensure that the unit of randomization and the unit of analysis are the same.¹⁶ In this manner standard statistical tests can be used. Often this is not possible given secondary research questions that may be targeted to the health professionals within the practice and not the practice as a whole. If data are analyzed at the level of the individual and not at the level of the cluster (in effect ignoring the clustering effect), then there is a strong possibility that P values will be artificially extreme and confidence intervals will be overly narrow, increasing the chances of spuriously significant findings and misleading conclusions.¹⁵ When using the individual physician as the unit of analysis, one must take into account the correlation between responses of individuals within the same cluster. For continuous outcome variables that are normally distributed, a mixed-effects analysis of variance (or covariance) is appropriate, with clusters nested within the comparison groups.¹⁷ For dichotomous variables, Donner and Klar suggest that an adjusted chi-square test be used.⁸ Although we focus on the issue of clustering for study designs using random allocation, the issue of clustering is also apparent in cross-sectional and cohort studies, where the practice-level and/or physician-level factors may have an impact on patient-level data. Researchers need to be aware of the possibility of intracluster correlation and the implications for analysis in these studies as well.¹⁸

In the example presented, the ICC for the outcome measure “up-to-datedness” was approximately 0.04 in contrast to the ICC for inappropriateness, which was 0.18. The required sample size per group for the outcome measure “up-to-datedness” would be 21 physicians compared with inappropriateness, where the sample size would be 25 per group. In contrast, if the study dealt with improving smoking cessation counseling or reducing chest x-rays in smokers, the sample size would be 27 or 42 physicians per group. Treating the unit of analysis and the unit of randomization the same would require only 19 physicians per group.

Campbell and colleagues¹⁹ looked at a number of primary and secondary care study data sets and found that ICCs for measures in primary care were generally between 0.05 and 0.15. In contrast, in this study the ICCs ranged from 0.005 to 0.66, depending on the measure. The difference in ICC between measures and across studies is interesting, and we can only speculate why some measures show more interdependence. It is possible that inappropriateness taps phenomena such as policies at the practice level which physicians can not easily influence, while up-to-datedness may help explain how physicians even when working in the same practice setting behave independently when it comes to delivering recommended preventive care. It is important to be aware and not to assume that because one measure may show independence that all measures under study show the same independence. For example, blood pressure measurement and urine proteinuria screening are different in terms of ICC. Differences between outcome measures should be taken into account when calculating required sample size and in statistical analysis when the unit of randomization and analysis are not the same.

Limitations

There are 2 limitations with this research. First, analysis of respondents and nonrespondents to the recruitment effort showed that the study participants were more likely to be younger and women. This would imply that our findings may not be generalizable to the HSO population as a whole. Second, the measures of preventive performance were based on a chart audit and as a consequence are susceptible to the potential problems associated with chart documentation. A low level of preventive performance does not necessarily mean that prevention is not being practiced or that it is being performed inconsistently within a group practice. It may indicate that a less sophisticated documentation process is being used.

Conclusion

Physicians clustered together in the same practice do not necessarily perform the delivery of preventive services equally. As demonstrated by the measure “up-to-datedness,” there is relatively little correlation among physicians working together for performance of many preventive maneuvers. For some maneuvers, most notably those that may be automatically performed as part of practice policy, there is modest correlation among physicians who work together. We hope that these findings assist other researchers in their decision making around the need to adjust sample sizes for the effect of clustering.

In conducting research with community-based primary care practices it is often not feasible to randomize individual physicians to the treatment conditions. This is due to problems of potential contamination between intervention and control subjects in the same practice setting or because the success of the intervention demands that all physicians in the practice setting adhere to a guideline. As a result, the practice itself is randomized to the conditions.

The randomization of physicians in groups, rather than each individual separately, has important consequences for sample size, interpretation, and analysis.^1-3 It is argued that groups of physicians are likely to be heterogeneous,⁴ giving rise to a component of variation that one must take into account in the analysis and that one can control only by studying many groups of physicians rather than many physicians within each group.⁴

Randomizing physicians by cluster and then analyzing the data by physician or patient has the potential to introduce possible bias in the results. It has been noted that many studies randomized groups of health professionals (cluster randomization) but analyzed the results by physician, thus resulting in a possible overestimation of the significance of the observed effects (unit of analysis error).⁵ Divine and colleagues⁶ observed that 38 out of 54 studies of physicians’ patient care practices had not appropriately accounted for the clustered nature of the study data. Similarly, Simpson and coworkers⁷ found that only 4 out of 21 primary prevention trials included sample size calculations or discussions of power that allowed for clustering, while 12 out of 21 took clustering into account in the statistical analysis. When the effect size of the intervention is small to moderate, analyzing results by individual without adjusting for the cluster phenomena can lead to false conclusions about the significance of the effectiveness of the intervention. For example, Donner and Klar⁸ show that for the data of Murray and colleagues⁹ the P value would be .03 if the effect of clustering were ignored, while it was greater than .1 after adjusting for the effect of clustering.

Using baseline data from a successful randomized controlled trial of primary care practices in Southern Ontario, Canada,¹⁰ we will explain the intracluster correlation coefficient (ICC) in determining the required sample size of physicians. The ICC is a measure of variation within and between clusters of physicians. It is a measure of the clustering effect or the lack of independence among the physicians that make up the cluster. The smaller the ICC, the more likely the physicians in the cluster behave independently, and analysis at the level of the physician can proceed without significant adjustment to sample size. The higher the ICC, the more closely the measure quantifies class or group rather than the individual physician, and the effective sample size is decreased to the number of classes rather than the number of individuals. Our objective was to provide information on the cluster effect of measuring the performance of various preventive maneuvers between groups of physicians to enable other researchers in the area of primary care prevention to avoid errors.

Methods

As part of a larger clinical trial to improve preventive practice, we conducted a cross-sectional study to provide a point estimate of preventive performance in capitation primary care practices. We chose the preventive maneuvers from the Canadian Task Force on the Periodic Health Examination.¹¹ According to their classification system, there is randomized clinical trial evidence to support “A” level (highly recommended) maneuvers and cohort and case controlled studies to support “B” (recommended) maneuvers. The task force also reviewed the quality of evidence for maneuvers that should not be done and identified these as “D” level maneuvers. Eight A and B level recommendations and 5 D level recommendations were identified by a panel of practicing family physicians. Selection criteria included the need to represent a broad spectrum of preventive interventions for both men and women patients of all ages, and the need to address diseases that were clinically important. The 8 recommended and 5 inappropriate maneuvers chosen for our study are listed in Table 1.

This study was conducted in 72 community-based health service organizations (HSOs) in Ontario located at 100 different sites primarily in the Toronto, Hamilton, and London areas in the spring of 1997. The Ottawa Civic Hospital research ethics committee approved our study.

Data Collection

Practice and physician characteristics were collected using a self-administered questionnaire to which 96% of 108 participating physicians responded (Table 2 has the questionnaire items). Preventive performance at the physician and overall practice level was determined using a chart audit.

Chart Audit. Patient charts were eligible for inclusion in the medical audit if they were for patients who were aged 17 years or older on the date of last visit and had visited the HSO at least once in the 2 years before the audit. The variables collected from the charts included demographic and patient characteristics as well as indicators of performance of preventive maneuvers.

The chart auditors obtained a list of patients within an HSO practice group of physicians and then randomly selected charts using computer-generated random numbers. The patient list was either constructed by the auditors or by using the medical office record computer system. The list included all rostered and nonrostered patients. Unique chart numbers or numeric identifiers were assigned to each patient. The required number of charts was randomly selected from the sampling frame, the chart was pulled, and eligibility for inclusion was determined. The auditors proceeded to find charts at random from the sampling frame until they obtained 100 eligible charts per practice.

To verify the quality of the data entered from the 100 randomly selected charts and to measure the inter-rater reliability between auditors, 20% of each HSO’s audited charts were independently verified by another auditor. If coding discrepancies were found in more than 5 of 20 charts, the entire 100 charts were audited and verified again.

Data Analysis. Our analysis with SPSS software version 8.0 (SPSS Inc, Chicago, Ill) focused primarily on calculating the extent to which each preventive maneuver was being performed according to the recommendations of the Canadian Task Force on the Periodic Health Exam. An index of “up-to-datedness” was computed for each physician and practice as the number of A and B preventive measures done divided by the number of eligible A and B measures. In addition, an index called “inappropriateness” was computed in the same manner to represent the D measures.

Frequencies and descriptive statistics were generated on all variables, and each variable was checked for data entry errors and inappropriate or illogical responses. Means and standard deviations were computed for continuous variables and frequency distributions were computed for categorical variables, such as sex and age group. In addition, chi-square tests were used to compare the background characteristics of participating and nonparticipating HSO physicians. Ninety-five percent confidence intervals were calculated for the mean preventive indexes. Finally, kwas computed as a measure of reliability between the 2 chart auditors.

The ICC was calculated for sample cluster means¹² of up-to-datedness and inappropriateness. The practice characteristic data revealed that the mean cluster size in terms of number of physicians per practice was 2.8 with a variance of 3.6 and a total of 106 physicians across 46 practices. To determine the between-subjects (practices) variance (s_b²) and within-subjects (practices) variance (s_w 2) for the ICC calculation, the one-way analysis of variance (ANOVA) procedure was run on both measures (up-to-datedness and inappropriateness) as well as each of the preventive maneuvers separately.¹³ ICC (D) was computed from the F statistics of the one-way ANOVA and the adjusted cluster size as follows: D = F-1/F+n₀-1 where n₀ is the mean practice size ([2.8] - [practice variance (3.6)/106 physicians]).

Finally, using the formula by Donner and coworkers¹⁴ and the ICC, the sample size for comparing 2 independent groups allowing for clustering for both up-to-datedness and inappropriateness was determined. The formula is: n = 2(Z_−/2+Z₀₂)²F²[1+(x_c-1)D]/*² where n is the per group sample size; Z_−/2= 1.96 and Z₀₂= 0.84 are the standard normal percentiles for the type I and type II error rates at 0.05 and 0.20, respectively; Fis the standard deviation in the outcome variable; * is the expected difference between the two means; x_cis the average cluster size; and D is the ICC. The sample size calculations were based on an expected difference of 0.09 between groups (with 80% power and 5% significance) and a standard deviation of 0.10. Table 3 shows the effect on per-group sample size for varying ICC values and average cluster sizes.

Results

A total of 46 HSOs were recruited out of a possible 100 sites, for a response rate of 46% at baseline. The response rate to the physician questionnaire was 98% (106 of 108). Physicians in practices that agreed to participate differed significantly from those who did not. Participating physicians were younger, having graduated in 1977 on average compared with 1971 (t=4.58 [df=191], P<.001) and were more likely to be women, 30.4% compared with 9.9% for nonparticipating physicians (c²=11.09 [df=1, N=193], P=.001). Table 2 provides descriptive information on practice and physician characteristics. Five practices of 46 needed to have the entire 100 charts re-audited. Final concordance between the 2 auditors for each practice verification was 85% (k=.71).

The mean up-to-datedness score for the practices or the mean proportion of A and B maneuvers performed was 53.5% (95% confidence interval [CI], 51.0%-56.0%) and the mean inappropriateness score was 21.5% (95% CI, 18.1%-24.9%). In other words, on average, 53.5% of patients eligible for recommended preventive maneuvers received them and 21.5% of eligible patients received inappropriate preventive maneuvers.

Table 1 gives the practice mean square, the error mean square, the ICC, and the required sample size per group for the overall measures of up-to-datedness and inappropriateness as well as for 13 preventive maneuvers individually. For inappropriateness, there was more variability between practices than within practices among physicians, resulting in a larger practice mean square and a significant F statistic (P <.05). For up-to-datedness, the variability within practices among physicians was greater than the variability between practices, although not significantly so. Table 1 shows the intraclass correlation as 0.0365 for up-to-datedness and 0.1790 for inappropriateness. Inappropriateness scores were not normally distributed, and 2 physicians had scores greater than 0.60. However, with these extreme outliers removed, the ICC for inappropriateness remained high at 0.14.

The ICC ranges from 0.005 for blood pressure measurement to 0.66 for chest x-rays of smokers. The variability between and within group clusters is the same for blood pressure measurement. For chest x-rays of smokers the variability between clusters is very significant and within clusters it is small, indicating that some practice clusters perform a larger number of chest x-rays on smokers than other practices. However, the performance of chest x-rays was not normally distributed, with 79% of physicians not performing them and one solo physician with an extreme score of 0.53. With this extreme outlier removed the ICC for chest x-rays was 0.25, with a mean square between practices of 0.0024 and a mean square within practices of 0.0012 (P<.01). Table 1 shows the effect on sample size for analysis at the level of the physician as the ICC varies.

Discussion

Statistical theory points to the consequences of cluster randomization as a reduction in effective sample size. This occurs because the individuals within a cluster cannot be regarded as independent. The precise effect of cluster randomization on sample size requirements depends on both the size of the cluster and the degree of within-cluster dependence as measured by ICC.² Cluster randomized trials are increasingly being used in health services research particularly for evaluating interventions involving organizational changes when it is not feasible to randomize at the level of the individual. Cluster randomization at the level of the practice minimizes the potential for contamination between treatment and control groups. However, the statistical power of a cluster randomized trial when the unit of randomization is the practice and the unit of analysis is the health professional can be greatly reduced in comparison to an individually randomized trial.¹⁵

To preserve power the researcher should, whenever possible, ensure that the unit of randomization and the unit of analysis are the same.¹⁶ In this manner standard statistical tests can be used. Often this is not possible given secondary research questions that may be targeted to the health professionals within the practice and not the practice as a whole. If data are analyzed at the level of the individual and not at the level of the cluster (in effect ignoring the clustering effect), then there is a strong possibility that P values will be artificially extreme and confidence intervals will be overly narrow, increasing the chances of spuriously significant findings and misleading conclusions.¹⁵ When using the individual physician as the unit of analysis, one must take into account the correlation between responses of individuals within the same cluster. For continuous outcome variables that are normally distributed, a mixed-effects analysis of variance (or covariance) is appropriate, with clusters nested within the comparison groups.¹⁷ For dichotomous variables, Donner and Klar suggest that an adjusted chi-square test be used.⁸ Although we focus on the issue of clustering for study designs using random allocation, the issue of clustering is also apparent in cross-sectional and cohort studies, where the practice-level and/or physician-level factors may have an impact on patient-level data. Researchers need to be aware of the possibility of intracluster correlation and the implications for analysis in these studies as well.¹⁸

In the example presented, the ICC for the outcome measure “up-to-datedness” was approximately 0.04 in contrast to the ICC for inappropriateness, which was 0.18. The required sample size per group for the outcome measure “up-to-datedness” would be 21 physicians compared with inappropriateness, where the sample size would be 25 per group. In contrast, if the study dealt with improving smoking cessation counseling or reducing chest x-rays in smokers, the sample size would be 27 or 42 physicians per group. Treating the unit of analysis and the unit of randomization the same would require only 19 physicians per group.

Campbell and colleagues¹⁹ looked at a number of primary and secondary care study data sets and found that ICCs for measures in primary care were generally between 0.05 and 0.15. In contrast, in this study the ICCs ranged from 0.005 to 0.66, depending on the measure. The difference in ICC between measures and across studies is interesting, and we can only speculate why some measures show more interdependence. It is possible that inappropriateness taps phenomena such as policies at the practice level which physicians can not easily influence, while up-to-datedness may help explain how physicians even when working in the same practice setting behave independently when it comes to delivering recommended preventive care. It is important to be aware and not to assume that because one measure may show independence that all measures under study show the same independence. For example, blood pressure measurement and urine proteinuria screening are different in terms of ICC. Differences between outcome measures should be taken into account when calculating required sample size and in statistical analysis when the unit of randomization and analysis are not the same.

Limitations

There are 2 limitations with this research. First, analysis of respondents and nonrespondents to the recruitment effort showed that the study participants were more likely to be younger and women. This would imply that our findings may not be generalizable to the HSO population as a whole. Second, the measures of preventive performance were based on a chart audit and as a consequence are susceptible to the potential problems associated with chart documentation. A low level of preventive performance does not necessarily mean that prevention is not being practiced or that it is being performed inconsistently within a group practice. It may indicate that a less sophisticated documentation process is being used.

Conclusion

Physicians clustered together in the same practice do not necessarily perform the delivery of preventive services equally. As demonstrated by the measure “up-to-datedness,” there is relatively little correlation among physicians working together for performance of many preventive maneuvers. For some maneuvers, most notably those that may be automatically performed as part of practice policy, there is modest correlation among physicians who work together. We hope that these findings assist other researchers in their decision making around the need to adjust sample sizes for the effect of clustering.

A randomized controlled field trial of a multifaceted intervention to improve preventive care tailored to the needs of participating family practices was conducted in Southern Ontario and delivered by nurses trained in the facilitation of prevention. We focus on the process evaluation and complement the outcome evaluation¹ by describing how the program was implemented in the intervention practices.

Improving preventive performance is both important and necessary. There is substantial room to improve the rates of appropriate preventive practice.² The Canadian Task Force on the Periodic Health xamination^3,4 has established guidelines for the delivery of preventive care that are supported by clinical evidence as effective in decreasing the impact of disease. However, evidence-based guidelines are not self-implementing.^5-7 Changing physicians’ long-held patterns of behavior and the environments in which they work is complex and difficult. Unless the barriers to change can be overcome and actions taken to put preventive care guidelines into practice, evidence-based guideline development efforts will be wasted, and the quality of preventive care will not improve.⁸

Several reviews have focussed on the effectiveness of different interventions for implementing guidelines and improving care.^6,7,9-13 Multifaceted interventions employing trained individuals who meet with providers in their practice settings to provide information and assist the practice in implementing evidence-based guidelines have been shown to be more effective than single interventions.^11-14 Tailoring interventions to the requirements of the practice has also been proposed as important in supporting practice changes and in attaining more successful outcomes in preventive care performance compared with interventions that are fixed and lack this flexibility.^15-17

As important as knowing what interventions work to improve preventive care performance is understanding why they work. The techniques of process evaluation allow the investigator to determine the extent to which the intervention designed to change practice patterns was actually implemented as planned. Adequate documentation of process facilitates replication and fine-tuning of the intervention.

Intervention Description

Our study built on the work of Fullard and colleagues18 and used a tailored multifaceted approach to getting evidence into action by focusing on the educational, attitudinal, and organizational barriers to change and tailoring interventions to the needs of the practice.^17-24 The intervention employed 3 prevention facilitators (PFs) with both master’s degrees in community nursing and skills and previous experience in facilitation. Each PF had primary responsibility for up to 8 primary care practices with up to 6 physicians per practice.

The PFs underwent 30 weeks of intensive training before being assigned to randomly selected intervention practices. The training covered an orientation session, medical office computer systems, medical practice management, prevention in primary care, evidence-based medicine, and facilitation and audit skills development. Approximately 28 hours per week were spent in training and 7 hours per week in preparation and planning. Six of the 30 weeks of training were spent applying skills in a primary care office setting. Once in the field, they were instructed to offer 7 intervention strategies designed to change practice patterns and improve preventive care performance. The strategies were identified from reviews of the literature and constituted the multifaceted component of the intervention.^10,11 The PFs were permitted to tailor these strategies to the needs and unique circumstances of the practice. The strategies were: (1) audit and ongoing feedback, (2) consensus building, (3) opinion leaders and networking, (4) academic detailing and education materials, (5) reminder systems, (6) patient-mediated activities, and (7) patient education materials.

The PFs worked with all physicians and allied health staff in the practice. They provided management support to practices and followed a quality improvement framework similar to that proposed by Leininger and coworkers.²⁵ For each practice the PFs were to: (1) present baseline preventive performance rates, (2) facilitate the development of a practice policy for preventive care, (3) assist in setting goals and desirable levels of performance, (4) assist in the development of a written plan for implementing preventive care, (5) assist in the development and adaptation of tools and the strategies to implement the prevention plan, (6) facilitate meetings to assess progress and modify the plan if necessary, and (7) conduct chart audits to measure the impact of the changes made. The intervention period lasted 18 months and ended in December 1998.

The Figure is the program logic model describing each of the 7 intervention component strategies and the associated work activities, outputs, and short-term and long-term objectives associated with each component. It served as a framework for the evaluation of the intervention.^26-28 The logic model allowed us to look inside the black box of the intervention^29,30 by linking implementation activity to outcomes, and provided a framework to explore which elements worked and why.

Intervention Outcomes

The goal of the intervention was to increase the performance of 8 preventive maneuvers supported by evidence as appropriate and decrease the performance of 5 preventive maneuvers supported by evidence as inappropriate.¹ An absolute change over time of 11.51% in preventive care performance in favor of intervention practices was found (F=19.29 [df=1,43], P<.0001. In other words, the intervention practices improved preventive performance by 36% going from 31% of eligible patients having received preventive care to 43% while the control practices remained at 32%.¹

Methods

Research Questions

There were 2 objectives to our process evaluation: to document the extent to which the intervention was implemented with fidelity and to gain insight into how facilitation worked to improve preventive performance. The process evaluation was designed to answer questions concerning: (1) the time involved to deliver intervention components, (2) the quality of the delivery of intervention components, and (3) physician satisfaction with the intervention components. Quality was assessed by examining the scope or range of delivery of the intervention components and by analyzing the feedback received from practices on the usefulness of the intervention components.

Setting

The intervention arm of the trial included 22 practices with 54 physicians (Table 1). All health service organizations (HSOs) in Southwestern Ontario were approached to participate in the study. HSOs are primary care practices reimbursed primarily through capitation rather than fee for service. A total of 46 of the 100 primary care practices were recruited (response rate=46%). At follow-up only one intervention practice was lost, because the entire practice had moved. Intervention and control group practices did not differ significantly on any of the measured demographic characteristics (Table 2). Complete details on practice recruitment and attrition rates are published elsewhere.¹

The practices covered a geographic area where the greatest distance between any 2 practices was more than 600 kilometers. PFs were assigned practices within a specific region of this geographic area. They arranged times to visit and work with intervention practices and traveled by car between visits to practices. PFs worked independently at their residences and corresponded with the project team through electronic mail regularly and quarterly with scheduled meetings.

Data Collection Tools

Each intervention practice was visited regularly by the same nurse facilitator who documented her activities and progress on 2 structured forms known as the weekly activity sheet and the monthly narrative report. Weekly activity sheets noted the number of hours spent on both on-site and offsite activities. Monthly narrative reports provided detailed information on the number of visits to a practice, the activities within each practice, the outcomes of those activities, the number of participants in meetings, and the plan for the following month. The activities in the narrative reports were summarized by intervention component to provide a cumulative overview of all intervention activity within a practice.

Also during the intervention, semistructured telephone interviews of participating physicians were conducted by 2 physician members of the project team at 6 months and 17 months. Participating physicians were asked what they had been happy and unhappy with and their ideas of improvement. Close-ended questions measured overall satisfaction with the intervention. The interview at 17 months also asked physicians if they would agree to have a nurse facilitator continue to visit their practice if funding were found.

At the end of the intervention, the PFs conducted interviews with each of the physicians identified as the primary contact in the intervention practices to solicit feedback on their experience. Physicians in both the intervention and control arm were sent a questionnaire by mail to report any changes that had taken place in their practice over the preceding 18 months.

Analysis

Data were analyzed to address the 3 research questions for the process evaluation utilizing the Logic Model as the conceptual framework (Figure). To determine how often various intervention components were delivered, the total hours spent at each practice and the total number of contacts with each practice by intervention component were calculated from the PF activity sheets.

To determine intervention quality, triangulation³¹ was used to attain a complete understanding of the quality of implementation. Multiple data sources and analysis methods were used to reveal the underlying dimensions of quality. All data sources were reviewed and analyses were conducted independently by 2 members of the investigation team. The members of the team held a debriefing session to discuss their findings and seek consensus. First, the monthly narrative reports across intervention sites were summarized to qualitatively describe the type, breadth, and scope of activity for each intervention component. Second, the activity descriptions and open-ended interview responses were content analyzed³² and coded, and frequencies were generated. The goal of this analysis was to identify significant descriptions of which intervention elements worked well and which did not. Finally, intervention and control practices were compared with contingency tables, and a chi-square statistic was used to determine differences on questionnaire responses concerning practice changes over the period of the intervention.

To determine physician satisfaction, open-ended satisfaction survey responses were coded and frequencies generated for ratings of overall satisfaction with the performance of the PF and the intervention.

Results

PF Program Implementation

Table 3 shows the number of hours spent on project activities during the period of the intervention. The PFs spent the largest proportion of their time (28%) on administrative duties, such as team meetings, telephone calls, internal reporting, preparing the project newsletter, coordinating networking conferences for intervention practices, photocopying, and filing. Sixteen percent of the PFs’ time was spent on-site facilitating changes to improve preventive care in the practice. Travel accounted for an average of 12% of the PFs’ time, although this varied depending on the distance to the practices.

Table 4 provides information on the number of contacts and hours spent on-site for each component of the intervention. On average, each intervention practice was contacted 33 times by a PF, with each visit lasting an average of 1 hour and 45 minutes. The most frequent forms of contact concerned developing reminder systems, conducting chart audits and providing feedback preventive care performance, and working to achieve consensus on the adoption of preventive care strategies. Both academic detailing to physicians and supplying critically appraised patient education materials averaged approximately 20 minutes but involved a great deal of preparation time. Few practices were interested in posters in the waiting room or a patient newsletter on prevention, so fewer contacts were made for those components.

Quality of Implementation

To assess quality, the frequency of each component of the intervention was tallied, physician feedback on the usefulness of intervention components was summarized, and self-reported practice changes between intervention and control physicians was reported.

Intervention Scope

Audit and Feedback. All 22 intervention practices received a presentation by the PF on the initial audit results to raise awareness of preventive care practice patterns. This was usually done in a kick-off meeting involving both physicians and nurses and often required more than one presentation to cover the various staff in the practice. Twenty practices requested subsequent analyses of data to follow their rates of performance. In addition, 18 practices requested audits of their charts for specific maneuvers, such as influenza vaccination and mammography

Consensus Building. All practices were involved in meetings with the PF to identify opportunities for improvement, assess needs, and select priority areas and strategies for improving preventive care performance. Interviews were conducted with nurses and other staff to promote their role in preventive care delivery.

Academic Detailing. Twenty-one out of 22 sites agreed to receive and discuss critically appraised evidence for the preventive maneuvers under study, and some requested similar information on other preventive areas, such as cholesterol and osteoporosis.

Reminder Systems. All of the intervention sites implemented some form of reminder system. Eighteen sites implemented a preventive care flow sheet; 2 sites used a chart stamp; and 2 sites implemented a computerized reminder system. Nineteen sites developed recall initiatives for flu vaccine, mammography, and Papanicolaou tests. Seventeen sites implemented chart stickers for smoking counseling or mammography.

Opinion Leaders. All sites received copies of the PF project newsletter that contained articles by influential individuals describing the importance of preventive care and descriptions of colleagues’ preventive care implementation efforts. Most practices attended a workshop that included an influential keynote speaker, and 27% of the participating physicians shared their knowledge about preventive care through publishing in the newsletter and/or public speaking.

Patient Education. All sites were provided with patient education materials from credible sources on request, and all received a binder of patient education materials constructed specifically to contain materials on the appropriate preventive maneuvers under study. The binders were regularly updated.

Patient Mediated. Posters designed to prompt patients to ask about folic acid, flu vaccine, and mammography were offered to all sites. Thirteen sites implemented a patient consent form for prostate-specific antigen (PSA) testing. Eight sites received preventive care diaries for patients. Five sites had a prevention newsletter for patients. Four sites agreed to pilot a health risk appraisal software program.

Physician Feedback

At the end of the intervention the facilitator asked physicians about their experience, including what was most and least useful to them. Table 5 provides a summary of the content analysis of physician responses.

Audit and feedback, both initially and subsequently, comprised the component most frequently considered to be important in creating change. Almost as often, the preventive care flow sheet was identified as useful. The facilitator sessions designed to seek consensus on preventive care guidelines and strategies for implementation were also appreciated.

Several physicians did not agree with the evidence on PSA testing. Others did not feel that counseling for folic acid was a priority. Some found the patient education binder cumbersome, and others found the sticker system for tobacco counseling unwieldy. Thus, both were underused. Two physicians noted that the preventive care wall chart was not helpful.

Physician Self-Reported Practice Changes

Eighty-six percent (93/108) of the intervention and control physicians responded to a questionnaire at the end of the study. Due to sample size, statistical power was limited to detecting an absolute difference of approximately 0.30 between groups, assuming an alpha of 0.05 and 80% power.³³ Table 6 shows that 71% of intervention physicians compared with 28% of control physicians reported an audit of their practice for preventive services (P<.001). By the end of the study, 65% of the intervention physicians versus 48% of the control physicians indicated that they had a prevention policy or screening protocol in place, and 70% of intervention physicians compared with 58% of control physicians had created reminder systems for disease prevention.

Satisfaction with PF Intervention

At the telephone interview 6 months into the intervention, the mean satisfaction rating of intervention physicians was 4.08 on a scale of 1 (very dissatisfied) to 5 (very satisfied) with 80% satisfied with the intervention. At 17 months the mean satisfaction rating had risen to 4.5 with a 95% satisfaction rate.

At 6 months, 85% of the practices were satisfied with the frequency of visits of their assigned facilitator. At 17 months there was a 64% satisfaction rate, with the remaining 36% wanting more visits from the facilitator. The physicians commented on how the intervention had focused them on prevention in their practice. When the physicians were asked if they would agree to have a facilitator visit their practice in the future if given the opportunity, 90% agreed.

Concerns included not being able to continue the recall of patients at the end of the intervention and questioning the inappropriate maneuvers as too controversial. A physician from a large practice with 6 physicians commented that the facilitator could not easily work in the complex practice environment.

Discussion

Our study demonstrates that PFs can significantly improve the delivery of preventive services and in the process make quality contributions to a practice environment with high satisfaction rates from participating physicians.

Our intervention had a higher frequency and intensity of visits than other studies of this genre. The PFs had an average of almost 2 visits per month lasting approximately 105 minutes per visit. Dietrich and colleagues²¹ reported only 4 visits over a 3-month period lasting an average of 120 minutes, and Hulscher and coworkers²² reported approximately 25 visits with an average duration of 73 minutes. Others have been even less frequent,²⁰ and in other studies it is not reported.^34-36

The critical intervention components as evidenced by physician feedback, changes between control and intervention practices, and the amount of facilitator time spent on each component were: (1) audit and feedback, (2) sharing and discussing information to build consensus on an action plan, and (3) a reminder system. Similarly, the Cancer Prevention in Community Practice Project achieved 100% success in implementing change using customized preventive care flowsheets.³⁷ Of the 7 intervention components, patient education materials and patient-mediated interventions such as posters in the waiting room were considered to be the least useful.

Overall, physicians and nurses working within the practices were very satisfied with the intervention, and 90% were willing to have the nurse facilitator continue working with their practice.

Lessons learned from the process evaluation for improving the delivery of the outreach facilitation intervention include:

Focusing on the 3 key intervention components (audit and feedback, seeking consensus on an action plan, and implementing a reminder system) and tailoring these to the needs of the practice

Preparing patient education and patient-mediated materials only if the practice requests such materials

Developing simpler strategies to encourage physicians to counsel their patients that smoke to quit smoking

Providing the facilitators an administrative assistant to reduce the amount of their time spent on administrative duties for the practices and increase time on-site

Strengths

The strengths of the study include the completeness of the data set, the theoretical framework for data collection, the use of multiple data sources and data collection methods, and the prospective data collection methodology.

Limitations

There are several limitations to the process evaluation methods. Much of the data was provided by the facilitators themselves, and therefore the possibility of bias exists. The study population consisted of HSOs, and therefore the results may not be generalizable. There is a possibility of social desirability bias in the satisfaction rates. Finally, our analyses of the process data were descriptive and exploratory.

Conclusions

Process evaluation often identifies future areas of research. Follow-up of the few practices that were dissatisfied with facilitation should be carried out to understand why they were dissatisfied. Sustainability needs to be addressed. For example, Dietrich and colleagues³⁸ found that 5-year durability of a preventive services office system depended on the physician’s preventive care philosophy. McCowan and coworkers³⁹ found that the effect of a facilitator was not sustained for 2 years. Finally, to maximize cost-effectiveness, more research is required to determine how much of a dose of facilitation is required and how frequently facilitators should visit to achieve a positive outcome.

Acknowledgments

We wish to acknowledge the financial support of the Ontario Ministry of Health, as well as the substantial contributions of the 3 nurse facilitators (Ingrid LeClaire, Ann MacLeod, and Ruth Blochlinger). We also wish to thank the many physicians and nurses who participated in the study.

A randomized controlled field trial of a multifaceted intervention to improve preventive care tailored to the needs of participating family practices was conducted in Southern Ontario and delivered by nurses trained in the facilitation of prevention. We focus on the process evaluation and complement the outcome evaluation¹ by describing how the program was implemented in the intervention practices.

Improving preventive performance is both important and necessary. There is substantial room to improve the rates of appropriate preventive practice.² The Canadian Task Force on the Periodic Health xamination^3,4 has established guidelines for the delivery of preventive care that are supported by clinical evidence as effective in decreasing the impact of disease. However, evidence-based guidelines are not self-implementing.^5-7 Changing physicians’ long-held patterns of behavior and the environments in which they work is complex and difficult. Unless the barriers to change can be overcome and actions taken to put preventive care guidelines into practice, evidence-based guideline development efforts will be wasted, and the quality of preventive care will not improve.⁸

Several reviews have focussed on the effectiveness of different interventions for implementing guidelines and improving care.^6,7,9-13 Multifaceted interventions employing trained individuals who meet with providers in their practice settings to provide information and assist the practice in implementing evidence-based guidelines have been shown to be more effective than single interventions.^11-14 Tailoring interventions to the requirements of the practice has also been proposed as important in supporting practice changes and in attaining more successful outcomes in preventive care performance compared with interventions that are fixed and lack this flexibility.^15-17

As important as knowing what interventions work to improve preventive care performance is understanding why they work. The techniques of process evaluation allow the investigator to determine the extent to which the intervention designed to change practice patterns was actually implemented as planned. Adequate documentation of process facilitates replication and fine-tuning of the intervention.

Intervention Description

Our study built on the work of Fullard and colleagues18 and used a tailored multifaceted approach to getting evidence into action by focusing on the educational, attitudinal, and organizational barriers to change and tailoring interventions to the needs of the practice.^17-24 The intervention employed 3 prevention facilitators (PFs) with both master’s degrees in community nursing and skills and previous experience in facilitation. Each PF had primary responsibility for up to 8 primary care practices with up to 6 physicians per practice.

The PFs underwent 30 weeks of intensive training before being assigned to randomly selected intervention practices. The training covered an orientation session, medical office computer systems, medical practice management, prevention in primary care, evidence-based medicine, and facilitation and audit skills development. Approximately 28 hours per week were spent in training and 7 hours per week in preparation and planning. Six of the 30 weeks of training were spent applying skills in a primary care office setting. Once in the field, they were instructed to offer 7 intervention strategies designed to change practice patterns and improve preventive care performance. The strategies were identified from reviews of the literature and constituted the multifaceted component of the intervention.^10,11 The PFs were permitted to tailor these strategies to the needs and unique circumstances of the practice. The strategies were: (1) audit and ongoing feedback, (2) consensus building, (3) opinion leaders and networking, (4) academic detailing and education materials, (5) reminder systems, (6) patient-mediated activities, and (7) patient education materials.

The PFs worked with all physicians and allied health staff in the practice. They provided management support to practices and followed a quality improvement framework similar to that proposed by Leininger and coworkers.²⁵ For each practice the PFs were to: (1) present baseline preventive performance rates, (2) facilitate the development of a practice policy for preventive care, (3) assist in setting goals and desirable levels of performance, (4) assist in the development of a written plan for implementing preventive care, (5) assist in the development and adaptation of tools and the strategies to implement the prevention plan, (6) facilitate meetings to assess progress and modify the plan if necessary, and (7) conduct chart audits to measure the impact of the changes made. The intervention period lasted 18 months and ended in December 1998.

The Figure is the program logic model describing each of the 7 intervention component strategies and the associated work activities, outputs, and short-term and long-term objectives associated with each component. It served as a framework for the evaluation of the intervention.^26-28 The logic model allowed us to look inside the black box of the intervention^29,30 by linking implementation activity to outcomes, and provided a framework to explore which elements worked and why.

Intervention Outcomes

The goal of the intervention was to increase the performance of 8 preventive maneuvers supported by evidence as appropriate and decrease the performance of 5 preventive maneuvers supported by evidence as inappropriate.¹ An absolute change over time of 11.51% in preventive care performance in favor of intervention practices was found (F=19.29 [df=1,43], P<.0001. In other words, the intervention practices improved preventive performance by 36% going from 31% of eligible patients having received preventive care to 43% while the control practices remained at 32%.¹

Methods

Research Questions

There were 2 objectives to our process evaluation: to document the extent to which the intervention was implemented with fidelity and to gain insight into how facilitation worked to improve preventive performance. The process evaluation was designed to answer questions concerning: (1) the time involved to deliver intervention components, (2) the quality of the delivery of intervention components, and (3) physician satisfaction with the intervention components. Quality was assessed by examining the scope or range of delivery of the intervention components and by analyzing the feedback received from practices on the usefulness of the intervention components.

Setting

The intervention arm of the trial included 22 practices with 54 physicians (Table 1). All health service organizations (HSOs) in Southwestern Ontario were approached to participate in the study. HSOs are primary care practices reimbursed primarily through capitation rather than fee for service. A total of 46 of the 100 primary care practices were recruited (response rate=46%). At follow-up only one intervention practice was lost, because the entire practice had moved. Intervention and control group practices did not differ significantly on any of the measured demographic characteristics (Table 2). Complete details on practice recruitment and attrition rates are published elsewhere.¹

The practices covered a geographic area where the greatest distance between any 2 practices was more than 600 kilometers. PFs were assigned practices within a specific region of this geographic area. They arranged times to visit and work with intervention practices and traveled by car between visits to practices. PFs worked independently at their residences and corresponded with the project team through electronic mail regularly and quarterly with scheduled meetings.

Data Collection Tools

Each intervention practice was visited regularly by the same nurse facilitator who documented her activities and progress on 2 structured forms known as the weekly activity sheet and the monthly narrative report. Weekly activity sheets noted the number of hours spent on both on-site and offsite activities. Monthly narrative reports provided detailed information on the number of visits to a practice, the activities within each practice, the outcomes of those activities, the number of participants in meetings, and the plan for the following month. The activities in the narrative reports were summarized by intervention component to provide a cumulative overview of all intervention activity within a practice.

Also during the intervention, semistructured telephone interviews of participating physicians were conducted by 2 physician members of the project team at 6 months and 17 months. Participating physicians were asked what they had been happy and unhappy with and their ideas of improvement. Close-ended questions measured overall satisfaction with the intervention. The interview at 17 months also asked physicians if they would agree to have a nurse facilitator continue to visit their practice if funding were found.

At the end of the intervention, the PFs conducted interviews with each of the physicians identified as the primary contact in the intervention practices to solicit feedback on their experience. Physicians in both the intervention and control arm were sent a questionnaire by mail to report any changes that had taken place in their practice over the preceding 18 months.

Analysis

Data were analyzed to address the 3 research questions for the process evaluation utilizing the Logic Model as the conceptual framework (Figure). To determine how often various intervention components were delivered, the total hours spent at each practice and the total number of contacts with each practice by intervention component were calculated from the PF activity sheets.

To determine intervention quality, triangulation³¹ was used to attain a complete understanding of the quality of implementation. Multiple data sources and analysis methods were used to reveal the underlying dimensions of quality. All data sources were reviewed and analyses were conducted independently by 2 members of the investigation team. The members of the team held a debriefing session to discuss their findings and seek consensus. First, the monthly narrative reports across intervention sites were summarized to qualitatively describe the type, breadth, and scope of activity for each intervention component. Second, the activity descriptions and open-ended interview responses were content analyzed³² and coded, and frequencies were generated. The goal of this analysis was to identify significant descriptions of which intervention elements worked well and which did not. Finally, intervention and control practices were compared with contingency tables, and a chi-square statistic was used to determine differences on questionnaire responses concerning practice changes over the period of the intervention.

To determine physician satisfaction, open-ended satisfaction survey responses were coded and frequencies generated for ratings of overall satisfaction with the performance of the PF and the intervention.

Results

PF Program Implementation

Table 3 shows the number of hours spent on project activities during the period of the intervention. The PFs spent the largest proportion of their time (28%) on administrative duties, such as team meetings, telephone calls, internal reporting, preparing the project newsletter, coordinating networking conferences for intervention practices, photocopying, and filing. Sixteen percent of the PFs’ time was spent on-site facilitating changes to improve preventive care in the practice. Travel accounted for an average of 12% of the PFs’ time, although this varied depending on the distance to the practices.

Table 4 provides information on the number of contacts and hours spent on-site for each component of the intervention. On average, each intervention practice was contacted 33 times by a PF, with each visit lasting an average of 1 hour and 45 minutes. The most frequent forms of contact concerned developing reminder systems, conducting chart audits and providing feedback preventive care performance, and working to achieve consensus on the adoption of preventive care strategies. Both academic detailing to physicians and supplying critically appraised patient education materials averaged approximately 20 minutes but involved a great deal of preparation time. Few practices were interested in posters in the waiting room or a patient newsletter on prevention, so fewer contacts were made for those components.

Quality of Implementation

To assess quality, the frequency of each component of the intervention was tallied, physician feedback on the usefulness of intervention components was summarized, and self-reported practice changes between intervention and control physicians was reported.

Intervention Scope

Audit and Feedback. All 22 intervention practices received a presentation by the PF on the initial audit results to raise awareness of preventive care practice patterns. This was usually done in a kick-off meeting involving both physicians and nurses and often required more than one presentation to cover the various staff in the practice. Twenty practices requested subsequent analyses of data to follow their rates of performance. In addition, 18 practices requested audits of their charts for specific maneuvers, such as influenza vaccination and mammography

Consensus Building. All practices were involved in meetings with the PF to identify opportunities for improvement, assess needs, and select priority areas and strategies for improving preventive care performance. Interviews were conducted with nurses and other staff to promote their role in preventive care delivery.

Academic Detailing. Twenty-one out of 22 sites agreed to receive and discuss critically appraised evidence for the preventive maneuvers under study, and some requested similar information on other preventive areas, such as cholesterol and osteoporosis.

Reminder Systems. All of the intervention sites implemented some form of reminder system. Eighteen sites implemented a preventive care flow sheet; 2 sites used a chart stamp; and 2 sites implemented a computerized reminder system. Nineteen sites developed recall initiatives for flu vaccine, mammography, and Papanicolaou tests. Seventeen sites implemented chart stickers for smoking counseling or mammography.

Opinion Leaders. All sites received copies of the PF project newsletter that contained articles by influential individuals describing the importance of preventive care and descriptions of colleagues’ preventive care implementation efforts. Most practices attended a workshop that included an influential keynote speaker, and 27% of the participating physicians shared their knowledge about preventive care through publishing in the newsletter and/or public speaking.

Patient Education. All sites were provided with patient education materials from credible sources on request, and all received a binder of patient education materials constructed specifically to contain materials on the appropriate preventive maneuvers under study. The binders were regularly updated.

Patient Mediated. Posters designed to prompt patients to ask about folic acid, flu vaccine, and mammography were offered to all sites. Thirteen sites implemented a patient consent form for prostate-specific antigen (PSA) testing. Eight sites received preventive care diaries for patients. Five sites had a prevention newsletter for patients. Four sites agreed to pilot a health risk appraisal software program.

Physician Feedback

At the end of the intervention the facilitator asked physicians about their experience, including what was most and least useful to them. Table 5 provides a summary of the content analysis of physician responses.

Audit and feedback, both initially and subsequently, comprised the component most frequently considered to be important in creating change. Almost as often, the preventive care flow sheet was identified as useful. The facilitator sessions designed to seek consensus on preventive care guidelines and strategies for implementation were also appreciated.

Several physicians did not agree with the evidence on PSA testing. Others did not feel that counseling for folic acid was a priority. Some found the patient education binder cumbersome, and others found the sticker system for tobacco counseling unwieldy. Thus, both were underused. Two physicians noted that the preventive care wall chart was not helpful.

Physician Self-Reported Practice Changes

Eighty-six percent (93/108) of the intervention and control physicians responded to a questionnaire at the end of the study. Due to sample size, statistical power was limited to detecting an absolute difference of approximately 0.30 between groups, assuming an alpha of 0.05 and 80% power.³³ Table 6 shows that 71% of intervention physicians compared with 28% of control physicians reported an audit of their practice for preventive services (P<.001). By the end of the study, 65% of the intervention physicians versus 48% of the control physicians indicated that they had a prevention policy or screening protocol in place, and 70% of intervention physicians compared with 58% of control physicians had created reminder systems for disease prevention.

Satisfaction with PF Intervention

At the telephone interview 6 months into the intervention, the mean satisfaction rating of intervention physicians was 4.08 on a scale of 1 (very dissatisfied) to 5 (very satisfied) with 80% satisfied with the intervention. At 17 months the mean satisfaction rating had risen to 4.5 with a 95% satisfaction rate.

At 6 months, 85% of the practices were satisfied with the frequency of visits of their assigned facilitator. At 17 months there was a 64% satisfaction rate, with the remaining 36% wanting more visits from the facilitator. The physicians commented on how the intervention had focused them on prevention in their practice. When the physicians were asked if they would agree to have a facilitator visit their practice in the future if given the opportunity, 90% agreed.

Concerns included not being able to continue the recall of patients at the end of the intervention and questioning the inappropriate maneuvers as too controversial. A physician from a large practice with 6 physicians commented that the facilitator could not easily work in the complex practice environment.

Discussion

Our study demonstrates that PFs can significantly improve the delivery of preventive services and in the process make quality contributions to a practice environment with high satisfaction rates from participating physicians.

Our intervention had a higher frequency and intensity of visits than other studies of this genre. The PFs had an average of almost 2 visits per month lasting approximately 105 minutes per visit. Dietrich and colleagues²¹ reported only 4 visits over a 3-month period lasting an average of 120 minutes, and Hulscher and coworkers²² reported approximately 25 visits with an average duration of 73 minutes. Others have been even less frequent,²⁰ and in other studies it is not reported.^34-36

The critical intervention components as evidenced by physician feedback, changes between control and intervention practices, and the amount of facilitator time spent on each component were: (1) audit and feedback, (2) sharing and discussing information to build consensus on an action plan, and (3) a reminder system. Similarly, the Cancer Prevention in Community Practice Project achieved 100% success in implementing change using customized preventive care flowsheets.³⁷ Of the 7 intervention components, patient education materials and patient-mediated interventions such as posters in the waiting room were considered to be the least useful.

Overall, physicians and nurses working within the practices were very satisfied with the intervention, and 90% were willing to have the nurse facilitator continue working with their practice.

Lessons learned from the process evaluation for improving the delivery of the outreach facilitation intervention include:

Focusing on the 3 key intervention components (audit and feedback, seeking consensus on an action plan, and implementing a reminder system) and tailoring these to the needs of the practice

Preparing patient education and patient-mediated materials only if the practice requests such materials

Developing simpler strategies to encourage physicians to counsel their patients that smoke to quit smoking

Providing the facilitators an administrative assistant to reduce the amount of their time spent on administrative duties for the practices and increase time on-site

Strengths

The strengths of the study include the completeness of the data set, the theoretical framework for data collection, the use of multiple data sources and data collection methods, and the prospective data collection methodology.

Limitations

There are several limitations to the process evaluation methods. Much of the data was provided by the facilitators themselves, and therefore the possibility of bias exists. The study population consisted of HSOs, and therefore the results may not be generalizable. There is a possibility of social desirability bias in the satisfaction rates. Finally, our analyses of the process data were descriptive and exploratory.

Conclusions

Process evaluation often identifies future areas of research. Follow-up of the few practices that were dissatisfied with facilitation should be carried out to understand why they were dissatisfied. Sustainability needs to be addressed. For example, Dietrich and colleagues³⁸ found that 5-year durability of a preventive services office system depended on the physician’s preventive care philosophy. McCowan and coworkers³⁹ found that the effect of a facilitator was not sustained for 2 years. Finally, to maximize cost-effectiveness, more research is required to determine how much of a dose of facilitation is required and how frequently facilitators should visit to achieve a positive outcome.

Acknowledgments

We wish to acknowledge the financial support of the Ontario Ministry of Health, as well as the substantial contributions of the 3 nurse facilitators (Ingrid LeClaire, Ann MacLeod, and Ruth Blochlinger). We also wish to thank the many physicians and nurses who participated in the study.

A randomized controlled field trial of a multifaceted intervention to improve preventive care tailored to the needs of participating family practices was conducted in Southern Ontario and delivered by nurses trained in the facilitation of prevention. We focus on the process evaluation and complement the outcome evaluation¹ by describing how the program was implemented in the intervention practices.

Improving preventive performance is both important and necessary. There is substantial room to improve the rates of appropriate preventive practice.² The Canadian Task Force on the Periodic Health xamination^3,4 has established guidelines for the delivery of preventive care that are supported by clinical evidence as effective in decreasing the impact of disease. However, evidence-based guidelines are not self-implementing.^5-7 Changing physicians’ long-held patterns of behavior and the environments in which they work is complex and difficult. Unless the barriers to change can be overcome and actions taken to put preventive care guidelines into practice, evidence-based guideline development efforts will be wasted, and the quality of preventive care will not improve.⁸

Several reviews have focussed on the effectiveness of different interventions for implementing guidelines and improving care.^6,7,9-13 Multifaceted interventions employing trained individuals who meet with providers in their practice settings to provide information and assist the practice in implementing evidence-based guidelines have been shown to be more effective than single interventions.^11-14 Tailoring interventions to the requirements of the practice has also been proposed as important in supporting practice changes and in attaining more successful outcomes in preventive care performance compared with interventions that are fixed and lack this flexibility.^15-17

As important as knowing what interventions work to improve preventive care performance is understanding why they work. The techniques of process evaluation allow the investigator to determine the extent to which the intervention designed to change practice patterns was actually implemented as planned. Adequate documentation of process facilitates replication and fine-tuning of the intervention.

Intervention Description

Our study built on the work of Fullard and colleagues18 and used a tailored multifaceted approach to getting evidence into action by focusing on the educational, attitudinal, and organizational barriers to change and tailoring interventions to the needs of the practice.^17-24 The intervention employed 3 prevention facilitators (PFs) with both master’s degrees in community nursing and skills and previous experience in facilitation. Each PF had primary responsibility for up to 8 primary care practices with up to 6 physicians per practice.

The PFs underwent 30 weeks of intensive training before being assigned to randomly selected intervention practices. The training covered an orientation session, medical office computer systems, medical practice management, prevention in primary care, evidence-based medicine, and facilitation and audit skills development. Approximately 28 hours per week were spent in training and 7 hours per week in preparation and planning. Six of the 30 weeks of training were spent applying skills in a primary care office setting. Once in the field, they were instructed to offer 7 intervention strategies designed to change practice patterns and improve preventive care performance. The strategies were identified from reviews of the literature and constituted the multifaceted component of the intervention.^10,11 The PFs were permitted to tailor these strategies to the needs and unique circumstances of the practice. The strategies were: (1) audit and ongoing feedback, (2) consensus building, (3) opinion leaders and networking, (4) academic detailing and education materials, (5) reminder systems, (6) patient-mediated activities, and (7) patient education materials.

The PFs worked with all physicians and allied health staff in the practice. They provided management support to practices and followed a quality improvement framework similar to that proposed by Leininger and coworkers.²⁵ For each practice the PFs were to: (1) present baseline preventive performance rates, (2) facilitate the development of a practice policy for preventive care, (3) assist in setting goals and desirable levels of performance, (4) assist in the development of a written plan for implementing preventive care, (5) assist in the development and adaptation of tools and the strategies to implement the prevention plan, (6) facilitate meetings to assess progress and modify the plan if necessary, and (7) conduct chart audits to measure the impact of the changes made. The intervention period lasted 18 months and ended in December 1998.

The Figure is the program logic model describing each of the 7 intervention component strategies and the associated work activities, outputs, and short-term and long-term objectives associated with each component. It served as a framework for the evaluation of the intervention.^26-28 The logic model allowed us to look inside the black box of the intervention^29,30 by linking implementation activity to outcomes, and provided a framework to explore which elements worked and why.

Intervention Outcomes

The goal of the intervention was to increase the performance of 8 preventive maneuvers supported by evidence as appropriate and decrease the performance of 5 preventive maneuvers supported by evidence as inappropriate.¹ An absolute change over time of 11.51% in preventive care performance in favor of intervention practices was found (F=19.29 [df=1,43], P<.0001. In other words, the intervention practices improved preventive performance by 36% going from 31% of eligible patients having received preventive care to 43% while the control practices remained at 32%.¹

Methods

Research Questions

There were 2 objectives to our process evaluation: to document the extent to which the intervention was implemented with fidelity and to gain insight into how facilitation worked to improve preventive performance. The process evaluation was designed to answer questions concerning: (1) the time involved to deliver intervention components, (2) the quality of the delivery of intervention components, and (3) physician satisfaction with the intervention components. Quality was assessed by examining the scope or range of delivery of the intervention components and by analyzing the feedback received from practices on the usefulness of the intervention components.

Setting

The intervention arm of the trial included 22 practices with 54 physicians (Table 1). All health service organizations (HSOs) in Southwestern Ontario were approached to participate in the study. HSOs are primary care practices reimbursed primarily through capitation rather than fee for service. A total of 46 of the 100 primary care practices were recruited (response rate=46%). At follow-up only one intervention practice was lost, because the entire practice had moved. Intervention and control group practices did not differ significantly on any of the measured demographic characteristics (Table 2). Complete details on practice recruitment and attrition rates are published elsewhere.¹

The practices covered a geographic area where the greatest distance between any 2 practices was more than 600 kilometers. PFs were assigned practices within a specific region of this geographic area. They arranged times to visit and work with intervention practices and traveled by car between visits to practices. PFs worked independently at their residences and corresponded with the project team through electronic mail regularly and quarterly with scheduled meetings.

Data Collection Tools

Each intervention practice was visited regularly by the same nurse facilitator who documented her activities and progress on 2 structured forms known as the weekly activity sheet and the monthly narrative report. Weekly activity sheets noted the number of hours spent on both on-site and offsite activities. Monthly narrative reports provided detailed information on the number of visits to a practice, the activities within each practice, the outcomes of those activities, the number of participants in meetings, and the plan for the following month. The activities in the narrative reports were summarized by intervention component to provide a cumulative overview of all intervention activity within a practice.

Also during the intervention, semistructured telephone interviews of participating physicians were conducted by 2 physician members of the project team at 6 months and 17 months. Participating physicians were asked what they had been happy and unhappy with and their ideas of improvement. Close-ended questions measured overall satisfaction with the intervention. The interview at 17 months also asked physicians if they would agree to have a nurse facilitator continue to visit their practice if funding were found.

At the end of the intervention, the PFs conducted interviews with each of the physicians identified as the primary contact in the intervention practices to solicit feedback on their experience. Physicians in both the intervention and control arm were sent a questionnaire by mail to report any changes that had taken place in their practice over the preceding 18 months.

Analysis

Data were analyzed to address the 3 research questions for the process evaluation utilizing the Logic Model as the conceptual framework (Figure). To determine how often various intervention components were delivered, the total hours spent at each practice and the total number of contacts with each practice by intervention component were calculated from the PF activity sheets.

To determine intervention quality, triangulation³¹ was used to attain a complete understanding of the quality of implementation. Multiple data sources and analysis methods were used to reveal the underlying dimensions of quality. All data sources were reviewed and analyses were conducted independently by 2 members of the investigation team. The members of the team held a debriefing session to discuss their findings and seek consensus. First, the monthly narrative reports across intervention sites were summarized to qualitatively describe the type, breadth, and scope of activity for each intervention component. Second, the activity descriptions and open-ended interview responses were content analyzed³² and coded, and frequencies were generated. The goal of this analysis was to identify significant descriptions of which intervention elements worked well and which did not. Finally, intervention and control practices were compared with contingency tables, and a chi-square statistic was used to determine differences on questionnaire responses concerning practice changes over the period of the intervention.

To determine physician satisfaction, open-ended satisfaction survey responses were coded and frequencies generated for ratings of overall satisfaction with the performance of the PF and the intervention.

Results

PF Program Implementation

Table 3 shows the number of hours spent on project activities during the period of the intervention. The PFs spent the largest proportion of their time (28%) on administrative duties, such as team meetings, telephone calls, internal reporting, preparing the project newsletter, coordinating networking conferences for intervention practices, photocopying, and filing. Sixteen percent of the PFs’ time was spent on-site facilitating changes to improve preventive care in the practice. Travel accounted for an average of 12% of the PFs’ time, although this varied depending on the distance to the practices.

Table 4 provides information on the number of contacts and hours spent on-site for each component of the intervention. On average, each intervention practice was contacted 33 times by a PF, with each visit lasting an average of 1 hour and 45 minutes. The most frequent forms of contact concerned developing reminder systems, conducting chart audits and providing feedback preventive care performance, and working to achieve consensus on the adoption of preventive care strategies. Both academic detailing to physicians and supplying critically appraised patient education materials averaged approximately 20 minutes but involved a great deal of preparation time. Few practices were interested in posters in the waiting room or a patient newsletter on prevention, so fewer contacts were made for those components.

Quality of Implementation

To assess quality, the frequency of each component of the intervention was tallied, physician feedback on the usefulness of intervention components was summarized, and self-reported practice changes between intervention and control physicians was reported.

Intervention Scope

Audit and Feedback. All 22 intervention practices received a presentation by the PF on the initial audit results to raise awareness of preventive care practice patterns. This was usually done in a kick-off meeting involving both physicians and nurses and often required more than one presentation to cover the various staff in the practice. Twenty practices requested subsequent analyses of data to follow their rates of performance. In addition, 18 practices requested audits of their charts for specific maneuvers, such as influenza vaccination and mammography

Consensus Building. All practices were involved in meetings with the PF to identify opportunities for improvement, assess needs, and select priority areas and strategies for improving preventive care performance. Interviews were conducted with nurses and other staff to promote their role in preventive care delivery.

Academic Detailing. Twenty-one out of 22 sites agreed to receive and discuss critically appraised evidence for the preventive maneuvers under study, and some requested similar information on other preventive areas, such as cholesterol and osteoporosis.

Reminder Systems. All of the intervention sites implemented some form of reminder system. Eighteen sites implemented a preventive care flow sheet; 2 sites used a chart stamp; and 2 sites implemented a computerized reminder system. Nineteen sites developed recall initiatives for flu vaccine, mammography, and Papanicolaou tests. Seventeen sites implemented chart stickers for smoking counseling or mammography.

Opinion Leaders. All sites received copies of the PF project newsletter that contained articles by influential individuals describing the importance of preventive care and descriptions of colleagues’ preventive care implementation efforts. Most practices attended a workshop that included an influential keynote speaker, and 27% of the participating physicians shared their knowledge about preventive care through publishing in the newsletter and/or public speaking.

Patient Education. All sites were provided with patient education materials from credible sources on request, and all received a binder of patient education materials constructed specifically to contain materials on the appropriate preventive maneuvers under study. The binders were regularly updated.

Patient Mediated. Posters designed to prompt patients to ask about folic acid, flu vaccine, and mammography were offered to all sites. Thirteen sites implemented a patient consent form for prostate-specific antigen (PSA) testing. Eight sites received preventive care diaries for patients. Five sites had a prevention newsletter for patients. Four sites agreed to pilot a health risk appraisal software program.

Physician Feedback

At the end of the intervention the facilitator asked physicians about their experience, including what was most and least useful to them. Table 5 provides a summary of the content analysis of physician responses.

Audit and feedback, both initially and subsequently, comprised the component most frequently considered to be important in creating change. Almost as often, the preventive care flow sheet was identified as useful. The facilitator sessions designed to seek consensus on preventive care guidelines and strategies for implementation were also appreciated.

Several physicians did not agree with the evidence on PSA testing. Others did not feel that counseling for folic acid was a priority. Some found the patient education binder cumbersome, and others found the sticker system for tobacco counseling unwieldy. Thus, both were underused. Two physicians noted that the preventive care wall chart was not helpful.

Physician Self-Reported Practice Changes

Eighty-six percent (93/108) of the intervention and control physicians responded to a questionnaire at the end of the study. Due to sample size, statistical power was limited to detecting an absolute difference of approximately 0.30 between groups, assuming an alpha of 0.05 and 80% power.³³ Table 6 shows that 71% of intervention physicians compared with 28% of control physicians reported an audit of their practice for preventive services (P<.001). By the end of the study, 65% of the intervention physicians versus 48% of the control physicians indicated that they had a prevention policy or screening protocol in place, and 70% of intervention physicians compared with 58% of control physicians had created reminder systems for disease prevention.

Satisfaction with PF Intervention

At the telephone interview 6 months into the intervention, the mean satisfaction rating of intervention physicians was 4.08 on a scale of 1 (very dissatisfied) to 5 (very satisfied) with 80% satisfied with the intervention. At 17 months the mean satisfaction rating had risen to 4.5 with a 95% satisfaction rate.

At 6 months, 85% of the practices were satisfied with the frequency of visits of their assigned facilitator. At 17 months there was a 64% satisfaction rate, with the remaining 36% wanting more visits from the facilitator. The physicians commented on how the intervention had focused them on prevention in their practice. When the physicians were asked if they would agree to have a facilitator visit their practice in the future if given the opportunity, 90% agreed.

Concerns included not being able to continue the recall of patients at the end of the intervention and questioning the inappropriate maneuvers as too controversial. A physician from a large practice with 6 physicians commented that the facilitator could not easily work in the complex practice environment.

Discussion

Our study demonstrates that PFs can significantly improve the delivery of preventive services and in the process make quality contributions to a practice environment with high satisfaction rates from participating physicians.

Our intervention had a higher frequency and intensity of visits than other studies of this genre. The PFs had an average of almost 2 visits per month lasting approximately 105 minutes per visit. Dietrich and colleagues²¹ reported only 4 visits over a 3-month period lasting an average of 120 minutes, and Hulscher and coworkers²² reported approximately 25 visits with an average duration of 73 minutes. Others have been even less frequent,²⁰ and in other studies it is not reported.^34-36

The critical intervention components as evidenced by physician feedback, changes between control and intervention practices, and the amount of facilitator time spent on each component were: (1) audit and feedback, (2) sharing and discussing information to build consensus on an action plan, and (3) a reminder system. Similarly, the Cancer Prevention in Community Practice Project achieved 100% success in implementing change using customized preventive care flowsheets.³⁷ Of the 7 intervention components, patient education materials and patient-mediated interventions such as posters in the waiting room were considered to be the least useful.

Overall, physicians and nurses working within the practices were very satisfied with the intervention, and 90% were willing to have the nurse facilitator continue working with their practice.

Lessons learned from the process evaluation for improving the delivery of the outreach facilitation intervention include:

Focusing on the 3 key intervention components (audit and feedback, seeking consensus on an action plan, and implementing a reminder system) and tailoring these to the needs of the practice

Preparing patient education and patient-mediated materials only if the practice requests such materials

Developing simpler strategies to encourage physicians to counsel their patients that smoke to quit smoking

Providing the facilitators an administrative assistant to reduce the amount of their time spent on administrative duties for the practices and increase time on-site

Strengths

The strengths of the study include the completeness of the data set, the theoretical framework for data collection, the use of multiple data sources and data collection methods, and the prospective data collection methodology.

Limitations

There are several limitations to the process evaluation methods. Much of the data was provided by the facilitators themselves, and therefore the possibility of bias exists. The study population consisted of HSOs, and therefore the results may not be generalizable. There is a possibility of social desirability bias in the satisfaction rates. Finally, our analyses of the process data were descriptive and exploratory.

Conclusions

Process evaluation often identifies future areas of research. Follow-up of the few practices that were dissatisfied with facilitation should be carried out to understand why they were dissatisfied. Sustainability needs to be addressed. For example, Dietrich and colleagues³⁸ found that 5-year durability of a preventive services office system depended on the physician’s preventive care philosophy. McCowan and coworkers³⁹ found that the effect of a facilitator was not sustained for 2 years. Finally, to maximize cost-effectiveness, more research is required to determine how much of a dose of facilitation is required and how frequently facilitators should visit to achieve a positive outcome.

Acknowledgments

We wish to acknowledge the financial support of the Ontario Ministry of Health, as well as the substantial contributions of the 3 nurse facilitators (Ingrid LeClaire, Ann MacLeod, and Ruth Blochlinger). We also wish to thank the many physicians and nurses who participated in the study.