Original Research

Family physicians and general internists report an average of 6 questions for each half-day of office practice,^1-3 and 70% of these questions remain unanswered. The 2 factors that significantly predict whether a physician will attempt to answer a clinical question are the physician’s belief that a definitive answer exists and the urgency of the patient’s problem.⁴

Gorman and colleagues³ reported that medical librarians found clear answers for 46% of 60 randomly selected questions from family physicians; 51% would affect practice. The medical librarians searched for an average of 43 minutes per question. In a second study,⁵ medical librarians used MEDLINE and textbooks to answer 86 questions from family physicians. The MEDLINE searches took a mean of 27 minutes, and textbook searches took a mean of 6 minutes. Search results answered 54% of the clinical questions completely or nearly completely. Physicians estimated that the answers would have a “major” or “fairly major” impact on practice for 35% of their questions. MEDLINE searches provided answers to 43% of the questions, while textbook searches provided answers for an additional 11%.

Many physicians do not have the searching skills or access to the range of knowledge resources that librarians use. Even if they did, they do not take the time to conduct such searches during patient care. One study¹ found that physicians spent less than 2 minutes on average seeking an answer to a question. Thus, most clinical questions remain unanswered.

Electronic medical databases that provide answers directly (not just reference citations) may make it easier for clinicians to obtain answers at the point of care. We found no systematic evaluation of the capacity of such databases to answer clinical questions. We conducted this study to determine the extent to which current electronic medical databases can answer family physicians’ point-of-care clinical questions.

Methods

Database Selection

We solicited nominations for potentially suitable databases from multiple E-mail lists (including communities of family physicians [Family-L], medical informaticians [FAM-MED] and medical librarians [MEDLIB-L, MCMLA-L]) and through Web searches. A selection team consisting of 3 family physicians (J.S., D.W., B.E.) and a medical librarian (none of whom had financial relationships with any databases) determined whether the nominated databases met our inclusion criteria Table 1.

Clinical Questions

More than 1200 clinical questions had been previously collected from observations of family physicians during office practice.^1,5 These questions had been classified by typology (eg, Is test X indicated in situation Y?) and by topic (eg, dermatology).¹ We selected questions from these sources that were categorized among the most common typologies (8 of 68 typologies covering 50% of the questions) and the most common topics (7 of 62 topics covering 43% of the questions). These combinations of typologies and topics accounted for 272 (23%) of the 1204 questions.

If necessary, each question was translated by 2 physicians (B.A. and D.W. working together) to meet the following criteria: (1) clear enough to imagine an applicable clinical scenario, (2) answerable (ie, the question could theoretically be answered using clinical references without further patient data regardless of whether an answer was known to exist), (3) clinically relevant, and (4) true to the original question (ie, containing the information need and the modifying factors of the original question).

Each question was then independently proofread by at least 2 other physicians and translated again if necessary. Thirteen questions (5%) that did not meet these criteria after a second translation were dropped. Forty-seven questions (17%) that referred to information needs that could be adequately answered using the Physicians’ Desk Reference⁶ were dropped (eg, Are Paxil tablets scored?). The remaining 212 questions represented 8 typologies.¹ Two or 3 questions were randomly selected from each typology for a total of 20 questions Table 2.

Testing

Two family physicians with experience in computer searching (B.A., D.W.) independently searched for answers using each of the included databases. In the case of DynaMed, for which Dr Alper is the medical director, another family physician was substituted as a searcher, and Dr Alper had no input or control over the testing or arbitration process for answers from DynaMed. Testing took place in April and May 2000.

Searching was performed using computers with Pentium III processors with a 100 megabyte-per-second network connection to the Internet and server-mounted CD-ROMs.

Each searcher used the same 20 questions to evaluate each database. The order of evaluation of the databases was at the discretion of the searchers, but the testing of a database was completed before starting the testing of another database. Searchers became familiar with each database before testing it by using the 5 screening questions.

A maximum of 10 minutes was allowed per question. Each answer was rated as adequate or inadequate. An answer was considered adequate if it contained sufficient information to guide clinical practice. For example, for the question “How do I determine the cause of chronic pruritus?”, the answer from the University of Iowa Family Practice Handbook (www.vh.org/Providers/ClinRef/FPHandbook/Chapter13/01-13.html) was considered adequate, because it included clinically useful recommendations: History should include details about (1) any skin lesions preceding the pruritus; (2) history of weight loss, fatigue, fever, malaise; (3) any recent stress emotionally; and (4) recent medications and travel. Physical examination with emphasis on the skin and its appendages — xerosis, excoriation, lichenification, hydration. Laboratory tests as suggested by the PE, which may include CBC, ESR, fasting glucose, renal or liver function tests, hepatitis panel, thyroid tests, stool for parasites, CXR.

Sources that provided general recommendations without information that could specifically guide clinical practice were considered inadequate. For example: “The cause of generalized pruritus should be sought and corrected. If no skin disease is apparent, a systemic disorder or drug-related cause should be sought.” The searcher recorded the answer and the time it took to obtain it rounded to the nearest number of minutes (1-10).

Scoring and Arbitration

The 2 physician searchers judged the adequacy of the answers to each question for each database. If the searchers both found adequate answers, the result was accepted as adequate, and the average time required to find and interpret the answer was recorded. If neither searcher found an adequate answer, then the answer was deemed inadequate. If only one searcher found an adequate answer, the second searcher evaluated that answer. If the answer was acceptable to the second searcher, it was considered an adequate answer, and the time for the first searcher was recorded.

When searchers disagreed on the adequacy of identified answers, an arbitration panel consisting of 3 family physicians who were not affiliated with any of the databases met independently from the searchers to determine the adequacy of the answers by consensus.

Analysis

Our primary outcome was the proportion of questions adequately answered by each database. We calculated 95% confidence limits for the proportions of adequate answers.⁷ Means and medians were determined for the time to reach adequate answers for each database. We calculated the k statistic for the independent findings of the 2 searchers and for the results after the searchers reviewed each other’s searches.⁸ We combined the results of individual databases to determine the proportion of questions answered by all combinations of 2, 3, and 4 databases. We considered the question adequately answered if any of the individual databases adequately answered the question.

Results

Thirty-eight databases were nominated, and 24 did not meet our inclusion criteria Table W1.* Fourteen databases met the inclusion criteria Table 3 and were evaluated with the set of 20 questions (280 answer assessments) by 2 searchers. The Figure summarizes the process of evaluating the answers. The initial agreement between searchers was good k=0.69). Discussion between the searchers resolved 21 (52.5%) of the 40 discrepant answer assessments. These were due to inadequate searching or timing out (searching for 10 minutes) by one searcher, who agreed with the adequacy of the answer found by the other searcher. The agreement between searchers at this stage was excellent (k= 0.94).

The remaining 19 discrepant assessments (for which the searchers had different opinions regarding the adequacy of the answers identified) were referred to the arbitration panel for determination of the final results. Ten of these were deemed adequate.

Results for individual databases in rank order of proportion of questions answered followed by average time to identify adequate answers are reported in Table 3. The combination of STAT!Ref and MDConsult could answer 85% of our set of 20 questions. Four combinations of 2 databases (STAT!Ref and either MAXX, MDChoice.com, Primary Care Guidelines, or Medscape) could answer 80% of our questions. Two combinations of 3 databases (STAT!Ref, MDConsult, and either DynaMed or MAXX) could answer 90% of our questions. Combinations of 4 databases answered the most sample questions (95%, 19/20). These combinations consisted of STAT!Ref, DynaMed, MAXX, and either MDConsult or American Family Physician.

We also evaluated combinations of databases that were available at no cost. The combination of the 2 no-cost databases that answered the largest proportion of questions (75%) was DynaMed and American Family Physician. The greatest proportion of clinical questions that could be answered using the freely available sources was 80%, and this required the use of 3 databases (DynaMed, MDChoice.com, and American Family Physician).

Discussion

Our study suggests that individual databases can answer a considerable proportion of family physicians’ clinical questions. Combinations of currently available databases can answer 75% or more. The searches in this study were based on the combination of efforts of 2 experienced physician searchers. These results may not be replicable in the practice setting but do provide an objective best-case scenario assessment of the content of these databases.

The time required to obtain answers, while much less than searching for original articles, is still longer than the 2-minute average time spent by family physicians in the study by Ely and colleagues.¹ Our time estimates are not precise, as time was not the primary focus of our study. Time was only recorded in 1-minute intervals, so searches that took 10 seconds were recorded as 1 minute. Even so, the existence of median times to obtain adequate answers greater than 2 minutes suggests that these databases may require more time than most physicians will take to pursue answers during patient care.

This is the first study to systematically evaluate how many questions can be answered by electronic medical databases. The strengths of this study include the use of a standard set of common questions asked by family physicians, testing by 2 experienced family physician searchers, and a systematic replicable approach to the evaluation. The only similar study we identified was one in which Graber and coworkers⁹ used 10 clinical questions and tested a commercial site, 2 medical meta-lists, 4 general search engines, and 9 medicine-specific search engines to determine the efficiency of answering clinical questions on the Web. Different approaches answered from 0 to 6 of the 10 questions, but that study looked primarily at sites that were not generally designed for use in clinical practice.

Limitations

Our study was limited by the relatively small number of questions, causing wide confidence intervals. Some answers were present in the databases but not found despite the use of 2 searchers. For example, a database manager identified 2 answers that were not found but would have been considered adequate.

We accepted answers as adequate if, in our judgment, they offered a practical course of action. We did not attempt to determine whether the individual asking the question believed that the answer was adequate nor did we attempt to validate the accuracy or currency of answers using independent standards. Many of the answers were based on sources that were several years old, and few were based on explicit evidence-based criteria. Although we determined the adequacy of answers for clinical practice through formal mechanisms, an in vivo study in which the clinicians asking the questions determined the adequacy of their findings during patient care activities would provide a more accurate assessment.

Our study presents a static evaluation of a dynamic field. Over time, answers may be lost because of lack of maintenance of resource links or may be gained by addition of new materials. Our use of questions gathered several years ago may not accurately reflect the ability of databases to answer current questions, which may be more likely to reflect new tests and treatments.

Many of the databases were designed for purposes other than meeting clinical information needs at the point of care. Performance in this study does not reflect the capacity of these databases to address their stated purposes. For example, the Translating Research Into Practice (TRIP) database is an excellent resource for searches of a large collection of evidence-based resources. These resources are generally limited to summaries of studies with the highest methodologic quality. The TRIP database did not perform well in our study partly because most of our test questions (consistent with questions in clinical practice) cannot currently be answered using studies of the highest methodologic quality. Another example is Medical Matrix, which provides a search engine and annotated summaries for exploring the entire medical Internet and not just clinical reference information.

We did not study the costs involved in using the databases we evaluated, and these costs may have changed since our study was conducted. Most of the databases we included were free to use at the time of the study and at the time of this report. The 3 collections of textbooks required access fees. STAT!Ref, which scored the highest in our study, did so because we used the complete collection available to us through our institutional library. This collection would cost an individual $2189 annually at the time of our study. A starter library was available for $199 annually and would only answer 40% of the questions.

Context

Family physicians and other primary care providers treat patients who have a wide variety of syndromes and symptoms. Because of the scope and breadth of primary care, it is nearly impossible for a clinician to keep up with rapidly changing medical information.¹⁰

Connelly and colleagues¹¹ surveyed 126 family physicians and found they used the Physicians’ Desk Reference and colleagues much more often than Index Medicus or computer-based bibliographic retrieval systems. Research literature was used infrequently and rated among the lowest in terms of credibility, availability, searchability, understandability, and applicability. Physicians preferred sources that had low cost and were relevant to specific patient problems over sources that had higher quality.

Conclusions

Current databases can answer a considerable proportion of clinical questions but have not reached their potential for efficiency. It is our hope that as electronic medical databases mature, they will be able to bridge this gap and bring the research literature to the point of care in useful and practical ways. This study provides a snapshot of how far we have come and how far we need to go to meet these needs.

Acknowledgments

Funding for our study was provided by a grant from the American Academy of Family Physicians to support the Center for Family Medicine Science and from 2 Bureau of Health Professions Awards (DHHS 1-D14-HP-00029-01, DHHS 5 T32 HP10038) from the Health Resources and Services Administration to the Department of Family and Community Medicine at University of Missouri-Columbia. The authors would like to acknowledge Erik Lindbloom, MD, MSPH, for assisting with the database testing as a substitute searcher for B.A.; E. Diane Johnson, MLS, for assisting with the selection of databases for study inclusion; Robert Phillips, Jr., MD, MSPH, for arbitration of questions and answers for which the searchers did not reach agreement along with B.E. and J.S.; David Cravens, Erik Lindbloom, Kevin Kane, Jim Brillhart, and Mark Ebell for proofreading the questions for clarity, answerability, and clinical relevance; John Ely and Lee Chambliss for providing clinical questions from their observations; Mark Ebell, John Ely, Erik Lindbloom, Jerry Osheroff, Lee Chambliss, David Mehr, Robin Kruse, John Smucny, and many others for constructive criticism in the design of this study; and Steve Zweig for editorial review.

Family physicians and general internists report an average of 6 questions for each half-day of office practice,^1-3 and 70% of these questions remain unanswered. The 2 factors that significantly predict whether a physician will attempt to answer a clinical question are the physician’s belief that a definitive answer exists and the urgency of the patient’s problem.⁴

Gorman and colleagues³ reported that medical librarians found clear answers for 46% of 60 randomly selected questions from family physicians; 51% would affect practice. The medical librarians searched for an average of 43 minutes per question. In a second study,⁵ medical librarians used MEDLINE and textbooks to answer 86 questions from family physicians. The MEDLINE searches took a mean of 27 minutes, and textbook searches took a mean of 6 minutes. Search results answered 54% of the clinical questions completely or nearly completely. Physicians estimated that the answers would have a “major” or “fairly major” impact on practice for 35% of their questions. MEDLINE searches provided answers to 43% of the questions, while textbook searches provided answers for an additional 11%.

Many physicians do not have the searching skills or access to the range of knowledge resources that librarians use. Even if they did, they do not take the time to conduct such searches during patient care. One study¹ found that physicians spent less than 2 minutes on average seeking an answer to a question. Thus, most clinical questions remain unanswered.

Electronic medical databases that provide answers directly (not just reference citations) may make it easier for clinicians to obtain answers at the point of care. We found no systematic evaluation of the capacity of such databases to answer clinical questions. We conducted this study to determine the extent to which current electronic medical databases can answer family physicians’ point-of-care clinical questions.

Methods

Database Selection

We solicited nominations for potentially suitable databases from multiple E-mail lists (including communities of family physicians [Family-L], medical informaticians [FAM-MED] and medical librarians [MEDLIB-L, MCMLA-L]) and through Web searches. A selection team consisting of 3 family physicians (J.S., D.W., B.E.) and a medical librarian (none of whom had financial relationships with any databases) determined whether the nominated databases met our inclusion criteria Table 1.

Clinical Questions

More than 1200 clinical questions had been previously collected from observations of family physicians during office practice.^1,5 These questions had been classified by typology (eg, Is test X indicated in situation Y?) and by topic (eg, dermatology).¹ We selected questions from these sources that were categorized among the most common typologies (8 of 68 typologies covering 50% of the questions) and the most common topics (7 of 62 topics covering 43% of the questions). These combinations of typologies and topics accounted for 272 (23%) of the 1204 questions.

If necessary, each question was translated by 2 physicians (B.A. and D.W. working together) to meet the following criteria: (1) clear enough to imagine an applicable clinical scenario, (2) answerable (ie, the question could theoretically be answered using clinical references without further patient data regardless of whether an answer was known to exist), (3) clinically relevant, and (4) true to the original question (ie, containing the information need and the modifying factors of the original question).

Each question was then independently proofread by at least 2 other physicians and translated again if necessary. Thirteen questions (5%) that did not meet these criteria after a second translation were dropped. Forty-seven questions (17%) that referred to information needs that could be adequately answered using the Physicians’ Desk Reference⁶ were dropped (eg, Are Paxil tablets scored?). The remaining 212 questions represented 8 typologies.¹ Two or 3 questions were randomly selected from each typology for a total of 20 questions Table 2.

Testing

Two family physicians with experience in computer searching (B.A., D.W.) independently searched for answers using each of the included databases. In the case of DynaMed, for which Dr Alper is the medical director, another family physician was substituted as a searcher, and Dr Alper had no input or control over the testing or arbitration process for answers from DynaMed. Testing took place in April and May 2000.

Searching was performed using computers with Pentium III processors with a 100 megabyte-per-second network connection to the Internet and server-mounted CD-ROMs.

Each searcher used the same 20 questions to evaluate each database. The order of evaluation of the databases was at the discretion of the searchers, but the testing of a database was completed before starting the testing of another database. Searchers became familiar with each database before testing it by using the 5 screening questions.

A maximum of 10 minutes was allowed per question. Each answer was rated as adequate or inadequate. An answer was considered adequate if it contained sufficient information to guide clinical practice. For example, for the question “How do I determine the cause of chronic pruritus?”, the answer from the University of Iowa Family Practice Handbook (www.vh.org/Providers/ClinRef/FPHandbook/Chapter13/01-13.html) was considered adequate, because it included clinically useful recommendations: History should include details about (1) any skin lesions preceding the pruritus; (2) history of weight loss, fatigue, fever, malaise; (3) any recent stress emotionally; and (4) recent medications and travel. Physical examination with emphasis on the skin and its appendages — xerosis, excoriation, lichenification, hydration. Laboratory tests as suggested by the PE, which may include CBC, ESR, fasting glucose, renal or liver function tests, hepatitis panel, thyroid tests, stool for parasites, CXR.

Sources that provided general recommendations without information that could specifically guide clinical practice were considered inadequate. For example: “The cause of generalized pruritus should be sought and corrected. If no skin disease is apparent, a systemic disorder or drug-related cause should be sought.” The searcher recorded the answer and the time it took to obtain it rounded to the nearest number of minutes (1-10).

Scoring and Arbitration

The 2 physician searchers judged the adequacy of the answers to each question for each database. If the searchers both found adequate answers, the result was accepted as adequate, and the average time required to find and interpret the answer was recorded. If neither searcher found an adequate answer, then the answer was deemed inadequate. If only one searcher found an adequate answer, the second searcher evaluated that answer. If the answer was acceptable to the second searcher, it was considered an adequate answer, and the time for the first searcher was recorded.

When searchers disagreed on the adequacy of identified answers, an arbitration panel consisting of 3 family physicians who were not affiliated with any of the databases met independently from the searchers to determine the adequacy of the answers by consensus.

Analysis

Our primary outcome was the proportion of questions adequately answered by each database. We calculated 95% confidence limits for the proportions of adequate answers.⁷ Means and medians were determined for the time to reach adequate answers for each database. We calculated the k statistic for the independent findings of the 2 searchers and for the results after the searchers reviewed each other’s searches.⁸ We combined the results of individual databases to determine the proportion of questions answered by all combinations of 2, 3, and 4 databases. We considered the question adequately answered if any of the individual databases adequately answered the question.

Results

Thirty-eight databases were nominated, and 24 did not meet our inclusion criteria Table W1.* Fourteen databases met the inclusion criteria Table 3 and were evaluated with the set of 20 questions (280 answer assessments) by 2 searchers. The Figure summarizes the process of evaluating the answers. The initial agreement between searchers was good k=0.69). Discussion between the searchers resolved 21 (52.5%) of the 40 discrepant answer assessments. These were due to inadequate searching or timing out (searching for 10 minutes) by one searcher, who agreed with the adequacy of the answer found by the other searcher. The agreement between searchers at this stage was excellent (k= 0.94).

The remaining 19 discrepant assessments (for which the searchers had different opinions regarding the adequacy of the answers identified) were referred to the arbitration panel for determination of the final results. Ten of these were deemed adequate.

Results for individual databases in rank order of proportion of questions answered followed by average time to identify adequate answers are reported in Table 3. The combination of STAT!Ref and MDConsult could answer 85% of our set of 20 questions. Four combinations of 2 databases (STAT!Ref and either MAXX, MDChoice.com, Primary Care Guidelines, or Medscape) could answer 80% of our questions. Two combinations of 3 databases (STAT!Ref, MDConsult, and either DynaMed or MAXX) could answer 90% of our questions. Combinations of 4 databases answered the most sample questions (95%, 19/20). These combinations consisted of STAT!Ref, DynaMed, MAXX, and either MDConsult or American Family Physician.

We also evaluated combinations of databases that were available at no cost. The combination of the 2 no-cost databases that answered the largest proportion of questions (75%) was DynaMed and American Family Physician. The greatest proportion of clinical questions that could be answered using the freely available sources was 80%, and this required the use of 3 databases (DynaMed, MDChoice.com, and American Family Physician).

Discussion

Our study suggests that individual databases can answer a considerable proportion of family physicians’ clinical questions. Combinations of currently available databases can answer 75% or more. The searches in this study were based on the combination of efforts of 2 experienced physician searchers. These results may not be replicable in the practice setting but do provide an objective best-case scenario assessment of the content of these databases.

The time required to obtain answers, while much less than searching for original articles, is still longer than the 2-minute average time spent by family physicians in the study by Ely and colleagues.¹ Our time estimates are not precise, as time was not the primary focus of our study. Time was only recorded in 1-minute intervals, so searches that took 10 seconds were recorded as 1 minute. Even so, the existence of median times to obtain adequate answers greater than 2 minutes suggests that these databases may require more time than most physicians will take to pursue answers during patient care.

This is the first study to systematically evaluate how many questions can be answered by electronic medical databases. The strengths of this study include the use of a standard set of common questions asked by family physicians, testing by 2 experienced family physician searchers, and a systematic replicable approach to the evaluation. The only similar study we identified was one in which Graber and coworkers⁹ used 10 clinical questions and tested a commercial site, 2 medical meta-lists, 4 general search engines, and 9 medicine-specific search engines to determine the efficiency of answering clinical questions on the Web. Different approaches answered from 0 to 6 of the 10 questions, but that study looked primarily at sites that were not generally designed for use in clinical practice.

Limitations

Our study was limited by the relatively small number of questions, causing wide confidence intervals. Some answers were present in the databases but not found despite the use of 2 searchers. For example, a database manager identified 2 answers that were not found but would have been considered adequate.

We accepted answers as adequate if, in our judgment, they offered a practical course of action. We did not attempt to determine whether the individual asking the question believed that the answer was adequate nor did we attempt to validate the accuracy or currency of answers using independent standards. Many of the answers were based on sources that were several years old, and few were based on explicit evidence-based criteria. Although we determined the adequacy of answers for clinical practice through formal mechanisms, an in vivo study in which the clinicians asking the questions determined the adequacy of their findings during patient care activities would provide a more accurate assessment.

Our study presents a static evaluation of a dynamic field. Over time, answers may be lost because of lack of maintenance of resource links or may be gained by addition of new materials. Our use of questions gathered several years ago may not accurately reflect the ability of databases to answer current questions, which may be more likely to reflect new tests and treatments.

Many of the databases were designed for purposes other than meeting clinical information needs at the point of care. Performance in this study does not reflect the capacity of these databases to address their stated purposes. For example, the Translating Research Into Practice (TRIP) database is an excellent resource for searches of a large collection of evidence-based resources. These resources are generally limited to summaries of studies with the highest methodologic quality. The TRIP database did not perform well in our study partly because most of our test questions (consistent with questions in clinical practice) cannot currently be answered using studies of the highest methodologic quality. Another example is Medical Matrix, which provides a search engine and annotated summaries for exploring the entire medical Internet and not just clinical reference information.

We did not study the costs involved in using the databases we evaluated, and these costs may have changed since our study was conducted. Most of the databases we included were free to use at the time of the study and at the time of this report. The 3 collections of textbooks required access fees. STAT!Ref, which scored the highest in our study, did so because we used the complete collection available to us through our institutional library. This collection would cost an individual $2189 annually at the time of our study. A starter library was available for $199 annually and would only answer 40% of the questions.

Context

Family physicians and other primary care providers treat patients who have a wide variety of syndromes and symptoms. Because of the scope and breadth of primary care, it is nearly impossible for a clinician to keep up with rapidly changing medical information.¹⁰

Connelly and colleagues¹¹ surveyed 126 family physicians and found they used the Physicians’ Desk Reference and colleagues much more often than Index Medicus or computer-based bibliographic retrieval systems. Research literature was used infrequently and rated among the lowest in terms of credibility, availability, searchability, understandability, and applicability. Physicians preferred sources that had low cost and were relevant to specific patient problems over sources that had higher quality.

Conclusions

Current databases can answer a considerable proportion of clinical questions but have not reached their potential for efficiency. It is our hope that as electronic medical databases mature, they will be able to bridge this gap and bring the research literature to the point of care in useful and practical ways. This study provides a snapshot of how far we have come and how far we need to go to meet these needs.

Acknowledgments

Funding for our study was provided by a grant from the American Academy of Family Physicians to support the Center for Family Medicine Science and from 2 Bureau of Health Professions Awards (DHHS 1-D14-HP-00029-01, DHHS 5 T32 HP10038) from the Health Resources and Services Administration to the Department of Family and Community Medicine at University of Missouri-Columbia. The authors would like to acknowledge Erik Lindbloom, MD, MSPH, for assisting with the database testing as a substitute searcher for B.A.; E. Diane Johnson, MLS, for assisting with the selection of databases for study inclusion; Robert Phillips, Jr., MD, MSPH, for arbitration of questions and answers for which the searchers did not reach agreement along with B.E. and J.S.; David Cravens, Erik Lindbloom, Kevin Kane, Jim Brillhart, and Mark Ebell for proofreading the questions for clarity, answerability, and clinical relevance; John Ely and Lee Chambliss for providing clinical questions from their observations; Mark Ebell, John Ely, Erik Lindbloom, Jerry Osheroff, Lee Chambliss, David Mehr, Robin Kruse, John Smucny, and many others for constructive criticism in the design of this study; and Steve Zweig for editorial review.

Family physicians and general internists report an average of 6 questions for each half-day of office practice,^1-3 and 70% of these questions remain unanswered. The 2 factors that significantly predict whether a physician will attempt to answer a clinical question are the physician’s belief that a definitive answer exists and the urgency of the patient’s problem.⁴

Gorman and colleagues³ reported that medical librarians found clear answers for 46% of 60 randomly selected questions from family physicians; 51% would affect practice. The medical librarians searched for an average of 43 minutes per question. In a second study,⁵ medical librarians used MEDLINE and textbooks to answer 86 questions from family physicians. The MEDLINE searches took a mean of 27 minutes, and textbook searches took a mean of 6 minutes. Search results answered 54% of the clinical questions completely or nearly completely. Physicians estimated that the answers would have a “major” or “fairly major” impact on practice for 35% of their questions. MEDLINE searches provided answers to 43% of the questions, while textbook searches provided answers for an additional 11%.

Many physicians do not have the searching skills or access to the range of knowledge resources that librarians use. Even if they did, they do not take the time to conduct such searches during patient care. One study¹ found that physicians spent less than 2 minutes on average seeking an answer to a question. Thus, most clinical questions remain unanswered.

Electronic medical databases that provide answers directly (not just reference citations) may make it easier for clinicians to obtain answers at the point of care. We found no systematic evaluation of the capacity of such databases to answer clinical questions. We conducted this study to determine the extent to which current electronic medical databases can answer family physicians’ point-of-care clinical questions.

Methods

Database Selection

We solicited nominations for potentially suitable databases from multiple E-mail lists (including communities of family physicians [Family-L], medical informaticians [FAM-MED] and medical librarians [MEDLIB-L, MCMLA-L]) and through Web searches. A selection team consisting of 3 family physicians (J.S., D.W., B.E.) and a medical librarian (none of whom had financial relationships with any databases) determined whether the nominated databases met our inclusion criteria Table 1.

Clinical Questions

More than 1200 clinical questions had been previously collected from observations of family physicians during office practice.^1,5 These questions had been classified by typology (eg, Is test X indicated in situation Y?) and by topic (eg, dermatology).¹ We selected questions from these sources that were categorized among the most common typologies (8 of 68 typologies covering 50% of the questions) and the most common topics (7 of 62 topics covering 43% of the questions). These combinations of typologies and topics accounted for 272 (23%) of the 1204 questions.

If necessary, each question was translated by 2 physicians (B.A. and D.W. working together) to meet the following criteria: (1) clear enough to imagine an applicable clinical scenario, (2) answerable (ie, the question could theoretically be answered using clinical references without further patient data regardless of whether an answer was known to exist), (3) clinically relevant, and (4) true to the original question (ie, containing the information need and the modifying factors of the original question).

Each question was then independently proofread by at least 2 other physicians and translated again if necessary. Thirteen questions (5%) that did not meet these criteria after a second translation were dropped. Forty-seven questions (17%) that referred to information needs that could be adequately answered using the Physicians’ Desk Reference⁶ were dropped (eg, Are Paxil tablets scored?). The remaining 212 questions represented 8 typologies.¹ Two or 3 questions were randomly selected from each typology for a total of 20 questions Table 2.

Testing

Two family physicians with experience in computer searching (B.A., D.W.) independently searched for answers using each of the included databases. In the case of DynaMed, for which Dr Alper is the medical director, another family physician was substituted as a searcher, and Dr Alper had no input or control over the testing or arbitration process for answers from DynaMed. Testing took place in April and May 2000.

Searching was performed using computers with Pentium III processors with a 100 megabyte-per-second network connection to the Internet and server-mounted CD-ROMs.

Each searcher used the same 20 questions to evaluate each database. The order of evaluation of the databases was at the discretion of the searchers, but the testing of a database was completed before starting the testing of another database. Searchers became familiar with each database before testing it by using the 5 screening questions.

A maximum of 10 minutes was allowed per question. Each answer was rated as adequate or inadequate. An answer was considered adequate if it contained sufficient information to guide clinical practice. For example, for the question “How do I determine the cause of chronic pruritus?”, the answer from the University of Iowa Family Practice Handbook (www.vh.org/Providers/ClinRef/FPHandbook/Chapter13/01-13.html) was considered adequate, because it included clinically useful recommendations: History should include details about (1) any skin lesions preceding the pruritus; (2) history of weight loss, fatigue, fever, malaise; (3) any recent stress emotionally; and (4) recent medications and travel. Physical examination with emphasis on the skin and its appendages — xerosis, excoriation, lichenification, hydration. Laboratory tests as suggested by the PE, which may include CBC, ESR, fasting glucose, renal or liver function tests, hepatitis panel, thyroid tests, stool for parasites, CXR.

Sources that provided general recommendations without information that could specifically guide clinical practice were considered inadequate. For example: “The cause of generalized pruritus should be sought and corrected. If no skin disease is apparent, a systemic disorder or drug-related cause should be sought.” The searcher recorded the answer and the time it took to obtain it rounded to the nearest number of minutes (1-10).

Scoring and Arbitration

The 2 physician searchers judged the adequacy of the answers to each question for each database. If the searchers both found adequate answers, the result was accepted as adequate, and the average time required to find and interpret the answer was recorded. If neither searcher found an adequate answer, then the answer was deemed inadequate. If only one searcher found an adequate answer, the second searcher evaluated that answer. If the answer was acceptable to the second searcher, it was considered an adequate answer, and the time for the first searcher was recorded.

When searchers disagreed on the adequacy of identified answers, an arbitration panel consisting of 3 family physicians who were not affiliated with any of the databases met independently from the searchers to determine the adequacy of the answers by consensus.

Analysis

Our primary outcome was the proportion of questions adequately answered by each database. We calculated 95% confidence limits for the proportions of adequate answers.⁷ Means and medians were determined for the time to reach adequate answers for each database. We calculated the k statistic for the independent findings of the 2 searchers and for the results after the searchers reviewed each other’s searches.⁸ We combined the results of individual databases to determine the proportion of questions answered by all combinations of 2, 3, and 4 databases. We considered the question adequately answered if any of the individual databases adequately answered the question.

Results

Thirty-eight databases were nominated, and 24 did not meet our inclusion criteria Table W1.* Fourteen databases met the inclusion criteria Table 3 and were evaluated with the set of 20 questions (280 answer assessments) by 2 searchers. The Figure summarizes the process of evaluating the answers. The initial agreement between searchers was good k=0.69). Discussion between the searchers resolved 21 (52.5%) of the 40 discrepant answer assessments. These were due to inadequate searching or timing out (searching for 10 minutes) by one searcher, who agreed with the adequacy of the answer found by the other searcher. The agreement between searchers at this stage was excellent (k= 0.94).

The remaining 19 discrepant assessments (for which the searchers had different opinions regarding the adequacy of the answers identified) were referred to the arbitration panel for determination of the final results. Ten of these were deemed adequate.

Results for individual databases in rank order of proportion of questions answered followed by average time to identify adequate answers are reported in Table 3. The combination of STAT!Ref and MDConsult could answer 85% of our set of 20 questions. Four combinations of 2 databases (STAT!Ref and either MAXX, MDChoice.com, Primary Care Guidelines, or Medscape) could answer 80% of our questions. Two combinations of 3 databases (STAT!Ref, MDConsult, and either DynaMed or MAXX) could answer 90% of our questions. Combinations of 4 databases answered the most sample questions (95%, 19/20). These combinations consisted of STAT!Ref, DynaMed, MAXX, and either MDConsult or American Family Physician.

We also evaluated combinations of databases that were available at no cost. The combination of the 2 no-cost databases that answered the largest proportion of questions (75%) was DynaMed and American Family Physician. The greatest proportion of clinical questions that could be answered using the freely available sources was 80%, and this required the use of 3 databases (DynaMed, MDChoice.com, and American Family Physician).

Discussion

Our study suggests that individual databases can answer a considerable proportion of family physicians’ clinical questions. Combinations of currently available databases can answer 75% or more. The searches in this study were based on the combination of efforts of 2 experienced physician searchers. These results may not be replicable in the practice setting but do provide an objective best-case scenario assessment of the content of these databases.

The time required to obtain answers, while much less than searching for original articles, is still longer than the 2-minute average time spent by family physicians in the study by Ely and colleagues.¹ Our time estimates are not precise, as time was not the primary focus of our study. Time was only recorded in 1-minute intervals, so searches that took 10 seconds were recorded as 1 minute. Even so, the existence of median times to obtain adequate answers greater than 2 minutes suggests that these databases may require more time than most physicians will take to pursue answers during patient care.

This is the first study to systematically evaluate how many questions can be answered by electronic medical databases. The strengths of this study include the use of a standard set of common questions asked by family physicians, testing by 2 experienced family physician searchers, and a systematic replicable approach to the evaluation. The only similar study we identified was one in which Graber and coworkers⁹ used 10 clinical questions and tested a commercial site, 2 medical meta-lists, 4 general search engines, and 9 medicine-specific search engines to determine the efficiency of answering clinical questions on the Web. Different approaches answered from 0 to 6 of the 10 questions, but that study looked primarily at sites that were not generally designed for use in clinical practice.

Limitations

Our study was limited by the relatively small number of questions, causing wide confidence intervals. Some answers were present in the databases but not found despite the use of 2 searchers. For example, a database manager identified 2 answers that were not found but would have been considered adequate.

We accepted answers as adequate if, in our judgment, they offered a practical course of action. We did not attempt to determine whether the individual asking the question believed that the answer was adequate nor did we attempt to validate the accuracy or currency of answers using independent standards. Many of the answers were based on sources that were several years old, and few were based on explicit evidence-based criteria. Although we determined the adequacy of answers for clinical practice through formal mechanisms, an in vivo study in which the clinicians asking the questions determined the adequacy of their findings during patient care activities would provide a more accurate assessment.

Our study presents a static evaluation of a dynamic field. Over time, answers may be lost because of lack of maintenance of resource links or may be gained by addition of new materials. Our use of questions gathered several years ago may not accurately reflect the ability of databases to answer current questions, which may be more likely to reflect new tests and treatments.

Many of the databases were designed for purposes other than meeting clinical information needs at the point of care. Performance in this study does not reflect the capacity of these databases to address their stated purposes. For example, the Translating Research Into Practice (TRIP) database is an excellent resource for searches of a large collection of evidence-based resources. These resources are generally limited to summaries of studies with the highest methodologic quality. The TRIP database did not perform well in our study partly because most of our test questions (consistent with questions in clinical practice) cannot currently be answered using studies of the highest methodologic quality. Another example is Medical Matrix, which provides a search engine and annotated summaries for exploring the entire medical Internet and not just clinical reference information.

We did not study the costs involved in using the databases we evaluated, and these costs may have changed since our study was conducted. Most of the databases we included were free to use at the time of the study and at the time of this report. The 3 collections of textbooks required access fees. STAT!Ref, which scored the highest in our study, did so because we used the complete collection available to us through our institutional library. This collection would cost an individual $2189 annually at the time of our study. A starter library was available for $199 annually and would only answer 40% of the questions.

Context

Family physicians and other primary care providers treat patients who have a wide variety of syndromes and symptoms. Because of the scope and breadth of primary care, it is nearly impossible for a clinician to keep up with rapidly changing medical information.¹⁰

Connelly and colleagues¹¹ surveyed 126 family physicians and found they used the Physicians’ Desk Reference and colleagues much more often than Index Medicus or computer-based bibliographic retrieval systems. Research literature was used infrequently and rated among the lowest in terms of credibility, availability, searchability, understandability, and applicability. Physicians preferred sources that had low cost and were relevant to specific patient problems over sources that had higher quality.

Conclusions

Current databases can answer a considerable proportion of clinical questions but have not reached their potential for efficiency. It is our hope that as electronic medical databases mature, they will be able to bridge this gap and bring the research literature to the point of care in useful and practical ways. This study provides a snapshot of how far we have come and how far we need to go to meet these needs.

Acknowledgments

Funding for our study was provided by a grant from the American Academy of Family Physicians to support the Center for Family Medicine Science and from 2 Bureau of Health Professions Awards (DHHS 1-D14-HP-00029-01, DHHS 5 T32 HP10038) from the Health Resources and Services Administration to the Department of Family and Community Medicine at University of Missouri-Columbia. The authors would like to acknowledge Erik Lindbloom, MD, MSPH, for assisting with the database testing as a substitute searcher for B.A.; E. Diane Johnson, MLS, for assisting with the selection of databases for study inclusion; Robert Phillips, Jr., MD, MSPH, for arbitration of questions and answers for which the searchers did not reach agreement along with B.E. and J.S.; David Cravens, Erik Lindbloom, Kevin Kane, Jim Brillhart, and Mark Ebell for proofreading the questions for clarity, answerability, and clinical relevance; John Ely and Lee Chambliss for providing clinical questions from their observations; Mark Ebell, John Ely, Erik Lindbloom, Jerry Osheroff, Lee Chambliss, David Mehr, Robin Kruse, John Smucny, and many others for constructive criticism in the design of this study; and Steve Zweig for editorial review.

Acute bronchitis is characterized by cough associated with other symptoms of upper respiratory infection. Although this condition is self-limited, most patients feel ill, and many do not perform their usual activities. The optimal treatment for this common condition in patients who do not have underlying pulmonary disease is not clear. Clinicians often prescribe antibiotics,^1,2 in spite of the fact that they are of little overall benefit.^3,4 It is important to examine the effectiveness of alternative approaches.

b2-agonists have been proposed, because healthy patients have impaired airflow when infected with pathogens known to cause acute bronchitis.^5-7 Also, cough is the primary symptom for some patients who have asthma,⁸ and many of these patients benefit from b2-agonists.⁹ b2-agonists are effective in reducing cough due to other causes, such as bronchoscopy¹⁰ and intravenous fentany^1,11

We conducted this systematic review to determine whether b2-agonists are effective for patients who have acute bronchitis without underlying pulmonary disease. If b2-agonists are effective, then they should be more widely used; only a minority of US family physicians currently prescribe them for acute bronchitis.^2,12

Methods

We attempted to locate all controlled trials that compared b2-agonists with placebo or an alternative treatment in patients older than 2 years who presented with acute bronchitis or acute cough without a clear etiology (eg, pneumonia, pertussis, or sinusitis). We included patients with acute cough, because the clinical definition of acute bronchitis is not standardized. Textbooks^13-15 and clinician studies^16,17 instruct that cough in association with an acute respiratory infection is required for a diagnosis; otherwise, there are differing criteria regarding the need for other symptoms and signs, such as dyspnea, abnormal chest findings, and sputum.

We searched MEDLINE (1966-2000), EMBASE (1974-2000), and The Cochrane Library (through August 2000) using the key words “bronchitis” or “cough”, together with the terms “adrenergic beta-agonist (exp),” “bronchodilator agents (exp),” “sympathomimetic (exp),” “albuterol,” “salbutamol,” “bitolterol,” “isoetharine,” “metaproterenol,” “pirbuterol,” “salmeterol,” “terbutaline,” “fenoterol,” “formoterol,” or “procaterol” (note that albuterol and salbutamol are the same compound). We also searched conference proceedings databases (Inside Conferences, 1993-99; Conference Papers Index, 1973-99); the System for Information on Grey Literature in Europe database (1980-2000); the reference lists of retrieved articles, review articles, and textbooks; and the Science Citation Index (1990-2000). Finally, we wrote to all US manufacturers of brand name b2-agonists. There were no language restrictions in our search.

Two investigators (C.F., J.S.) independently reviewed all the retrieved titles and abstracts. Studies selected by either investigator as possibly meeting the inclusion criteria were retrieved in their entirety. One investigator (J.S.) then deleted the journal of publication, title, authors, affiliations, and results sections of each study that met the inclusion criteria, and compiled a list of all the reported outcomes. The list of outcomes was forwarded to the other 3 investigators who independently, and then through discussion, determined which outcomes would be included in our review. The main criterion for selection was that the outcome should be directly important to patients. The same 3 investigators then graded the quality of each study using the 5-point Jadad scale, with points given for method of randomization (0-2), adequacy of blinding (0-2), and description of withdrawals (0-1).¹⁸ The Jadad scale is a validated, well-accepted, and frequently used quality assessment scale. Agreement on quality was assessed with a k score, and disagreements were resolved by discussion. Trials were excluded if all investigators agreed that the trial did not meet our inclusion criteria. The remaining articles in their entirety were then distributed to all investigators, each of whom independently extracted data for the selected outcomes. Disagreements were resolved by discussion. We attempted to contact authors to obtain missing data.

Summary statistics were calculated using Review Manager 4.1 software (Update Software, Oxford, England). We used fixed effects models for outcomes without statistically significant heterogeneity (at P <.10) and random effects models for outcomes with significant heterogeneity. For dichotomous outcomes, we reported relative risks (RRs), absolute risk reductions, and numbers needed to treat (NNTs), and for continuous outcomes, standardized mean differences (SMD). We considered a level of P less than .05 to be statistically significant.

Results

Included Studies

The major characteristics of the trials are shown in Table 1. We included 6 controlled trials comparing b2-agonists and placebo,^19-24 and one trial comparing a b2-agonist with erythromycin.²⁵ A trial comparing a b2-agonist with placebo in children²⁶ was excluded because all participants had recurrent cough and the mean duration of cough (8 weeks) was much longer than the maximum of 30 days used in the other trials.

All trials enrolled patients that presented to primary care settings. The stated diagnoses were “acute bronchitis,”^21,22,25 “acute cough,”^19,20 and “acute transient cough.”^23,24 Both trials in children excluded participants with abnormal lung examinations¹⁹ or “with bronchial obstruction needing bronchodilating medication.”²³ None of the adult trials excluded patients with wheezing; the percentage with wheezing ranged from 20% to 44% in the 4 trials that mentioned it. All adult trials included both smokers an nonsmokers.

The only trial that mentioned how well patients adhered to study medications²⁵ reported more than 95% compliance for both groups. Regarding co-interventions, 3 trials prohibited other antitussives^19,23,24 ; 3 trials allowed them and recorded their use as an outcome^20,21,25 ; and one trial did not mention co-interventions. ²² One trial prohibited the use of antibiotics²⁴ ; other trials comparing b2-agonists to placebo allowed the use of antibiotics at the discretion of the clinician (except as noted for the 1994 study by Hueston²¹). No trials were clearly sponsored by pharmaceutical manufacturers, but the medications were supplied free of charge by manufacturers in 3 studies.^19,22,24

The quality of the trials varied from 2 to 4 on the Jadad scale Table 1. The k score for reviewers’ quality scores was 0.27, indicating only fair agreement. The majority of the disagreements related to different initial interpretations of the adequacy of blinding and description of withdrawals. These differences were resolved with further discussion.

Data Analysis

The clinical heterogeneity of the trials was so great that examining them as a single group did not seem reasonable. Therefore, we initially examined the trials as follows: (1) those in children, (2) those in adults comparing b2-agonists with placebo, and (3) those in adults comparing b2-agonists with erythromycin. We then combined the data from the trial that compared a b2-agonist with erythromycin with that from the other trials in adults in a secondary analysis.

Trials in Children

Neither trial involving children demonstrated any benefits from albuterol Table 2. Combining the daily cough scores for days 1 to 3 for these trials revealed a trend toward worse scores in the group receiving albuterol Table 3. The results from the 2 trials were homogeneous.

Trials in Adults Comparing b2-agonists with Placebo

The results of the placebo-controlled trials in adults were mixed; one trial found no benefit from b2-agonists, and 3 found at least one benefit. Combining the daily cough severity scores for the 3 trials that included this outcome^20,22,24showed a small nonsignificant trend toward improvement on all days. The results from the individual trials were heterogeneous for day 1 and homogeneous for the other days.

Combining data from the trials that examined persistence of symptoms after a full 7 days of treatment^20-22 yielded no significant difference in presence of cough or night cough Table 4. Combined data also do not show a difference regarding the presence of a productive cough after 7 days or a difference regarding whether patients were working after 4 days. There was significant heterogeneity for 3 of the 4 dichotomous outcomes: cough, productive cough, and return to work.

Trials in Adults Comparing b2-agonists with Erythromycin

In the 1994 Hueston study,²¹ patients given albuterol were less likely to have a cough or a productive cough after 7 days than those given erythromycin, but there were no differences in the presence of night cough after 7 days or in mean days until improvement in cough, well-being, or return to work or normal activities. When the data from this study are combined with that from the other adult trials, there are no significant differences regarding presence after 7 days of cough (RR=0.77; 95% confidence interval [CI], 0.54-1.09), productive cough (RR=0.66; 95% CI, 0.35-1.25), or night cough (RR=0.85; 95% CI, 0.57-1.26).

Adverse Effects

In the trials in children, 11% of the patients given albuterol had shaking or tremor versus 0% given placebo or only dextromethorphan (RR=6.76; 95% CI, 0.86-53.18; NNT=9; 95% CI, 5-100); the results were homogeneous. There were no differences regarding other adverse effects in the trials in children. In the adult trials, patients given b2-agonists were more likely to report tremor, shaking, or nervousness; the percentage of patients having these side effects in the 3 trials that reported specific side effects ranged from 35% to 67% versus control rates of 0% to 23% (RR=7.94; 95% CI, 1.17-53.94; NNT=2.3; 95% CI, 2-3). These data are from the trials that used inhaled fenoterol and oral albuterol.^20,22,25 However, in the 1991 Hueston study,²⁵ only 9% of the patients given inhaled albuterol reported any side effects; therefore, there is considerable heterogeneity among the results of the individual trials. There were no significant differences regarding other adverse effects between the b2-agonist group and control groups as a whole, but the trial comparing albuterol with erythromycin noted more gastrointestinal side effects in the erythromycin group (NNT=3; 95% CI, 2-8).

Subgroup Analyses

In the study by Melbye and colleagues,²² the subgroup of patients with evidence of airway obstruction (defined as wheezing on initial examination, a forced expiratory volume in 1 second of <80% predicted, or a positive response to a methacholine challenge test) who were given fenoterol had lower symptom scores beginning at day 2 than those in this subgroup who were given placebo. This was also true for the smaller subgroup that just had wheezing, but no difference was noted for patients with a normal lung examination. No other trial did a subgroup analysis limited to patients with evidence of airflow obstruction. The 1994 Hueston study²¹ reported that among patients given albuterol, those with wheezing were slightly less likely to be coughing after 7 days than those without wheezing, but the difference was not statistically significant.

Melbye and coworkers²² found that patients who smoked or had also received antibiotics had greater reductions in total symptom scores on day 7 if given fenoterol. Smokers had similar responses to nonsmokers in the studies by Hueston.^21,25 Littenberg and colleagues²⁰ found that patients given erythromycin trended toward lower cough severity scores if given albuterol instead of placebo, and patients not given erythromycin showed a trend toward higher scores if given albuterol. The 1994 Hueston study²¹ reported that the differences between the groups given and not given albuterol persisted after stratification by erythromycin use.

Discussion

The findings from our review do not support the routine use of b2-agonists for patients who do not have underlying pulmonary disease and present with an acute cough or acute bronchitis. These results must be interpreted in light of the patients that were enrolled in the trials. In particular, because the 2 trials in children excluded patients who were wheezing, the utility of b2-agonists in children with acute cough and evidence of airway obstruction is unknown. b2-agonists do lead to modest short-term improvements in clinical scores in children younger than 2 years who have bronchiolitis.²⁷

The discordant results seen in the trials of adults may reflect different patient populations. Although the inclusion criteria were similar in these trials, more patients were wheezing on initial examination in the Hueston studies^21,25 than in the studies by Littenberg and coworkers²⁰ or Melbye and colleagues.²² Wheezing in unforced expiration is a specific finding for airflow obstruction²⁸; and therefore, more patients in the Hueston trials^21,25 were likely to have had obstruction than in Littenberg and coworkers’ study²⁰ (and since the lungs were auscultated in forced expiration in the latter trial, the actual number with airflow obstruction may have been even less than indicated). The fact that only the subgroup with airway obstruction improved with b2-agonists in the trial by Melbye and colleagues²² reflects the possible importance of this baseline characteristic.

Limitations

Our review has some limitations. Although it includes all of the available data regarding the effectiveness of b2-agonists for patients with acute bronchitis or acute cough, the number of studies and total number of patients included are small. Therefore, our review has limited power to detect differences between patients who were and were not given b2-agonists. In the combined data of trials in adults, there was a trend toward improvements regarding cough, productive cough, night cough, and return to work, but these differences did not reach statistical significance. The midpoint estimates for the relative risk reductions range from 14% to 24% for these outcomes, but all overlap 0. There was also a clinically minor and statistically nonsignificant trend toward lower daily cough severity scores in patients randomized to the b2-agonists.

The studies were also all of a short duration. There is no information as to whether treatment with b2-agonists would alter outcomes beyond 3 to 7 days. This is an important omission, because many patients in these studies were still bothered by symptoms at the end of the trials.

Only 2 studies evaluated inhaled b2-agonists, which would currently be the most likely formulation used in adults and older children. Neither of these studies used spacing devices. The delivery of the medicine may have been suboptimal and resulted in less benefit than might have been seen had spacers been used.

Overall, the quality of the trials was fair to good . There may have been additional biases, however, because most of the trials had unequal distribution of co-interventions and did not record compliance with study medications. Also, even though the studies were all double-blinded, the fact that the majority of the patients in one trial knew which study medication they had been given indicates that the blinding may not have been adequate in these studies because of the taste or side effects of the study medications.

Conclusions

Our review highlights the gaps in evidence regarding the utility of b2-agonists in the treatment of acute cough and acute bronchitis in patients without underlying pulmonary disease. Although there is a possibility that these agents may be useful, additional data demonstrating benefit is required before they can be routinely recommended. There is a particular need for identifying clinical characteristics that can predict which patients might benefit. For example, there is a complete lack of data in children older than 2 years who have signs of airway obstruction. More evidence on the risk-benefit ratio of b2-agonists in adults with clinical signs of airflow limitation is also necessary. Additional areas of useful research would be in evaluating long-acting b2-agonists (because of ease of adherence), in evaluating the benefits of inhaled b2-agonists with spacing devices, and in comparing b2-agonists with other symptomatic treatments.

Acknowledgments

We thank Bill Hueston, Ben Littenberg, Hasse Melbye, and Peter Rowe for providing unpublished information; Bill Grant for assistance with statistics; and Ron D’Souza and Steve MacDonald of the Cochrane Collaboration and Bette Jean Ingui for assistance with database searches.

Acute bronchitis is characterized by cough associated with other symptoms of upper respiratory infection. Although this condition is self-limited, most patients feel ill, and many do not perform their usual activities. The optimal treatment for this common condition in patients who do not have underlying pulmonary disease is not clear. Clinicians often prescribe antibiotics,^1,2 in spite of the fact that they are of little overall benefit.^3,4 It is important to examine the effectiveness of alternative approaches.

b2-agonists have been proposed, because healthy patients have impaired airflow when infected with pathogens known to cause acute bronchitis.^5-7 Also, cough is the primary symptom for some patients who have asthma,⁸ and many of these patients benefit from b2-agonists.⁹ b2-agonists are effective in reducing cough due to other causes, such as bronchoscopy¹⁰ and intravenous fentany^1,11

We conducted this systematic review to determine whether b2-agonists are effective for patients who have acute bronchitis without underlying pulmonary disease. If b2-agonists are effective, then they should be more widely used; only a minority of US family physicians currently prescribe them for acute bronchitis.^2,12

Methods

We attempted to locate all controlled trials that compared b2-agonists with placebo or an alternative treatment in patients older than 2 years who presented with acute bronchitis or acute cough without a clear etiology (eg, pneumonia, pertussis, or sinusitis). We included patients with acute cough, because the clinical definition of acute bronchitis is not standardized. Textbooks^13-15 and clinician studies^16,17 instruct that cough in association with an acute respiratory infection is required for a diagnosis; otherwise, there are differing criteria regarding the need for other symptoms and signs, such as dyspnea, abnormal chest findings, and sputum.

We searched MEDLINE (1966-2000), EMBASE (1974-2000), and The Cochrane Library (through August 2000) using the key words “bronchitis” or “cough”, together with the terms “adrenergic beta-agonist (exp),” “bronchodilator agents (exp),” “sympathomimetic (exp),” “albuterol,” “salbutamol,” “bitolterol,” “isoetharine,” “metaproterenol,” “pirbuterol,” “salmeterol,” “terbutaline,” “fenoterol,” “formoterol,” or “procaterol” (note that albuterol and salbutamol are the same compound). We also searched conference proceedings databases (Inside Conferences, 1993-99; Conference Papers Index, 1973-99); the System for Information on Grey Literature in Europe database (1980-2000); the reference lists of retrieved articles, review articles, and textbooks; and the Science Citation Index (1990-2000). Finally, we wrote to all US manufacturers of brand name b2-agonists. There were no language restrictions in our search.

Two investigators (C.F., J.S.) independently reviewed all the retrieved titles and abstracts. Studies selected by either investigator as possibly meeting the inclusion criteria were retrieved in their entirety. One investigator (J.S.) then deleted the journal of publication, title, authors, affiliations, and results sections of each study that met the inclusion criteria, and compiled a list of all the reported outcomes. The list of outcomes was forwarded to the other 3 investigators who independently, and then through discussion, determined which outcomes would be included in our review. The main criterion for selection was that the outcome should be directly important to patients. The same 3 investigators then graded the quality of each study using the 5-point Jadad scale, with points given for method of randomization (0-2), adequacy of blinding (0-2), and description of withdrawals (0-1).¹⁸ The Jadad scale is a validated, well-accepted, and frequently used quality assessment scale. Agreement on quality was assessed with a k score, and disagreements were resolved by discussion. Trials were excluded if all investigators agreed that the trial did not meet our inclusion criteria. The remaining articles in their entirety were then distributed to all investigators, each of whom independently extracted data for the selected outcomes. Disagreements were resolved by discussion. We attempted to contact authors to obtain missing data.

Summary statistics were calculated using Review Manager 4.1 software (Update Software, Oxford, England). We used fixed effects models for outcomes without statistically significant heterogeneity (at P <.10) and random effects models for outcomes with significant heterogeneity. For dichotomous outcomes, we reported relative risks (RRs), absolute risk reductions, and numbers needed to treat (NNTs), and for continuous outcomes, standardized mean differences (SMD). We considered a level of P less than .05 to be statistically significant.

Results

Included Studies

The major characteristics of the trials are shown in Table 1. We included 6 controlled trials comparing b2-agonists and placebo,^19-24 and one trial comparing a b2-agonist with erythromycin.²⁵ A trial comparing a b2-agonist with placebo in children²⁶ was excluded because all participants had recurrent cough and the mean duration of cough (8 weeks) was much longer than the maximum of 30 days used in the other trials.

All trials enrolled patients that presented to primary care settings. The stated diagnoses were “acute bronchitis,”^21,22,25 “acute cough,”^19,20 and “acute transient cough.”^23,24 Both trials in children excluded participants with abnormal lung examinations¹⁹ or “with bronchial obstruction needing bronchodilating medication.”²³ None of the adult trials excluded patients with wheezing; the percentage with wheezing ranged from 20% to 44% in the 4 trials that mentioned it. All adult trials included both smokers an nonsmokers.

The only trial that mentioned how well patients adhered to study medications²⁵ reported more than 95% compliance for both groups. Regarding co-interventions, 3 trials prohibited other antitussives^19,23,24 ; 3 trials allowed them and recorded their use as an outcome^20,21,25 ; and one trial did not mention co-interventions. ²² One trial prohibited the use of antibiotics²⁴ ; other trials comparing b2-agonists to placebo allowed the use of antibiotics at the discretion of the clinician (except as noted for the 1994 study by Hueston²¹). No trials were clearly sponsored by pharmaceutical manufacturers, but the medications were supplied free of charge by manufacturers in 3 studies.^19,22,24

The quality of the trials varied from 2 to 4 on the Jadad scale Table 1. The k score for reviewers’ quality scores was 0.27, indicating only fair agreement. The majority of the disagreements related to different initial interpretations of the adequacy of blinding and description of withdrawals. These differences were resolved with further discussion.

Data Analysis

The clinical heterogeneity of the trials was so great that examining them as a single group did not seem reasonable. Therefore, we initially examined the trials as follows: (1) those in children, (2) those in adults comparing b2-agonists with placebo, and (3) those in adults comparing b2-agonists with erythromycin. We then combined the data from the trial that compared a b2-agonist with erythromycin with that from the other trials in adults in a secondary analysis.

Trials in Children

Neither trial involving children demonstrated any benefits from albuterol Table 2. Combining the daily cough scores for days 1 to 3 for these trials revealed a trend toward worse scores in the group receiving albuterol Table 3. The results from the 2 trials were homogeneous.

Trials in Adults Comparing b2-agonists with Placebo

The results of the placebo-controlled trials in adults were mixed; one trial found no benefit from b2-agonists, and 3 found at least one benefit. Combining the daily cough severity scores for the 3 trials that included this outcome^20,22,24showed a small nonsignificant trend toward improvement on all days. The results from the individual trials were heterogeneous for day 1 and homogeneous for the other days.

Combining data from the trials that examined persistence of symptoms after a full 7 days of treatment^20-22 yielded no significant difference in presence of cough or night cough Table 4. Combined data also do not show a difference regarding the presence of a productive cough after 7 days or a difference regarding whether patients were working after 4 days. There was significant heterogeneity for 3 of the 4 dichotomous outcomes: cough, productive cough, and return to work.

Trials in Adults Comparing b2-agonists with Erythromycin

In the 1994 Hueston study,²¹ patients given albuterol were less likely to have a cough or a productive cough after 7 days than those given erythromycin, but there were no differences in the presence of night cough after 7 days or in mean days until improvement in cough, well-being, or return to work or normal activities. When the data from this study are combined with that from the other adult trials, there are no significant differences regarding presence after 7 days of cough (RR=0.77; 95% confidence interval [CI], 0.54-1.09), productive cough (RR=0.66; 95% CI, 0.35-1.25), or night cough (RR=0.85; 95% CI, 0.57-1.26).

Adverse Effects

In the trials in children, 11% of the patients given albuterol had shaking or tremor versus 0% given placebo or only dextromethorphan (RR=6.76; 95% CI, 0.86-53.18; NNT=9; 95% CI, 5-100); the results were homogeneous. There were no differences regarding other adverse effects in the trials in children. In the adult trials, patients given b2-agonists were more likely to report tremor, shaking, or nervousness; the percentage of patients having these side effects in the 3 trials that reported specific side effects ranged from 35% to 67% versus control rates of 0% to 23% (RR=7.94; 95% CI, 1.17-53.94; NNT=2.3; 95% CI, 2-3). These data are from the trials that used inhaled fenoterol and oral albuterol.^20,22,25 However, in the 1991 Hueston study,²⁵ only 9% of the patients given inhaled albuterol reported any side effects; therefore, there is considerable heterogeneity among the results of the individual trials. There were no significant differences regarding other adverse effects between the b2-agonist group and control groups as a whole, but the trial comparing albuterol with erythromycin noted more gastrointestinal side effects in the erythromycin group (NNT=3; 95% CI, 2-8).

Subgroup Analyses

In the study by Melbye and colleagues,²² the subgroup of patients with evidence of airway obstruction (defined as wheezing on initial examination, a forced expiratory volume in 1 second of <80% predicted, or a positive response to a methacholine challenge test) who were given fenoterol had lower symptom scores beginning at day 2 than those in this subgroup who were given placebo. This was also true for the smaller subgroup that just had wheezing, but no difference was noted for patients with a normal lung examination. No other trial did a subgroup analysis limited to patients with evidence of airflow obstruction. The 1994 Hueston study²¹ reported that among patients given albuterol, those with wheezing were slightly less likely to be coughing after 7 days than those without wheezing, but the difference was not statistically significant.

Melbye and coworkers²² found that patients who smoked or had also received antibiotics had greater reductions in total symptom scores on day 7 if given fenoterol. Smokers had similar responses to nonsmokers in the studies by Hueston.^21,25 Littenberg and colleagues²⁰ found that patients given erythromycin trended toward lower cough severity scores if given albuterol instead of placebo, and patients not given erythromycin showed a trend toward higher scores if given albuterol. The 1994 Hueston study²¹ reported that the differences between the groups given and not given albuterol persisted after stratification by erythromycin use.

Discussion

The findings from our review do not support the routine use of b2-agonists for patients who do not have underlying pulmonary disease and present with an acute cough or acute bronchitis. These results must be interpreted in light of the patients that were enrolled in the trials. In particular, because the 2 trials in children excluded patients who were wheezing, the utility of b2-agonists in children with acute cough and evidence of airway obstruction is unknown. b2-agonists do lead to modest short-term improvements in clinical scores in children younger than 2 years who have bronchiolitis.²⁷

The discordant results seen in the trials of adults may reflect different patient populations. Although the inclusion criteria were similar in these trials, more patients were wheezing on initial examination in the Hueston studies^21,25 than in the studies by Littenberg and coworkers²⁰ or Melbye and colleagues.²² Wheezing in unforced expiration is a specific finding for airflow obstruction²⁸; and therefore, more patients in the Hueston trials^21,25 were likely to have had obstruction than in Littenberg and coworkers’ study²⁰ (and since the lungs were auscultated in forced expiration in the latter trial, the actual number with airflow obstruction may have been even less than indicated). The fact that only the subgroup with airway obstruction improved with b2-agonists in the trial by Melbye and colleagues²² reflects the possible importance of this baseline characteristic.

Limitations

Our review has some limitations. Although it includes all of the available data regarding the effectiveness of b2-agonists for patients with acute bronchitis or acute cough, the number of studies and total number of patients included are small. Therefore, our review has limited power to detect differences between patients who were and were not given b2-agonists. In the combined data of trials in adults, there was a trend toward improvements regarding cough, productive cough, night cough, and return to work, but these differences did not reach statistical significance. The midpoint estimates for the relative risk reductions range from 14% to 24% for these outcomes, but all overlap 0. There was also a clinically minor and statistically nonsignificant trend toward lower daily cough severity scores in patients randomized to the b2-agonists.

The studies were also all of a short duration. There is no information as to whether treatment with b2-agonists would alter outcomes beyond 3 to 7 days. This is an important omission, because many patients in these studies were still bothered by symptoms at the end of the trials.

Only 2 studies evaluated inhaled b2-agonists, which would currently be the most likely formulation used in adults and older children. Neither of these studies used spacing devices. The delivery of the medicine may have been suboptimal and resulted in less benefit than might have been seen had spacers been used.

Overall, the quality of the trials was fair to good . There may have been additional biases, however, because most of the trials had unequal distribution of co-interventions and did not record compliance with study medications. Also, even though the studies were all double-blinded, the fact that the majority of the patients in one trial knew which study medication they had been given indicates that the blinding may not have been adequate in these studies because of the taste or side effects of the study medications.

Conclusions

Our review highlights the gaps in evidence regarding the utility of b2-agonists in the treatment of acute cough and acute bronchitis in patients without underlying pulmonary disease. Although there is a possibility that these agents may be useful, additional data demonstrating benefit is required before they can be routinely recommended. There is a particular need for identifying clinical characteristics that can predict which patients might benefit. For example, there is a complete lack of data in children older than 2 years who have signs of airway obstruction. More evidence on the risk-benefit ratio of b2-agonists in adults with clinical signs of airflow limitation is also necessary. Additional areas of useful research would be in evaluating long-acting b2-agonists (because of ease of adherence), in evaluating the benefits of inhaled b2-agonists with spacing devices, and in comparing b2-agonists with other symptomatic treatments.

Acknowledgments

We thank Bill Hueston, Ben Littenberg, Hasse Melbye, and Peter Rowe for providing unpublished information; Bill Grant for assistance with statistics; and Ron D’Souza and Steve MacDonald of the Cochrane Collaboration and Bette Jean Ingui for assistance with database searches.

Acute bronchitis is characterized by cough associated with other symptoms of upper respiratory infection. Although this condition is self-limited, most patients feel ill, and many do not perform their usual activities. The optimal treatment for this common condition in patients who do not have underlying pulmonary disease is not clear. Clinicians often prescribe antibiotics,^1,2 in spite of the fact that they are of little overall benefit.^3,4 It is important to examine the effectiveness of alternative approaches.

b2-agonists have been proposed, because healthy patients have impaired airflow when infected with pathogens known to cause acute bronchitis.^5-7 Also, cough is the primary symptom for some patients who have asthma,⁸ and many of these patients benefit from b2-agonists.⁹ b2-agonists are effective in reducing cough due to other causes, such as bronchoscopy¹⁰ and intravenous fentany^1,11

We conducted this systematic review to determine whether b2-agonists are effective for patients who have acute bronchitis without underlying pulmonary disease. If b2-agonists are effective, then they should be more widely used; only a minority of US family physicians currently prescribe them for acute bronchitis.^2,12

Methods

We attempted to locate all controlled trials that compared b2-agonists with placebo or an alternative treatment in patients older than 2 years who presented with acute bronchitis or acute cough without a clear etiology (eg, pneumonia, pertussis, or sinusitis). We included patients with acute cough, because the clinical definition of acute bronchitis is not standardized. Textbooks^13-15 and clinician studies^16,17 instruct that cough in association with an acute respiratory infection is required for a diagnosis; otherwise, there are differing criteria regarding the need for other symptoms and signs, such as dyspnea, abnormal chest findings, and sputum.

We searched MEDLINE (1966-2000), EMBASE (1974-2000), and The Cochrane Library (through August 2000) using the key words “bronchitis” or “cough”, together with the terms “adrenergic beta-agonist (exp),” “bronchodilator agents (exp),” “sympathomimetic (exp),” “albuterol,” “salbutamol,” “bitolterol,” “isoetharine,” “metaproterenol,” “pirbuterol,” “salmeterol,” “terbutaline,” “fenoterol,” “formoterol,” or “procaterol” (note that albuterol and salbutamol are the same compound). We also searched conference proceedings databases (Inside Conferences, 1993-99; Conference Papers Index, 1973-99); the System for Information on Grey Literature in Europe database (1980-2000); the reference lists of retrieved articles, review articles, and textbooks; and the Science Citation Index (1990-2000). Finally, we wrote to all US manufacturers of brand name b2-agonists. There were no language restrictions in our search.

Two investigators (C.F., J.S.) independently reviewed all the retrieved titles and abstracts. Studies selected by either investigator as possibly meeting the inclusion criteria were retrieved in their entirety. One investigator (J.S.) then deleted the journal of publication, title, authors, affiliations, and results sections of each study that met the inclusion criteria, and compiled a list of all the reported outcomes. The list of outcomes was forwarded to the other 3 investigators who independently, and then through discussion, determined which outcomes would be included in our review. The main criterion for selection was that the outcome should be directly important to patients. The same 3 investigators then graded the quality of each study using the 5-point Jadad scale, with points given for method of randomization (0-2), adequacy of blinding (0-2), and description of withdrawals (0-1).¹⁸ The Jadad scale is a validated, well-accepted, and frequently used quality assessment scale. Agreement on quality was assessed with a k score, and disagreements were resolved by discussion. Trials were excluded if all investigators agreed that the trial did not meet our inclusion criteria. The remaining articles in their entirety were then distributed to all investigators, each of whom independently extracted data for the selected outcomes. Disagreements were resolved by discussion. We attempted to contact authors to obtain missing data.

Summary statistics were calculated using Review Manager 4.1 software (Update Software, Oxford, England). We used fixed effects models for outcomes without statistically significant heterogeneity (at P <.10) and random effects models for outcomes with significant heterogeneity. For dichotomous outcomes, we reported relative risks (RRs), absolute risk reductions, and numbers needed to treat (NNTs), and for continuous outcomes, standardized mean differences (SMD). We considered a level of P less than .05 to be statistically significant.

Results

Included Studies

The major characteristics of the trials are shown in Table 1. We included 6 controlled trials comparing b2-agonists and placebo,^19-24 and one trial comparing a b2-agonist with erythromycin.²⁵ A trial comparing a b2-agonist with placebo in children²⁶ was excluded because all participants had recurrent cough and the mean duration of cough (8 weeks) was much longer than the maximum of 30 days used in the other trials.

All trials enrolled patients that presented to primary care settings. The stated diagnoses were “acute bronchitis,”^21,22,25 “acute cough,”^19,20 and “acute transient cough.”^23,24 Both trials in children excluded participants with abnormal lung examinations¹⁹ or “with bronchial obstruction needing bronchodilating medication.”²³ None of the adult trials excluded patients with wheezing; the percentage with wheezing ranged from 20% to 44% in the 4 trials that mentioned it. All adult trials included both smokers an nonsmokers.

The only trial that mentioned how well patients adhered to study medications²⁵ reported more than 95% compliance for both groups. Regarding co-interventions, 3 trials prohibited other antitussives^19,23,24 ; 3 trials allowed them and recorded their use as an outcome^20,21,25 ; and one trial did not mention co-interventions. ²² One trial prohibited the use of antibiotics²⁴ ; other trials comparing b2-agonists to placebo allowed the use of antibiotics at the discretion of the clinician (except as noted for the 1994 study by Hueston²¹). No trials were clearly sponsored by pharmaceutical manufacturers, but the medications were supplied free of charge by manufacturers in 3 studies.^19,22,24

The quality of the trials varied from 2 to 4 on the Jadad scale Table 1. The k score for reviewers’ quality scores was 0.27, indicating only fair agreement. The majority of the disagreements related to different initial interpretations of the adequacy of blinding and description of withdrawals. These differences were resolved with further discussion.

Data Analysis

The clinical heterogeneity of the trials was so great that examining them as a single group did not seem reasonable. Therefore, we initially examined the trials as follows: (1) those in children, (2) those in adults comparing b2-agonists with placebo, and (3) those in adults comparing b2-agonists with erythromycin. We then combined the data from the trial that compared a b2-agonist with erythromycin with that from the other trials in adults in a secondary analysis.

Trials in Children

Neither trial involving children demonstrated any benefits from albuterol Table 2. Combining the daily cough scores for days 1 to 3 for these trials revealed a trend toward worse scores in the group receiving albuterol Table 3. The results from the 2 trials were homogeneous.

Trials in Adults Comparing b2-agonists with Placebo

The results of the placebo-controlled trials in adults were mixed; one trial found no benefit from b2-agonists, and 3 found at least one benefit. Combining the daily cough severity scores for the 3 trials that included this outcome^20,22,24showed a small nonsignificant trend toward improvement on all days. The results from the individual trials were heterogeneous for day 1 and homogeneous for the other days.

Combining data from the trials that examined persistence of symptoms after a full 7 days of treatment^20-22 yielded no significant difference in presence of cough or night cough Table 4. Combined data also do not show a difference regarding the presence of a productive cough after 7 days or a difference regarding whether patients were working after 4 days. There was significant heterogeneity for 3 of the 4 dichotomous outcomes: cough, productive cough, and return to work.

Trials in Adults Comparing b2-agonists with Erythromycin

In the 1994 Hueston study,²¹ patients given albuterol were less likely to have a cough or a productive cough after 7 days than those given erythromycin, but there were no differences in the presence of night cough after 7 days or in mean days until improvement in cough, well-being, or return to work or normal activities. When the data from this study are combined with that from the other adult trials, there are no significant differences regarding presence after 7 days of cough (RR=0.77; 95% confidence interval [CI], 0.54-1.09), productive cough (RR=0.66; 95% CI, 0.35-1.25), or night cough (RR=0.85; 95% CI, 0.57-1.26).

Adverse Effects

In the trials in children, 11% of the patients given albuterol had shaking or tremor versus 0% given placebo or only dextromethorphan (RR=6.76; 95% CI, 0.86-53.18; NNT=9; 95% CI, 5-100); the results were homogeneous. There were no differences regarding other adverse effects in the trials in children. In the adult trials, patients given b2-agonists were more likely to report tremor, shaking, or nervousness; the percentage of patients having these side effects in the 3 trials that reported specific side effects ranged from 35% to 67% versus control rates of 0% to 23% (RR=7.94; 95% CI, 1.17-53.94; NNT=2.3; 95% CI, 2-3). These data are from the trials that used inhaled fenoterol and oral albuterol.^20,22,25 However, in the 1991 Hueston study,²⁵ only 9% of the patients given inhaled albuterol reported any side effects; therefore, there is considerable heterogeneity among the results of the individual trials. There were no significant differences regarding other adverse effects between the b2-agonist group and control groups as a whole, but the trial comparing albuterol with erythromycin noted more gastrointestinal side effects in the erythromycin group (NNT=3; 95% CI, 2-8).

Subgroup Analyses

In the study by Melbye and colleagues,²² the subgroup of patients with evidence of airway obstruction (defined as wheezing on initial examination, a forced expiratory volume in 1 second of <80% predicted, or a positive response to a methacholine challenge test) who were given fenoterol had lower symptom scores beginning at day 2 than those in this subgroup who were given placebo. This was also true for the smaller subgroup that just had wheezing, but no difference was noted for patients with a normal lung examination. No other trial did a subgroup analysis limited to patients with evidence of airflow obstruction. The 1994 Hueston study²¹ reported that among patients given albuterol, those with wheezing were slightly less likely to be coughing after 7 days than those without wheezing, but the difference was not statistically significant.

Melbye and coworkers²² found that patients who smoked or had also received antibiotics had greater reductions in total symptom scores on day 7 if given fenoterol. Smokers had similar responses to nonsmokers in the studies by Hueston.^21,25 Littenberg and colleagues²⁰ found that patients given erythromycin trended toward lower cough severity scores if given albuterol instead of placebo, and patients not given erythromycin showed a trend toward higher scores if given albuterol. The 1994 Hueston study²¹ reported that the differences between the groups given and not given albuterol persisted after stratification by erythromycin use.

Discussion

The findings from our review do not support the routine use of b2-agonists for patients who do not have underlying pulmonary disease and present with an acute cough or acute bronchitis. These results must be interpreted in light of the patients that were enrolled in the trials. In particular, because the 2 trials in children excluded patients who were wheezing, the utility of b2-agonists in children with acute cough and evidence of airway obstruction is unknown. b2-agonists do lead to modest short-term improvements in clinical scores in children younger than 2 years who have bronchiolitis.²⁷

The discordant results seen in the trials of adults may reflect different patient populations. Although the inclusion criteria were similar in these trials, more patients were wheezing on initial examination in the Hueston studies^21,25 than in the studies by Littenberg and coworkers²⁰ or Melbye and colleagues.²² Wheezing in unforced expiration is a specific finding for airflow obstruction²⁸; and therefore, more patients in the Hueston trials^21,25 were likely to have had obstruction than in Littenberg and coworkers’ study²⁰ (and since the lungs were auscultated in forced expiration in the latter trial, the actual number with airflow obstruction may have been even less than indicated). The fact that only the subgroup with airway obstruction improved with b2-agonists in the trial by Melbye and colleagues²² reflects the possible importance of this baseline characteristic.

Limitations

Our review has some limitations. Although it includes all of the available data regarding the effectiveness of b2-agonists for patients with acute bronchitis or acute cough, the number of studies and total number of patients included are small. Therefore, our review has limited power to detect differences between patients who were and were not given b2-agonists. In the combined data of trials in adults, there was a trend toward improvements regarding cough, productive cough, night cough, and return to work, but these differences did not reach statistical significance. The midpoint estimates for the relative risk reductions range from 14% to 24% for these outcomes, but all overlap 0. There was also a clinically minor and statistically nonsignificant trend toward lower daily cough severity scores in patients randomized to the b2-agonists.

The studies were also all of a short duration. There is no information as to whether treatment with b2-agonists would alter outcomes beyond 3 to 7 days. This is an important omission, because many patients in these studies were still bothered by symptoms at the end of the trials.

Only 2 studies evaluated inhaled b2-agonists, which would currently be the most likely formulation used in adults and older children. Neither of these studies used spacing devices. The delivery of the medicine may have been suboptimal and resulted in less benefit than might have been seen had spacers been used.

Overall, the quality of the trials was fair to good . There may have been additional biases, however, because most of the trials had unequal distribution of co-interventions and did not record compliance with study medications. Also, even though the studies were all double-blinded, the fact that the majority of the patients in one trial knew which study medication they had been given indicates that the blinding may not have been adequate in these studies because of the taste or side effects of the study medications.

Conclusions

Our review highlights the gaps in evidence regarding the utility of b2-agonists in the treatment of acute cough and acute bronchitis in patients without underlying pulmonary disease. Although there is a possibility that these agents may be useful, additional data demonstrating benefit is required before they can be routinely recommended. There is a particular need for identifying clinical characteristics that can predict which patients might benefit. For example, there is a complete lack of data in children older than 2 years who have signs of airway obstruction. More evidence on the risk-benefit ratio of b2-agonists in adults with clinical signs of airflow limitation is also necessary. Additional areas of useful research would be in evaluating long-acting b2-agonists (because of ease of adherence), in evaluating the benefits of inhaled b2-agonists with spacing devices, and in comparing b2-agonists with other symptomatic treatments.

Acknowledgments

We thank Bill Hueston, Ben Littenberg, Hasse Melbye, and Peter Rowe for providing unpublished information; Bill Grant for assistance with statistics; and Ron D’Souza and Steve MacDonald of the Cochrane Collaboration and Bette Jean Ingui for assistance with database searches.

Various physical diagnostic tests are available to assess meniscal lesions, such as assessment of joint effusion and joint line tenderness (JLT), the McMurray test, and the Apley compression test.^1-4 Many meniscal tests, however, are not easy to perform and seem to be prone to errors.^1,2,4 Also, the diagnostic accuracy of the various meniscal tests has been questioned,^3-5 and conflicting results regarding that accuracy have been reported.⁶ Therefore, we systematically reviewed the medical literature to summarize the available evidence about the diagnostic accuracy of physical diagnostic tests for assessing meniscal lesions of the knee and to combine the results of individual studies when possible. We focused on the most common meniscal tests: the assessment of joint effusion, the McMurray test, JLT, and the Apley compression test.

Methods

Selection of Studies

We conducted a literature search of MEDLINE (1966-1999) and EMBASE (1988-1999) to identify articles written in English, French, German, or Dutch. The Medical Subject Headings (MeSH) terms “knee injuries,” “knee joint,” “knee,” and “menisci tibial,” and the text words “knee” and “effusion” were used. The results of this strategy were combined with a validated search strategy for the identification of diagnostic studies using the MeSH terms “sensitivity and specificity” (exploded), “physical examination” and “not (animal not (human and animal))” and the text words “sensitivity,” “specificity,” “false positive,” “false negative,” “accuracy,” and “screening,”⁷ supplemented with the text words “physical examination” and “clinical examination.” Also, the cited references of relevant publications were examined.

Studies were eligible for inclusion if they addressed the accuracy of at least one physical diagnostic test for the assessment of meniscal lesions of the knee and used arthrotomy, arthroscopy, or magnetic resonance imaging (MRI) as the gold standard. Studies were excluded if no reference group (nondiseased group or subjects with lesions other than the lesion of study) had been included, if only test-positives had been included, if the study pertained to cadavers only, or if only physical examination under anesthesia was considered.

The studies were selected by 2 reviewers independently. A preliminary selection of each study was made by checking the title, the abstract, or both. A definite selection was made by reading the complete article. During a consensus meeting disagreements regarding the selection of studies were discussed, and a definite selection was made. If disagreement persisted, a third reviewer made the final decision.

Assessment of Methodologic Quality and Data Abstraction

The methodologic quality of the selected studies was assessed, and data were abstracted by 2 reviewers independently. A checklist adapted from Irwig and colleagues⁸ and the Cochrane Methods Group on Systematic Review of Screening and Diagnostic Tests⁹ was used for quality assessment. This checklist consisted of 6 criteria for study validity, 5 criteria relevant to the clinical applicability of the results, and 5 items pertaining to the index Table w1, Table w1a test.* In a subsequent consensus meeting, both assessors discussed each criterion on which they initially disagreed. If disagreement persisted, a third reviewer made the final decision.

Statistical Analysis

Statistical analysis was performed according to a strategy adapted from Midgette and colleagues. Figure W1 ¹⁰** For each study, the sensitivity and specificity of each index test were calculated. The c2 test was used to assess the homogeneity of the sensitivity and the specificity among studies. If homogeneity of both sensitivity and specificity was not rejected (P >.10), summary estimates of sensitivity and specificity were calculated.¹⁰ Heterogeneity of sensitivity and specificity might be caused by differences between studies in how clinicians define a positive test result.⁸ In that case, the pairs of sensitivity and specificity will be negatively correlated, as indicated by a negative Spearman rank correlation coefficient (Rs). When the pairs of sensitivity and specificity are negatively correlated, these pairs can be considered to be originating from a common receiver operating characteristic (ROC) curve, and a summary ROC (SROC) curve was estimated by meta-regression.^8,10,11 The better the diagnostic accuracy of the test, the larger the area under the curve.

Differences between study characteristics are another potential source of heterogeneity of sensitivity and specificity.⁸ Those other sources of heterogeneity were assessed by adding the following characteristics to the meta-regression model: study validity items (most valid category of each item vs other categories), setting (primary care vs other), the spectrum of the diseased and the nondiseased (broad spectrum vs small spectrum), the prevalence of meniscal lesions, and the year of publication. When a significant subgroup was identified (P <.05), separate analyses were performed for each subgroup.

The summary estimates of sensitivity and specificity were used to calculate the predictive value of a positive (PV+) and negative (PV-) test result for circumstances with varying prevalences of meniscal lesions. When the sensitivities or specificities were heterogeneous between studies, however, the summary estimate of sensitivity was used for calculating predictive values with the accompanying specificity, estimated from the SROC curve.

Results

Selection of Studies

The literature search revealed a total of 402 potentially eligible studies, of which 10 were selected for inclusion.^12-21 Three other studies were found by reference tracking.^22-24 Thus, 13 studies met the selection criteria. The reply to a letter to the editor to one of the studies contained additional information and was also considered for analysis.^17,25,26

Methodologic Quality and Study Characteristics

The index test and reference standard had been measured independently (blindly) of each other in only 2 studies.^16,21 Verification bias seemed to be present in all studies (patients with an abnormal physical test result were more likely to undergo the gold standard test, inflating the sensitivity and decreasing the specificity). Nine studies applied arthroscopy as the gold standard,^{12-14,16,17,19-21,24} and 1 study used MRI.¹⁵ No study was performed in a primary care setting. In 7 studies a broad spectrum of knee lesions was reported,^{12-15,17,20,21} and in 4 studies the spectrum was not specified Table 1.^18,19,22,23 A broad spectrum of conditions in the reference group (nondiseased) was present in 8 studies,^{12-15,17,20,22,23} while in 4 studies the spectrum was not specified.^18,19,21,24 Details regarding the index tests were poorly reported, except in 2 studies.^17,21 In all studies that addressed the McMurray test, the experience of a “thud” or “click” was used for designating a test as positive.^{12,13,15-19,22} Only 2 studies mentioned assessment of the index test independent of knowledge of other clinical information (including the results of other meniscal tests).^17,21Table w2* The age and sex distribution of the patients and the duration of complaints are presented in Table 1.

Accuracy of Meniscal Tests

The accuracy of the assessment of joint effusion was determined in 4 studies, the McMurray test in 11, JLT in 10, the Apley compression test in 3, and 5 studies addressed various other tests. No data were presented in or could be derived from 1 study pertaining to joint effusion, 3 studies regarding the McMurray test,^14,23,24 and 1 study on JLT,²⁴ while from 1 study pertaining to both the McMurray test and JLT only the point estimates of the various test characteristics were reported without the original number of patients in the various categories.¹⁵ Of the study of Evans and coworkers,^17,26 who presented data of an inexperienced and experienced researcher, only the latter results were used. Of the study of Abdon and colleagues,¹⁴ who made a distinction between tenderness of the medial and posterior part of the joint line, only the data of the medial part were considered. It should be noted that 2 studies incorporated a very small number of nondiseased subjects.^23,24 Also, one of those studies presented results from individual knees instead of subjects.²⁴ Part of their results pertained to both knees of the same subject, which violates the assumption of (statistical) independence of the observations. Therefore, this study was excluded from further analysis. Finally, some studies did not make a distinction between medial and lateral meniscal lesions,^{13,17,19,22,23} while others presented the results for medial and lateral meniscal lesions separately.^{12,14,15,18,20} Of the latter studies, only the results of medial meniscal tests were used for statistical analysis.

The diagnostic accuracy of assessment of joint effusion and the various meniscal tests is shown in Table 2. There was significant heterogeneity of sensitivity and specificity of all tests, except for specificity of the Apley compression test (P=.89).

Sensitivity and specificity were negatively correlated for joint effusion (Rs = -1.0), the McMurray test (Rs = -0.43), and JLT (Rs = -0.62). This means that as one increased, the other decreased, which is to be expected. The SROC curves Figure 1 indicate little discriminative power of those meniscal tests. No significant subgroups were detected for both tests. The power of meta-regression analysis, however, was low because of the small number of available studies.

Sensitivity and specificity of the Apley compression test were not correlated (Rs = 0.0) and no SROC curve was estimated. Sources of heterogeneity could not be identified. Only 3 studies, however, addressed this test.

Figure 2 shows the positive predictive value (PV+) and negative predictive value (PV-) for the assessment of joint effusion, the McMurray test, and JLT, according to varying prevalences of meniscal lesions. The summary estimate of sensitivity and accompanying specificity (derived from the SROC curve) were used for joint effusion (0.43 and 0.70), the McMurray test (0.48 and 0.86), and JLT (0.77 and 0.41). Only the McMurray test had a favorable estimated PV+. The PV+ of joint effusion and JLT exceeded the presumed prevalences only slightly, indicating poor additional diagnostic value. The PV- of all tests was poor.

Discussion

Our goal was to summarize the available evidence on the accuracy of various physical diagnostic tests for assessing meniscal lesions of the knee. The accuracy of those tests seems to be poor, and only a positive McMurray test result seems to be of some diagnostic significance.

However, because of the small number and poor quality of the studies found, we have significant concerns about the application of these results. Because of the methodologic flaws, the estimates of the various parameters of test accuracy probably will be biased, and the results of this meta-analysis should be interpreted with care. In view of the presence of review bias and verification bias in the various studies, the sensitivity of the various meniscal tests will be overestimated. The effect of those biases on specificity estimates, however, is less clear: Those specificities could be either overestimated or underestimated. Therefore, a rigorous conclusion regarding the diagnostic accuracy of the various meniscal tests cannot be made. Also, analysis of the influence of other potential sources of bias (like the type of gold standard, setting, and spectrum) was impeded by the low number of studies or the lack of information from studies.

The various physical diagnostic meniscal tests do not seem to be very helpful in guiding clinical decision making, and physicians should be aware of the very limited value of those tests. In the clinical determination of a meniscal lesion, however, meniscal tests are, of course, not applied in isolation. Combining the results of the various tests might improve accurate diagnosis of a meniscal lesion, and including other characteristics as well (eg, elements of history-taking) will further improve diagnosis setting. Those characteristics may even have more diagnostic power than the meniscal tests. Abdon and coworkers¹⁴ performed a discriminant analysis and addressed the McMurray test, JLT, and various other signs and symptoms jointly. Of the meniscal tests only, JLT resulted in some additional discriminative power (apart from various elements of history-taking). The results of their analysis, however, are not readily understandable, and the contribution of the individual items to improve the ability to diagnose meniscal lesions correctly remains obscure. Reanalysis of their results by multiple logistic regression might give results that are more directly applicable in clinical practice.

Because no study has been performed in primary care, and test characteristics are influenced by referral filters,²⁷ one can only speculate what the effect will be of extrapolating the observed results to a primary care setting. If family physicians, who will be less experienced in performing those meniscal tests, apply as low a threshold for interpreting a test result as positive, the sensitivity of those tests will be higher, but the specificity will be lower. The predictive value of a negative test result will be affected only slightly, but the predictive value of a positive test result will decrease. On the other hand, when family physicians would apply a high threshold for test positivity, sensitivity decreases and specificity increases, resulting in an increased predictive value of a positive test result. Because of the case mix of patients with traumatic knee problems in primary care (ranging from vague minor knee disorders to clear-cut meniscal lesions), the prior probability (or prevalence) of having a meniscal lesion will be low in primary care, which means that the diagnostic gain will be low also Figure 2.

Recommendations For Future Research

Methodologically sound research on the diagnostic accuracy of the various physical diagnostic tests (determined both for each test separately and for all tests jointly) in combination with patient characteristics (eg, age, physical fitness, and functional demands) and elements of the medical history (eg, the type of trauma and the nature of the complaints) is needed. Such research will be more relevant to clinical practice and patient care if the effect of a correct early diagnosis on the functional outcome of the patient is assessed as well.

Recommendations For Clinical Practice

For the time being, there is little evidence that the diagnosis of meniscal lesions of the knee can be improved by applying the assessment of joint effusion, the McMurray test, JLT, or the Apley compression test. The need for applying more advanced diagnostic methods (eg, MRI) or referral for surgical treatment can be based only on the severity of the patient’s complaints.

Various physical diagnostic tests are available to assess meniscal lesions, such as assessment of joint effusion and joint line tenderness (JLT), the McMurray test, and the Apley compression test.^1-4 Many meniscal tests, however, are not easy to perform and seem to be prone to errors.^1,2,4 Also, the diagnostic accuracy of the various meniscal tests has been questioned,^3-5 and conflicting results regarding that accuracy have been reported.⁶ Therefore, we systematically reviewed the medical literature to summarize the available evidence about the diagnostic accuracy of physical diagnostic tests for assessing meniscal lesions of the knee and to combine the results of individual studies when possible. We focused on the most common meniscal tests: the assessment of joint effusion, the McMurray test, JLT, and the Apley compression test.

Methods

Selection of Studies

We conducted a literature search of MEDLINE (1966-1999) and EMBASE (1988-1999) to identify articles written in English, French, German, or Dutch. The Medical Subject Headings (MeSH) terms “knee injuries,” “knee joint,” “knee,” and “menisci tibial,” and the text words “knee” and “effusion” were used. The results of this strategy were combined with a validated search strategy for the identification of diagnostic studies using the MeSH terms “sensitivity and specificity” (exploded), “physical examination” and “not (animal not (human and animal))” and the text words “sensitivity,” “specificity,” “false positive,” “false negative,” “accuracy,” and “screening,”⁷ supplemented with the text words “physical examination” and “clinical examination.” Also, the cited references of relevant publications were examined.

Studies were eligible for inclusion if they addressed the accuracy of at least one physical diagnostic test for the assessment of meniscal lesions of the knee and used arthrotomy, arthroscopy, or magnetic resonance imaging (MRI) as the gold standard. Studies were excluded if no reference group (nondiseased group or subjects with lesions other than the lesion of study) had been included, if only test-positives had been included, if the study pertained to cadavers only, or if only physical examination under anesthesia was considered.

The studies were selected by 2 reviewers independently. A preliminary selection of each study was made by checking the title, the abstract, or both. A definite selection was made by reading the complete article. During a consensus meeting disagreements regarding the selection of studies were discussed, and a definite selection was made. If disagreement persisted, a third reviewer made the final decision.

Assessment of Methodologic Quality and Data Abstraction

The methodologic quality of the selected studies was assessed, and data were abstracted by 2 reviewers independently. A checklist adapted from Irwig and colleagues⁸ and the Cochrane Methods Group on Systematic Review of Screening and Diagnostic Tests⁹ was used for quality assessment. This checklist consisted of 6 criteria for study validity, 5 criteria relevant to the clinical applicability of the results, and 5 items pertaining to the index Table w1, Table w1a test.* In a subsequent consensus meeting, both assessors discussed each criterion on which they initially disagreed. If disagreement persisted, a third reviewer made the final decision.

Statistical Analysis

Statistical analysis was performed according to a strategy adapted from Midgette and colleagues. Figure W1 ¹⁰** For each study, the sensitivity and specificity of each index test were calculated. The c2 test was used to assess the homogeneity of the sensitivity and the specificity among studies. If homogeneity of both sensitivity and specificity was not rejected (P >.10), summary estimates of sensitivity and specificity were calculated.¹⁰ Heterogeneity of sensitivity and specificity might be caused by differences between studies in how clinicians define a positive test result.⁸ In that case, the pairs of sensitivity and specificity will be negatively correlated, as indicated by a negative Spearman rank correlation coefficient (Rs). When the pairs of sensitivity and specificity are negatively correlated, these pairs can be considered to be originating from a common receiver operating characteristic (ROC) curve, and a summary ROC (SROC) curve was estimated by meta-regression.^8,10,11 The better the diagnostic accuracy of the test, the larger the area under the curve.

Differences between study characteristics are another potential source of heterogeneity of sensitivity and specificity.⁸ Those other sources of heterogeneity were assessed by adding the following characteristics to the meta-regression model: study validity items (most valid category of each item vs other categories), setting (primary care vs other), the spectrum of the diseased and the nondiseased (broad spectrum vs small spectrum), the prevalence of meniscal lesions, and the year of publication. When a significant subgroup was identified (P <.05), separate analyses were performed for each subgroup.

The summary estimates of sensitivity and specificity were used to calculate the predictive value of a positive (PV+) and negative (PV-) test result for circumstances with varying prevalences of meniscal lesions. When the sensitivities or specificities were heterogeneous between studies, however, the summary estimate of sensitivity was used for calculating predictive values with the accompanying specificity, estimated from the SROC curve.

Results

Selection of Studies

The literature search revealed a total of 402 potentially eligible studies, of which 10 were selected for inclusion.^12-21 Three other studies were found by reference tracking.^22-24 Thus, 13 studies met the selection criteria. The reply to a letter to the editor to one of the studies contained additional information and was also considered for analysis.^17,25,26

Methodologic Quality and Study Characteristics

The index test and reference standard had been measured independently (blindly) of each other in only 2 studies.^16,21 Verification bias seemed to be present in all studies (patients with an abnormal physical test result were more likely to undergo the gold standard test, inflating the sensitivity and decreasing the specificity). Nine studies applied arthroscopy as the gold standard,^{12-14,16,17,19-21,24} and 1 study used MRI.¹⁵ No study was performed in a primary care setting. In 7 studies a broad spectrum of knee lesions was reported,^{12-15,17,20,21} and in 4 studies the spectrum was not specified Table 1.^18,19,22,23 A broad spectrum of conditions in the reference group (nondiseased) was present in 8 studies,^{12-15,17,20,22,23} while in 4 studies the spectrum was not specified.^18,19,21,24 Details regarding the index tests were poorly reported, except in 2 studies.^17,21 In all studies that addressed the McMurray test, the experience of a “thud” or “click” was used for designating a test as positive.^{12,13,15-19,22} Only 2 studies mentioned assessment of the index test independent of knowledge of other clinical information (including the results of other meniscal tests).^17,21Table w2* The age and sex distribution of the patients and the duration of complaints are presented in Table 1.

Accuracy of Meniscal Tests

The accuracy of the assessment of joint effusion was determined in 4 studies, the McMurray test in 11, JLT in 10, the Apley compression test in 3, and 5 studies addressed various other tests. No data were presented in or could be derived from 1 study pertaining to joint effusion, 3 studies regarding the McMurray test,^14,23,24 and 1 study on JLT,²⁴ while from 1 study pertaining to both the McMurray test and JLT only the point estimates of the various test characteristics were reported without the original number of patients in the various categories.¹⁵ Of the study of Evans and coworkers,^17,26 who presented data of an inexperienced and experienced researcher, only the latter results were used. Of the study of Abdon and colleagues,¹⁴ who made a distinction between tenderness of the medial and posterior part of the joint line, only the data of the medial part were considered. It should be noted that 2 studies incorporated a very small number of nondiseased subjects.^23,24 Also, one of those studies presented results from individual knees instead of subjects.²⁴ Part of their results pertained to both knees of the same subject, which violates the assumption of (statistical) independence of the observations. Therefore, this study was excluded from further analysis. Finally, some studies did not make a distinction between medial and lateral meniscal lesions,^{13,17,19,22,23} while others presented the results for medial and lateral meniscal lesions separately.^{12,14,15,18,20} Of the latter studies, only the results of medial meniscal tests were used for statistical analysis.

The diagnostic accuracy of assessment of joint effusion and the various meniscal tests is shown in Table 2. There was significant heterogeneity of sensitivity and specificity of all tests, except for specificity of the Apley compression test (P=.89).

Sensitivity and specificity were negatively correlated for joint effusion (Rs = -1.0), the McMurray test (Rs = -0.43), and JLT (Rs = -0.62). This means that as one increased, the other decreased, which is to be expected. The SROC curves Figure 1 indicate little discriminative power of those meniscal tests. No significant subgroups were detected for both tests. The power of meta-regression analysis, however, was low because of the small number of available studies.

Sensitivity and specificity of the Apley compression test were not correlated (Rs = 0.0) and no SROC curve was estimated. Sources of heterogeneity could not be identified. Only 3 studies, however, addressed this test.

Figure 2 shows the positive predictive value (PV+) and negative predictive value (PV-) for the assessment of joint effusion, the McMurray test, and JLT, according to varying prevalences of meniscal lesions. The summary estimate of sensitivity and accompanying specificity (derived from the SROC curve) were used for joint effusion (0.43 and 0.70), the McMurray test (0.48 and 0.86), and JLT (0.77 and 0.41). Only the McMurray test had a favorable estimated PV+. The PV+ of joint effusion and JLT exceeded the presumed prevalences only slightly, indicating poor additional diagnostic value. The PV- of all tests was poor.

Discussion

Our goal was to summarize the available evidence on the accuracy of various physical diagnostic tests for assessing meniscal lesions of the knee. The accuracy of those tests seems to be poor, and only a positive McMurray test result seems to be of some diagnostic significance.

However, because of the small number and poor quality of the studies found, we have significant concerns about the application of these results. Because of the methodologic flaws, the estimates of the various parameters of test accuracy probably will be biased, and the results of this meta-analysis should be interpreted with care. In view of the presence of review bias and verification bias in the various studies, the sensitivity of the various meniscal tests will be overestimated. The effect of those biases on specificity estimates, however, is less clear: Those specificities could be either overestimated or underestimated. Therefore, a rigorous conclusion regarding the diagnostic accuracy of the various meniscal tests cannot be made. Also, analysis of the influence of other potential sources of bias (like the type of gold standard, setting, and spectrum) was impeded by the low number of studies or the lack of information from studies.

The various physical diagnostic meniscal tests do not seem to be very helpful in guiding clinical decision making, and physicians should be aware of the very limited value of those tests. In the clinical determination of a meniscal lesion, however, meniscal tests are, of course, not applied in isolation. Combining the results of the various tests might improve accurate diagnosis of a meniscal lesion, and including other characteristics as well (eg, elements of history-taking) will further improve diagnosis setting. Those characteristics may even have more diagnostic power than the meniscal tests. Abdon and coworkers¹⁴ performed a discriminant analysis and addressed the McMurray test, JLT, and various other signs and symptoms jointly. Of the meniscal tests only, JLT resulted in some additional discriminative power (apart from various elements of history-taking). The results of their analysis, however, are not readily understandable, and the contribution of the individual items to improve the ability to diagnose meniscal lesions correctly remains obscure. Reanalysis of their results by multiple logistic regression might give results that are more directly applicable in clinical practice.

Because no study has been performed in primary care, and test characteristics are influenced by referral filters,²⁷ one can only speculate what the effect will be of extrapolating the observed results to a primary care setting. If family physicians, who will be less experienced in performing those meniscal tests, apply as low a threshold for interpreting a test result as positive, the sensitivity of those tests will be higher, but the specificity will be lower. The predictive value of a negative test result will be affected only slightly, but the predictive value of a positive test result will decrease. On the other hand, when family physicians would apply a high threshold for test positivity, sensitivity decreases and specificity increases, resulting in an increased predictive value of a positive test result. Because of the case mix of patients with traumatic knee problems in primary care (ranging from vague minor knee disorders to clear-cut meniscal lesions), the prior probability (or prevalence) of having a meniscal lesion will be low in primary care, which means that the diagnostic gain will be low also Figure 2.

Recommendations For Future Research

Methodologically sound research on the diagnostic accuracy of the various physical diagnostic tests (determined both for each test separately and for all tests jointly) in combination with patient characteristics (eg, age, physical fitness, and functional demands) and elements of the medical history (eg, the type of trauma and the nature of the complaints) is needed. Such research will be more relevant to clinical practice and patient care if the effect of a correct early diagnosis on the functional outcome of the patient is assessed as well.

Recommendations For Clinical Practice

For the time being, there is little evidence that the diagnosis of meniscal lesions of the knee can be improved by applying the assessment of joint effusion, the McMurray test, JLT, or the Apley compression test. The need for applying more advanced diagnostic methods (eg, MRI) or referral for surgical treatment can be based only on the severity of the patient’s complaints.

Various physical diagnostic tests are available to assess meniscal lesions, such as assessment of joint effusion and joint line tenderness (JLT), the McMurray test, and the Apley compression test.^1-4 Many meniscal tests, however, are not easy to perform and seem to be prone to errors.^1,2,4 Also, the diagnostic accuracy of the various meniscal tests has been questioned,^3-5 and conflicting results regarding that accuracy have been reported.⁶ Therefore, we systematically reviewed the medical literature to summarize the available evidence about the diagnostic accuracy of physical diagnostic tests for assessing meniscal lesions of the knee and to combine the results of individual studies when possible. We focused on the most common meniscal tests: the assessment of joint effusion, the McMurray test, JLT, and the Apley compression test.

Methods

Selection of Studies

We conducted a literature search of MEDLINE (1966-1999) and EMBASE (1988-1999) to identify articles written in English, French, German, or Dutch. The Medical Subject Headings (MeSH) terms “knee injuries,” “knee joint,” “knee,” and “menisci tibial,” and the text words “knee” and “effusion” were used. The results of this strategy were combined with a validated search strategy for the identification of diagnostic studies using the MeSH terms “sensitivity and specificity” (exploded), “physical examination” and “not (animal not (human and animal))” and the text words “sensitivity,” “specificity,” “false positive,” “false negative,” “accuracy,” and “screening,”⁷ supplemented with the text words “physical examination” and “clinical examination.” Also, the cited references of relevant publications were examined.

Studies were eligible for inclusion if they addressed the accuracy of at least one physical diagnostic test for the assessment of meniscal lesions of the knee and used arthrotomy, arthroscopy, or magnetic resonance imaging (MRI) as the gold standard. Studies were excluded if no reference group (nondiseased group or subjects with lesions other than the lesion of study) had been included, if only test-positives had been included, if the study pertained to cadavers only, or if only physical examination under anesthesia was considered.

The studies were selected by 2 reviewers independently. A preliminary selection of each study was made by checking the title, the abstract, or both. A definite selection was made by reading the complete article. During a consensus meeting disagreements regarding the selection of studies were discussed, and a definite selection was made. If disagreement persisted, a third reviewer made the final decision.

Assessment of Methodologic Quality and Data Abstraction

The methodologic quality of the selected studies was assessed, and data were abstracted by 2 reviewers independently. A checklist adapted from Irwig and colleagues⁸ and the Cochrane Methods Group on Systematic Review of Screening and Diagnostic Tests⁹ was used for quality assessment. This checklist consisted of 6 criteria for study validity, 5 criteria relevant to the clinical applicability of the results, and 5 items pertaining to the index Table w1, Table w1a test.* In a subsequent consensus meeting, both assessors discussed each criterion on which they initially disagreed. If disagreement persisted, a third reviewer made the final decision.

Statistical Analysis

Statistical analysis was performed according to a strategy adapted from Midgette and colleagues. Figure W1 ¹⁰** For each study, the sensitivity and specificity of each index test were calculated. The c2 test was used to assess the homogeneity of the sensitivity and the specificity among studies. If homogeneity of both sensitivity and specificity was not rejected (P >.10), summary estimates of sensitivity and specificity were calculated.¹⁰ Heterogeneity of sensitivity and specificity might be caused by differences between studies in how clinicians define a positive test result.⁸ In that case, the pairs of sensitivity and specificity will be negatively correlated, as indicated by a negative Spearman rank correlation coefficient (Rs). When the pairs of sensitivity and specificity are negatively correlated, these pairs can be considered to be originating from a common receiver operating characteristic (ROC) curve, and a summary ROC (SROC) curve was estimated by meta-regression.^8,10,11 The better the diagnostic accuracy of the test, the larger the area under the curve.

Differences between study characteristics are another potential source of heterogeneity of sensitivity and specificity.⁸ Those other sources of heterogeneity were assessed by adding the following characteristics to the meta-regression model: study validity items (most valid category of each item vs other categories), setting (primary care vs other), the spectrum of the diseased and the nondiseased (broad spectrum vs small spectrum), the prevalence of meniscal lesions, and the year of publication. When a significant subgroup was identified (P <.05), separate analyses were performed for each subgroup.

The summary estimates of sensitivity and specificity were used to calculate the predictive value of a positive (PV+) and negative (PV-) test result for circumstances with varying prevalences of meniscal lesions. When the sensitivities or specificities were heterogeneous between studies, however, the summary estimate of sensitivity was used for calculating predictive values with the accompanying specificity, estimated from the SROC curve.

Results

Selection of Studies

The literature search revealed a total of 402 potentially eligible studies, of which 10 were selected for inclusion.^12-21 Three other studies were found by reference tracking.^22-24 Thus, 13 studies met the selection criteria. The reply to a letter to the editor to one of the studies contained additional information and was also considered for analysis.^17,25,26

Methodologic Quality and Study Characteristics

The index test and reference standard had been measured independently (blindly) of each other in only 2 studies.^16,21 Verification bias seemed to be present in all studies (patients with an abnormal physical test result were more likely to undergo the gold standard test, inflating the sensitivity and decreasing the specificity). Nine studies applied arthroscopy as the gold standard,^{12-14,16,17,19-21,24} and 1 study used MRI.¹⁵ No study was performed in a primary care setting. In 7 studies a broad spectrum of knee lesions was reported,^{12-15,17,20,21} and in 4 studies the spectrum was not specified Table 1.^18,19,22,23 A broad spectrum of conditions in the reference group (nondiseased) was present in 8 studies,^{12-15,17,20,22,23} while in 4 studies the spectrum was not specified.^18,19,21,24 Details regarding the index tests were poorly reported, except in 2 studies.^17,21 In all studies that addressed the McMurray test, the experience of a “thud” or “click” was used for designating a test as positive.^{12,13,15-19,22} Only 2 studies mentioned assessment of the index test independent of knowledge of other clinical information (including the results of other meniscal tests).^17,21Table w2* The age and sex distribution of the patients and the duration of complaints are presented in Table 1.

Accuracy of Meniscal Tests

The accuracy of the assessment of joint effusion was determined in 4 studies, the McMurray test in 11, JLT in 10, the Apley compression test in 3, and 5 studies addressed various other tests. No data were presented in or could be derived from 1 study pertaining to joint effusion, 3 studies regarding the McMurray test,^14,23,24 and 1 study on JLT,²⁴ while from 1 study pertaining to both the McMurray test and JLT only the point estimates of the various test characteristics were reported without the original number of patients in the various categories.¹⁵ Of the study of Evans and coworkers,^17,26 who presented data of an inexperienced and experienced researcher, only the latter results were used. Of the study of Abdon and colleagues,¹⁴ who made a distinction between tenderness of the medial and posterior part of the joint line, only the data of the medial part were considered. It should be noted that 2 studies incorporated a very small number of nondiseased subjects.^23,24 Also, one of those studies presented results from individual knees instead of subjects.²⁴ Part of their results pertained to both knees of the same subject, which violates the assumption of (statistical) independence of the observations. Therefore, this study was excluded from further analysis. Finally, some studies did not make a distinction between medial and lateral meniscal lesions,^{13,17,19,22,23} while others presented the results for medial and lateral meniscal lesions separately.^{12,14,15,18,20} Of the latter studies, only the results of medial meniscal tests were used for statistical analysis.

The diagnostic accuracy of assessment of joint effusion and the various meniscal tests is shown in Table 2. There was significant heterogeneity of sensitivity and specificity of all tests, except for specificity of the Apley compression test (P=.89).

Sensitivity and specificity were negatively correlated for joint effusion (Rs = -1.0), the McMurray test (Rs = -0.43), and JLT (Rs = -0.62). This means that as one increased, the other decreased, which is to be expected. The SROC curves Figure 1 indicate little discriminative power of those meniscal tests. No significant subgroups were detected for both tests. The power of meta-regression analysis, however, was low because of the small number of available studies.

Sensitivity and specificity of the Apley compression test were not correlated (Rs = 0.0) and no SROC curve was estimated. Sources of heterogeneity could not be identified. Only 3 studies, however, addressed this test.

Figure 2 shows the positive predictive value (PV+) and negative predictive value (PV-) for the assessment of joint effusion, the McMurray test, and JLT, according to varying prevalences of meniscal lesions. The summary estimate of sensitivity and accompanying specificity (derived from the SROC curve) were used for joint effusion (0.43 and 0.70), the McMurray test (0.48 and 0.86), and JLT (0.77 and 0.41). Only the McMurray test had a favorable estimated PV+. The PV+ of joint effusion and JLT exceeded the presumed prevalences only slightly, indicating poor additional diagnostic value. The PV- of all tests was poor.

Discussion

Our goal was to summarize the available evidence on the accuracy of various physical diagnostic tests for assessing meniscal lesions of the knee. The accuracy of those tests seems to be poor, and only a positive McMurray test result seems to be of some diagnostic significance.

However, because of the small number and poor quality of the studies found, we have significant concerns about the application of these results. Because of the methodologic flaws, the estimates of the various parameters of test accuracy probably will be biased, and the results of this meta-analysis should be interpreted with care. In view of the presence of review bias and verification bias in the various studies, the sensitivity of the various meniscal tests will be overestimated. The effect of those biases on specificity estimates, however, is less clear: Those specificities could be either overestimated or underestimated. Therefore, a rigorous conclusion regarding the diagnostic accuracy of the various meniscal tests cannot be made. Also, analysis of the influence of other potential sources of bias (like the type of gold standard, setting, and spectrum) was impeded by the low number of studies or the lack of information from studies.

The various physical diagnostic meniscal tests do not seem to be very helpful in guiding clinical decision making, and physicians should be aware of the very limited value of those tests. In the clinical determination of a meniscal lesion, however, meniscal tests are, of course, not applied in isolation. Combining the results of the various tests might improve accurate diagnosis of a meniscal lesion, and including other characteristics as well (eg, elements of history-taking) will further improve diagnosis setting. Those characteristics may even have more diagnostic power than the meniscal tests. Abdon and coworkers¹⁴ performed a discriminant analysis and addressed the McMurray test, JLT, and various other signs and symptoms jointly. Of the meniscal tests only, JLT resulted in some additional discriminative power (apart from various elements of history-taking). The results of their analysis, however, are not readily understandable, and the contribution of the individual items to improve the ability to diagnose meniscal lesions correctly remains obscure. Reanalysis of their results by multiple logistic regression might give results that are more directly applicable in clinical practice.

Because no study has been performed in primary care, and test characteristics are influenced by referral filters,²⁷ one can only speculate what the effect will be of extrapolating the observed results to a primary care setting. If family physicians, who will be less experienced in performing those meniscal tests, apply as low a threshold for interpreting a test result as positive, the sensitivity of those tests will be higher, but the specificity will be lower. The predictive value of a negative test result will be affected only slightly, but the predictive value of a positive test result will decrease. On the other hand, when family physicians would apply a high threshold for test positivity, sensitivity decreases and specificity increases, resulting in an increased predictive value of a positive test result. Because of the case mix of patients with traumatic knee problems in primary care (ranging from vague minor knee disorders to clear-cut meniscal lesions), the prior probability (or prevalence) of having a meniscal lesion will be low in primary care, which means that the diagnostic gain will be low also Figure 2.

Recommendations For Future Research

Methodologically sound research on the diagnostic accuracy of the various physical diagnostic tests (determined both for each test separately and for all tests jointly) in combination with patient characteristics (eg, age, physical fitness, and functional demands) and elements of the medical history (eg, the type of trauma and the nature of the complaints) is needed. Such research will be more relevant to clinical practice and patient care if the effect of a correct early diagnosis on the functional outcome of the patient is assessed as well.

Recommendations For Clinical Practice

For the time being, there is little evidence that the diagnosis of meniscal lesions of the knee can be improved by applying the assessment of joint effusion, the McMurray test, JLT, or the Apley compression test. The need for applying more advanced diagnostic methods (eg, MRI) or referral for surgical treatment can be based only on the severity of the patient’s complaints.

Pneumonia is a leading cause of morbidity, mortality, and hospitalization of nursing home residents.^1-8 Atypical presentations and fewer presenting signs and symptoms in older patients complicate diagnosis.^9,10 Also, clinician (physician, nurse practitioner, and physician assistant) visits to nursing homes are often sporadic, and radiology facilities are rarely on the premises. As a consequence, residents are commonly sent to emergency departments for evaluation,^4,11,12 which undoubtedly contributes to a high hospitalization rate.

Clinicians who periodically see nursing home residents could benefit from a simple clinical tool to identify pneumonia. No large studies of community nursing home residents have systematically studied findings associated with pneumonia. As part of the Missouri LRI Project, we examined how well clinical findings predict radiographic pneumonia.

Methods

The Missouri LRI Project was a prospective observational study in 36 nursing homes in Central Missouri and St. Louis designed to investigate predictors of 2 outcomes of lower respiratory infections (LRIs): mortality and functional decline. Potential cases were identified from August 15, 1995, through September 29, 1998; however, all facilities were not involved until fall 1997. Study facilities were similar in size, ownership, and occupancy to national estimates from the 1995 National Nursing Home Survey (data available on request).¹³

We trained nursing home staff to report ill residents with any of 6 respiratory symptoms (eg, cough, dyspnea, sputum production) or 6 general symptoms (eg, fever, decline in mobility, mental status changes). Project nurses called and visited facilities frequently to reinforce reporting. Under a physician-authorized protocol, ill residents with a possible LRI received a standardized evaluation by a trained project nurse and usually a chest radiograph, complete blood count, and a chemistry panel. Complete criteria for triggering an evaluation are listed in Table 1. For this paper, we were concerned with the 90% of evaluated residents who received a chest radiograph. Criteria for excluding residents from evaluation are summarized in the Figure 1.

The nurse evaluation included an inventory of current symptoms, a review of important chronic conditions (eg, congestive heart failure), and a targeted physical examination. The examination included vital signs and the following body areas or systems: ears, nose, and throat; cardiac; abdominal; neurologic; extremities; skin; and a detailed lung examination. Most project nurses had advanced practice training; the remainder had extensive clinical experience and training in physical assessment. All received an individualized training session with a project geriatrician. Project nurses had substantially more experience than the nursing home staff, who usually report clinical findings to physicians.

Results of the evaluation were reported to the attending physician, who made all treatment decisions. Since the evaluations were clinically appropriate care authorized by individual attending physicians, the institutional review boards that reviewed the project allowed us to substantially simplify the consent process to a simple acceptance or refusal of the evaluation. In 9.2% of evaluations the resident was transferred to the hospital before project nurses could complete a physical assessment. In these instances, we obtained vital sign and clinical examination data from hospital records.

Radiographic Classification

Since all subjects had at least one illness symptom, for this analysis we classified the presence or absence of pneumonia on the basis of reported radiographic findings. Using defined criteria, 2 clinicians independently separated radiology reports into 3 categories: (a) negative, (b) possible, or (c) probable or definite for pneumonia (hereafter, probable pneumonia). For example, a report describing “new left lower lobe infiltrate suggestive of pneumonia” would have been rated as probable, while a report indicating “possible infiltrate” or “infiltrate suggestive of pneumonia or congestive heart failure” would have been rated as possible. As radiologists rarely provide completely unequivocal readings, we did not separate probable and definite pneumonia. In St. Louis 2 clinicians evaluated the reports, and in central Missouri 2 of 4 clinicians considered each report. Where there was disagreement, all 6 raters from the 2 sites independently reviewed the reports and then attempted to reach consensus. For 13% of radiographs, the project radiologist independently interpreted the actual films. This occurred when: (1) consensus could not be achieved; or (2) consensus was possible pneumonia, but probable pneumonia was needed to quality the episode as an LRI under the project definition.

Statistical Analyses

As residents could be included more than once, the unit of analysis throughout is episode of illness. In our major analysis, we developed a multivariable logistic model to estimate the probability of radiographic pneumonia (possible or probable). Before beginning modeling, we imputed mean values for missing continuous data and the largest category for missing dichotomous variables (the number of missing values is noted in Table 2). Data imputation is less biased than dropping cases in developing multivariable models.¹⁴

Illness episodes were then randomly assigned to a two thirds model-development and a one third model-validation sample. On the basis of the literature and clinical experience, we defined categories of variables that might relate to the presence or absence of pneumonia, such as lung findings (eg, crackles, wheezes), respiratory symptoms (eg, cough, sputum production), vital signs, findings of delirium (eg, acute confusion, decreased alertness), and laboratory findings. Restricting our focus to the development sample, we selected the best representatives of these groups on clinical and statistical grounds. For continuous variables, we considered the shape of the relationship to presence of pneumonia. For example, both very high and very low pulse rates predicted increased risk of pneumonia. In such cases, we considered several different ways to represent the variable in the model. We also limited the range of some variables to avoid undue influence of outliers (approximately the 1% most extreme values). For example, pulse rate above 140 was set equal to 140.

We then employed forward and backward stepwise logistic regression with possible or probable pneumonia (also referred to as positive x-ray results) as the dependent variable. For final model inclusion, we required variables to bear a plausible relationship to the diagnosis of pneumonia and meet a statistical significance criterion (a=.05).

To obtain final estimates of the relationship of each model variable to pneumonia probability, we considered adjustments for 2 kinds of correlation within our data: (1) individuals are nested within facilities, and (2) subjects could be represented by more than one episode.¹⁵ Using generalized estimating equations (GEE) in Proc Genmod in SAS software (SAS Institute, Cary, NC),¹⁶ we noted that the effect of facilities was minor, but the effect of repeat episodes by the same subject was more marked. Consequently, we used GEE to account for repeat episodes on subjects. To avoid unstable GEE estimates, we dropped 5 episodes in the development sample and 8 in the overall sample (episodes beyond the 5th and 6th per individual, respectively).

Using parameter estimates from the development sample, we tested the model’s discrimination and calibration in the validation sample.¹⁷ To assess discrimination, we used the c-statistic, which evaluates among all possible pairs of individuals whether those with higher predicted risk are more likely to die. The c-statistic is also equal to the area under the receiver operating characteristic curve. To assess calibration—agreement between observed and predicted mortality over the range of predicted risk—we used the Hosmer-Lemeshow goodness-of-fit statistic.¹⁸ We then used estimates fitted to the overall sample to develop a simple additive score to provide a clinically usable prediction rule. Statistical analyses were performed with SAS statistical software.¹⁶

Results

Project nurses performed 2592 evaluations. In 90% (2337), residents received chest x-rays either in the nursing home or on hospital transfer. In 3 additional cases crucial information was missing from nursing home records. This left for final analysis 2334 episodes in 1474 individuals Figure 1.

Fifty-five percent of radiographs were interpreted as negative, 12% showed possible pneumonia, and 33% showed probable pneumonia. Most nursing home residents with pneumonia had few presenting symptoms; 80% had 3 or fewer respiratory or general symptoms. However, only 7.5% of subjects evaluated had no respiratory symptoms. Table 2 shows the relationship of selected variables to radiographic findings of absent, possible, or probable pneumonia. Though a few signs and symptoms are more common in those with positive (possible or probable pneumonia) than negative chest x-ray results, most did not discriminate at all. Fever (temperature Ž38°C) was present in 44.4% of positives but only 28.5% of negatives (P=.001).

Multivariable Analysis and Prediction Score

Our GEE model to predict radiographic pneumonia includes 3 vital sign abnormalities (fever, rapid pulse, and rapid respiratory rate), 2 lung findings (presence of crackles and absence of wheezes), 2 potential indicators of delirium (somnolence or decreased alertness and acute confusion), and elevated white blood count. Table 3 reports GEE estimates for the entire sample. Though only exhibiting fair overall performance, the model did well at distinguishing subjects with a high probability of pneumonia. In the 20% of subjects with the highest predicted risks, more than two thirds had pneumonia.

For the full range of values, the model derived on the development sample showed a c-statistic of 0.672, which reduced to 0.632 in the validation sample. A value of 1.0 would indicate perfect discrimination between those who did and did not have radiographic pneumonia, while a value of 0.5 would indicate no better than chance discrimination. Model calibration was not acceptable in the validation sample (Hosmer-Lemeshow goodness-of-fit statistic, P=.008). Inspection suggested the disagreement between predicted and observed probability of pneumonia was primarily with lower-risk estimates.

Because the model performed relatively well at distinguishing subjects very likely to have pneumonia, we created a simple point system aimed at identifying such high-risk individuals. Table 4 shows the scoring system. For 33% of subjects (score Ž3), there was a 56% or higher probability of radiographic pneumonia. An additional 24% of subjects (score of 2) had 44% probability of radiographic pneumonia. However, even those with the lowest scores (-1 to 0, 15% of subjects) still had a 24% probability of pneumonia. The relationship between the score and the probability of radiographic evidence of pneumonia is shown in Figure W1.*

Discussion

In a large community-based sample, we considered presenting symptoms, signs, and laboratory findings associated with radiographic pneumonia. Individual findings discriminated poorly, and we could not separate out a very-low-risk group. However, our simple scoring system identified approximately one third to slightly more than one half with high probability of pneumonia—individuals who might be treated without a confirmatory chest x-ray. If our data are confirmed, they suggest a simple clinical strategy in patients with respiratory or general symptoms Table 1 that might suggest pneumonia: (1) if there are no respiratory symptoms, consider other conditions, such as a urinary tract infection, that might fully explain the symptoms; (2) obtain information to apply our symptom score Table 4; (3) for those with scores of 2 or higher (some might choose 3 instead), treat for pneumonia; (4) for those with scores of -1, 0, or 1, obtain a chest radiograph as a guide to treatment.

Considering individual findings, fever was significantly more common in pneumonia, but only 43% of those with possible or probable pneumonia had a temperature of at least 38°C. This reaffirms common wisdom and previous findings that fever is frequently absent in elderly people with pneumonia.^9,19 We also confirmed that few signs or symptoms are the norm for nursing home-acquired pneumonia.

Chest examination findings also do not adequately distinguish patients with and without pneumonia Table 2. Also, even expert physicians frequently differ on lung examination findings.²⁰ Nonetheless, presence of crackles and absence of wheezing contribute to our scoring system. Both findings are seen with multiple conditions, but in our data crackles are slightly more associated with pneumonia, while wheezing is more strongly associated with other diseases.

The other components of our scoring system are clinical factors commonly associated with pneumonia. Though none individually discriminates well between those with and without pneumonia Table 2, several combined serve to identify a high-risk group.

Four previous studies from emergency department or outpatient settings developed clinical prediction rules to identify pneumonia.^21-24 Criteria for identifying subjects varied substantially, and each rule has limited accuracy in predicting radiographic pneumonia.²⁰ We had adequate data to evaluate 3 of the rules.^21-23 As is usually the case when transporting a prediction rule to a new sample, none performed any better than our rule (data not shown). Our sample created the very difficult challenge for any prediction rule of a very high overall prevalence of pneumonia (45%). That made it unlikely that we could identify a low-risk group in whom x-ray studies could be readily forgone, but we were able to identify a highrisk group.

Limitations

Our findings are subject to several limitations. All facilities in our study were located in central or eastern Missouri, and not all physicians or eligible residents in those facilities participated. Compared with national data, we studied an unusually representative sample of nursing home residents from 36 facilities, including rural and urban locations. Also, in episodes excluded because of physician nonparticipation, residents were very similar to included residents in age, vital signs, and presenting symptoms (data available on request). More important, we lack an independent validation sample from a different cohort. Clinical prediction rules usually do not perform as well in independent samples. This is exemplified by the poor performance of the 3 rules we considered from other settings. Overall, our logistic model was only modest in discriminating and was not well calibrated for low-risk episodes in our reserved validation sample. Although we have developed a promising scoring system to identify residents with high probability of radiographic pneumonia, it needs to be validated in other samples of nursing home residents to determine its ultimate usefulness. For all these reasons, our results may not generalize.

Also, although we identified residents prospectively, project nurses were unable to evaluate 9.2% of residents before transfer to a hospital. Clinical findings abstracted from medical records, such as lung findings, may not have been complete. It is also possible that project nurses could have missed some important findings. However, our staff provided a higher level of expertise than is typically available in nursing homes. In fact, this may limit application of our findings. Nursing home staff vary widely in their ability to accurately examine residents or even identify illness. In many instances, facility staff had not obtained vital signs at the point when we identified a resident as ill enough to qualify for an evaluation.²⁵ Therefore, in many nursing homes, physicians may lack confidence to apply our rule without an evaluation by a physician, advanced practice nurse, or physician assistant.

Finally, determining whether subjects had pneumonia primarily depended on our classification of radiographic reports. Though radiographs generally included 2 views, many were portable films of variable quality, and frequently there was no previous radiograph for comparison. In some subjects with pneumonia, radiographic infiltrates might not yet have developed. Also, even under ideal conditions, radiologists commonly disagree on the presence of pneumonia.²⁶ Some subjects may have been misclassified. However, unless radiographic technique or interpretation was specifically related to clinical predictors, misclassification would simply diminish the relationship of predictors to pneumonia rather than creating a bias. We reviewed reports rather than radiographs, because that is the information usually available to clinicians faced with diagnosis and treatment decisions. We also paid special attention to avoiding any bias in the interpretations. All data were recorded before interpreting radiology reports and the interpretations were performed independent of clinical data. We also made special efforts to assure consistency in labeling radiology reports as possible, probable, or negative for pneumonia. When lack of agreement persisted, the study radiologist reinterpreted the actual films.

Conclusions

Most nursing home residents with pneumonia have few symptoms. We created a simple scoring to identify nursing home residents who have a high probability of radiographic pneumonia. If our results are confirmed, physicians might consider initiating treatment without an x-ray in such residents. Low scores do not rule out pneumonia, and most physicians would want to press for further diagnosis or treatment in this group.

Acknowledgments

This study was supported by the Agency for Healthcare Research and Quality (grant HS08551) and Dr Mehr’s Robert Wood Johnson Foundation Generalist Physician Faculty Scholars award. Dr Kruse was partially supported by an Institutional National Research Service Award (PE10038) from the Health Resources and Services Administration. Our project would not have been possible without the support of the many attending physicians, administrators, and staff of the involved nursing homes. Dr Clive Levine re-read more than 200 radiographs; Karen Davenport provided crucial administrative support; and Karen Madrone, MPA, assisted with manuscript preparation. Many other unnamed project staff also contributed.

Pneumonia is a leading cause of morbidity, mortality, and hospitalization of nursing home residents.^1-8 Atypical presentations and fewer presenting signs and symptoms in older patients complicate diagnosis.^9,10 Also, clinician (physician, nurse practitioner, and physician assistant) visits to nursing homes are often sporadic, and radiology facilities are rarely on the premises. As a consequence, residents are commonly sent to emergency departments for evaluation,^4,11,12 which undoubtedly contributes to a high hospitalization rate.

Clinicians who periodically see nursing home residents could benefit from a simple clinical tool to identify pneumonia. No large studies of community nursing home residents have systematically studied findings associated with pneumonia. As part of the Missouri LRI Project, we examined how well clinical findings predict radiographic pneumonia.

Methods

The Missouri LRI Project was a prospective observational study in 36 nursing homes in Central Missouri and St. Louis designed to investigate predictors of 2 outcomes of lower respiratory infections (LRIs): mortality and functional decline. Potential cases were identified from August 15, 1995, through September 29, 1998; however, all facilities were not involved until fall 1997. Study facilities were similar in size, ownership, and occupancy to national estimates from the 1995 National Nursing Home Survey (data available on request).¹³

We trained nursing home staff to report ill residents with any of 6 respiratory symptoms (eg, cough, dyspnea, sputum production) or 6 general symptoms (eg, fever, decline in mobility, mental status changes). Project nurses called and visited facilities frequently to reinforce reporting. Under a physician-authorized protocol, ill residents with a possible LRI received a standardized evaluation by a trained project nurse and usually a chest radiograph, complete blood count, and a chemistry panel. Complete criteria for triggering an evaluation are listed in Table 1. For this paper, we were concerned with the 90% of evaluated residents who received a chest radiograph. Criteria for excluding residents from evaluation are summarized in the Figure 1.

The nurse evaluation included an inventory of current symptoms, a review of important chronic conditions (eg, congestive heart failure), and a targeted physical examination. The examination included vital signs and the following body areas or systems: ears, nose, and throat; cardiac; abdominal; neurologic; extremities; skin; and a detailed lung examination. Most project nurses had advanced practice training; the remainder had extensive clinical experience and training in physical assessment. All received an individualized training session with a project geriatrician. Project nurses had substantially more experience than the nursing home staff, who usually report clinical findings to physicians.

Results of the evaluation were reported to the attending physician, who made all treatment decisions. Since the evaluations were clinically appropriate care authorized by individual attending physicians, the institutional review boards that reviewed the project allowed us to substantially simplify the consent process to a simple acceptance or refusal of the evaluation. In 9.2% of evaluations the resident was transferred to the hospital before project nurses could complete a physical assessment. In these instances, we obtained vital sign and clinical examination data from hospital records.

Radiographic Classification

Since all subjects had at least one illness symptom, for this analysis we classified the presence or absence of pneumonia on the basis of reported radiographic findings. Using defined criteria, 2 clinicians independently separated radiology reports into 3 categories: (a) negative, (b) possible, or (c) probable or definite for pneumonia (hereafter, probable pneumonia). For example, a report describing “new left lower lobe infiltrate suggestive of pneumonia” would have been rated as probable, while a report indicating “possible infiltrate” or “infiltrate suggestive of pneumonia or congestive heart failure” would have been rated as possible. As radiologists rarely provide completely unequivocal readings, we did not separate probable and definite pneumonia. In St. Louis 2 clinicians evaluated the reports, and in central Missouri 2 of 4 clinicians considered each report. Where there was disagreement, all 6 raters from the 2 sites independently reviewed the reports and then attempted to reach consensus. For 13% of radiographs, the project radiologist independently interpreted the actual films. This occurred when: (1) consensus could not be achieved; or (2) consensus was possible pneumonia, but probable pneumonia was needed to quality the episode as an LRI under the project definition.

Statistical Analyses

As residents could be included more than once, the unit of analysis throughout is episode of illness. In our major analysis, we developed a multivariable logistic model to estimate the probability of radiographic pneumonia (possible or probable). Before beginning modeling, we imputed mean values for missing continuous data and the largest category for missing dichotomous variables (the number of missing values is noted in Table 2). Data imputation is less biased than dropping cases in developing multivariable models.¹⁴

Illness episodes were then randomly assigned to a two thirds model-development and a one third model-validation sample. On the basis of the literature and clinical experience, we defined categories of variables that might relate to the presence or absence of pneumonia, such as lung findings (eg, crackles, wheezes), respiratory symptoms (eg, cough, sputum production), vital signs, findings of delirium (eg, acute confusion, decreased alertness), and laboratory findings. Restricting our focus to the development sample, we selected the best representatives of these groups on clinical and statistical grounds. For continuous variables, we considered the shape of the relationship to presence of pneumonia. For example, both very high and very low pulse rates predicted increased risk of pneumonia. In such cases, we considered several different ways to represent the variable in the model. We also limited the range of some variables to avoid undue influence of outliers (approximately the 1% most extreme values). For example, pulse rate above 140 was set equal to 140.

We then employed forward and backward stepwise logistic regression with possible or probable pneumonia (also referred to as positive x-ray results) as the dependent variable. For final model inclusion, we required variables to bear a plausible relationship to the diagnosis of pneumonia and meet a statistical significance criterion (a=.05).

To obtain final estimates of the relationship of each model variable to pneumonia probability, we considered adjustments for 2 kinds of correlation within our data: (1) individuals are nested within facilities, and (2) subjects could be represented by more than one episode.¹⁵ Using generalized estimating equations (GEE) in Proc Genmod in SAS software (SAS Institute, Cary, NC),¹⁶ we noted that the effect of facilities was minor, but the effect of repeat episodes by the same subject was more marked. Consequently, we used GEE to account for repeat episodes on subjects. To avoid unstable GEE estimates, we dropped 5 episodes in the development sample and 8 in the overall sample (episodes beyond the 5th and 6th per individual, respectively).

Using parameter estimates from the development sample, we tested the model’s discrimination and calibration in the validation sample.¹⁷ To assess discrimination, we used the c-statistic, which evaluates among all possible pairs of individuals whether those with higher predicted risk are more likely to die. The c-statistic is also equal to the area under the receiver operating characteristic curve. To assess calibration—agreement between observed and predicted mortality over the range of predicted risk—we used the Hosmer-Lemeshow goodness-of-fit statistic.¹⁸ We then used estimates fitted to the overall sample to develop a simple additive score to provide a clinically usable prediction rule. Statistical analyses were performed with SAS statistical software.¹⁶

Results

Project nurses performed 2592 evaluations. In 90% (2337), residents received chest x-rays either in the nursing home or on hospital transfer. In 3 additional cases crucial information was missing from nursing home records. This left for final analysis 2334 episodes in 1474 individuals Figure 1.

Fifty-five percent of radiographs were interpreted as negative, 12% showed possible pneumonia, and 33% showed probable pneumonia. Most nursing home residents with pneumonia had few presenting symptoms; 80% had 3 or fewer respiratory or general symptoms. However, only 7.5% of subjects evaluated had no respiratory symptoms. Table 2 shows the relationship of selected variables to radiographic findings of absent, possible, or probable pneumonia. Though a few signs and symptoms are more common in those with positive (possible or probable pneumonia) than negative chest x-ray results, most did not discriminate at all. Fever (temperature Ž38°C) was present in 44.4% of positives but only 28.5% of negatives (P=.001).

Multivariable Analysis and Prediction Score

Our GEE model to predict radiographic pneumonia includes 3 vital sign abnormalities (fever, rapid pulse, and rapid respiratory rate), 2 lung findings (presence of crackles and absence of wheezes), 2 potential indicators of delirium (somnolence or decreased alertness and acute confusion), and elevated white blood count. Table 3 reports GEE estimates for the entire sample. Though only exhibiting fair overall performance, the model did well at distinguishing subjects with a high probability of pneumonia. In the 20% of subjects with the highest predicted risks, more than two thirds had pneumonia.

For the full range of values, the model derived on the development sample showed a c-statistic of 0.672, which reduced to 0.632 in the validation sample. A value of 1.0 would indicate perfect discrimination between those who did and did not have radiographic pneumonia, while a value of 0.5 would indicate no better than chance discrimination. Model calibration was not acceptable in the validation sample (Hosmer-Lemeshow goodness-of-fit statistic, P=.008). Inspection suggested the disagreement between predicted and observed probability of pneumonia was primarily with lower-risk estimates.

Because the model performed relatively well at distinguishing subjects very likely to have pneumonia, we created a simple point system aimed at identifying such high-risk individuals. Table 4 shows the scoring system. For 33% of subjects (score Ž3), there was a 56% or higher probability of radiographic pneumonia. An additional 24% of subjects (score of 2) had 44% probability of radiographic pneumonia. However, even those with the lowest scores (-1 to 0, 15% of subjects) still had a 24% probability of pneumonia. The relationship between the score and the probability of radiographic evidence of pneumonia is shown in Figure W1.*

Discussion

In a large community-based sample, we considered presenting symptoms, signs, and laboratory findings associated with radiographic pneumonia. Individual findings discriminated poorly, and we could not separate out a very-low-risk group. However, our simple scoring system identified approximately one third to slightly more than one half with high probability of pneumonia—individuals who might be treated without a confirmatory chest x-ray. If our data are confirmed, they suggest a simple clinical strategy in patients with respiratory or general symptoms Table 1 that might suggest pneumonia: (1) if there are no respiratory symptoms, consider other conditions, such as a urinary tract infection, that might fully explain the symptoms; (2) obtain information to apply our symptom score Table 4; (3) for those with scores of 2 or higher (some might choose 3 instead), treat for pneumonia; (4) for those with scores of -1, 0, or 1, obtain a chest radiograph as a guide to treatment.

Considering individual findings, fever was significantly more common in pneumonia, but only 43% of those with possible or probable pneumonia had a temperature of at least 38°C. This reaffirms common wisdom and previous findings that fever is frequently absent in elderly people with pneumonia.^9,19 We also confirmed that few signs or symptoms are the norm for nursing home-acquired pneumonia.

Chest examination findings also do not adequately distinguish patients with and without pneumonia Table 2. Also, even expert physicians frequently differ on lung examination findings.²⁰ Nonetheless, presence of crackles and absence of wheezing contribute to our scoring system. Both findings are seen with multiple conditions, but in our data crackles are slightly more associated with pneumonia, while wheezing is more strongly associated with other diseases.

The other components of our scoring system are clinical factors commonly associated with pneumonia. Though none individually discriminates well between those with and without pneumonia Table 2, several combined serve to identify a high-risk group.

Four previous studies from emergency department or outpatient settings developed clinical prediction rules to identify pneumonia.^21-24 Criteria for identifying subjects varied substantially, and each rule has limited accuracy in predicting radiographic pneumonia.²⁰ We had adequate data to evaluate 3 of the rules.^21-23 As is usually the case when transporting a prediction rule to a new sample, none performed any better than our rule (data not shown). Our sample created the very difficult challenge for any prediction rule of a very high overall prevalence of pneumonia (45%). That made it unlikely that we could identify a low-risk group in whom x-ray studies could be readily forgone, but we were able to identify a highrisk group.

Limitations

Our findings are subject to several limitations. All facilities in our study were located in central or eastern Missouri, and not all physicians or eligible residents in those facilities participated. Compared with national data, we studied an unusually representative sample of nursing home residents from 36 facilities, including rural and urban locations. Also, in episodes excluded because of physician nonparticipation, residents were very similar to included residents in age, vital signs, and presenting symptoms (data available on request). More important, we lack an independent validation sample from a different cohort. Clinical prediction rules usually do not perform as well in independent samples. This is exemplified by the poor performance of the 3 rules we considered from other settings. Overall, our logistic model was only modest in discriminating and was not well calibrated for low-risk episodes in our reserved validation sample. Although we have developed a promising scoring system to identify residents with high probability of radiographic pneumonia, it needs to be validated in other samples of nursing home residents to determine its ultimate usefulness. For all these reasons, our results may not generalize.

Also, although we identified residents prospectively, project nurses were unable to evaluate 9.2% of residents before transfer to a hospital. Clinical findings abstracted from medical records, such as lung findings, may not have been complete. It is also possible that project nurses could have missed some important findings. However, our staff provided a higher level of expertise than is typically available in nursing homes. In fact, this may limit application of our findings. Nursing home staff vary widely in their ability to accurately examine residents or even identify illness. In many instances, facility staff had not obtained vital signs at the point when we identified a resident as ill enough to qualify for an evaluation.²⁵ Therefore, in many nursing homes, physicians may lack confidence to apply our rule without an evaluation by a physician, advanced practice nurse, or physician assistant.

Finally, determining whether subjects had pneumonia primarily depended on our classification of radiographic reports. Though radiographs generally included 2 views, many were portable films of variable quality, and frequently there was no previous radiograph for comparison. In some subjects with pneumonia, radiographic infiltrates might not yet have developed. Also, even under ideal conditions, radiologists commonly disagree on the presence of pneumonia.²⁶ Some subjects may have been misclassified. However, unless radiographic technique or interpretation was specifically related to clinical predictors, misclassification would simply diminish the relationship of predictors to pneumonia rather than creating a bias. We reviewed reports rather than radiographs, because that is the information usually available to clinicians faced with diagnosis and treatment decisions. We also paid special attention to avoiding any bias in the interpretations. All data were recorded before interpreting radiology reports and the interpretations were performed independent of clinical data. We also made special efforts to assure consistency in labeling radiology reports as possible, probable, or negative for pneumonia. When lack of agreement persisted, the study radiologist reinterpreted the actual films.

Conclusions

Most nursing home residents with pneumonia have few symptoms. We created a simple scoring to identify nursing home residents who have a high probability of radiographic pneumonia. If our results are confirmed, physicians might consider initiating treatment without an x-ray in such residents. Low scores do not rule out pneumonia, and most physicians would want to press for further diagnosis or treatment in this group.

Acknowledgments

This study was supported by the Agency for Healthcare Research and Quality (grant HS08551) and Dr Mehr’s Robert Wood Johnson Foundation Generalist Physician Faculty Scholars award. Dr Kruse was partially supported by an Institutional National Research Service Award (PE10038) from the Health Resources and Services Administration. Our project would not have been possible without the support of the many attending physicians, administrators, and staff of the involved nursing homes. Dr Clive Levine re-read more than 200 radiographs; Karen Davenport provided crucial administrative support; and Karen Madrone, MPA, assisted with manuscript preparation. Many other unnamed project staff also contributed.

Pneumonia is a leading cause of morbidity, mortality, and hospitalization of nursing home residents.^1-8 Atypical presentations and fewer presenting signs and symptoms in older patients complicate diagnosis.^9,10 Also, clinician (physician, nurse practitioner, and physician assistant) visits to nursing homes are often sporadic, and radiology facilities are rarely on the premises. As a consequence, residents are commonly sent to emergency departments for evaluation,^4,11,12 which undoubtedly contributes to a high hospitalization rate.

Clinicians who periodically see nursing home residents could benefit from a simple clinical tool to identify pneumonia. No large studies of community nursing home residents have systematically studied findings associated with pneumonia. As part of the Missouri LRI Project, we examined how well clinical findings predict radiographic pneumonia.

Methods

The Missouri LRI Project was a prospective observational study in 36 nursing homes in Central Missouri and St. Louis designed to investigate predictors of 2 outcomes of lower respiratory infections (LRIs): mortality and functional decline. Potential cases were identified from August 15, 1995, through September 29, 1998; however, all facilities were not involved until fall 1997. Study facilities were similar in size, ownership, and occupancy to national estimates from the 1995 National Nursing Home Survey (data available on request).¹³

We trained nursing home staff to report ill residents with any of 6 respiratory symptoms (eg, cough, dyspnea, sputum production) or 6 general symptoms (eg, fever, decline in mobility, mental status changes). Project nurses called and visited facilities frequently to reinforce reporting. Under a physician-authorized protocol, ill residents with a possible LRI received a standardized evaluation by a trained project nurse and usually a chest radiograph, complete blood count, and a chemistry panel. Complete criteria for triggering an evaluation are listed in Table 1. For this paper, we were concerned with the 90% of evaluated residents who received a chest radiograph. Criteria for excluding residents from evaluation are summarized in the Figure 1.

The nurse evaluation included an inventory of current symptoms, a review of important chronic conditions (eg, congestive heart failure), and a targeted physical examination. The examination included vital signs and the following body areas or systems: ears, nose, and throat; cardiac; abdominal; neurologic; extremities; skin; and a detailed lung examination. Most project nurses had advanced practice training; the remainder had extensive clinical experience and training in physical assessment. All received an individualized training session with a project geriatrician. Project nurses had substantially more experience than the nursing home staff, who usually report clinical findings to physicians.

Results of the evaluation were reported to the attending physician, who made all treatment decisions. Since the evaluations were clinically appropriate care authorized by individual attending physicians, the institutional review boards that reviewed the project allowed us to substantially simplify the consent process to a simple acceptance or refusal of the evaluation. In 9.2% of evaluations the resident was transferred to the hospital before project nurses could complete a physical assessment. In these instances, we obtained vital sign and clinical examination data from hospital records.

Radiographic Classification

Since all subjects had at least one illness symptom, for this analysis we classified the presence or absence of pneumonia on the basis of reported radiographic findings. Using defined criteria, 2 clinicians independently separated radiology reports into 3 categories: (a) negative, (b) possible, or (c) probable or definite for pneumonia (hereafter, probable pneumonia). For example, a report describing “new left lower lobe infiltrate suggestive of pneumonia” would have been rated as probable, while a report indicating “possible infiltrate” or “infiltrate suggestive of pneumonia or congestive heart failure” would have been rated as possible. As radiologists rarely provide completely unequivocal readings, we did not separate probable and definite pneumonia. In St. Louis 2 clinicians evaluated the reports, and in central Missouri 2 of 4 clinicians considered each report. Where there was disagreement, all 6 raters from the 2 sites independently reviewed the reports and then attempted to reach consensus. For 13% of radiographs, the project radiologist independently interpreted the actual films. This occurred when: (1) consensus could not be achieved; or (2) consensus was possible pneumonia, but probable pneumonia was needed to quality the episode as an LRI under the project definition.

Statistical Analyses

As residents could be included more than once, the unit of analysis throughout is episode of illness. In our major analysis, we developed a multivariable logistic model to estimate the probability of radiographic pneumonia (possible or probable). Before beginning modeling, we imputed mean values for missing continuous data and the largest category for missing dichotomous variables (the number of missing values is noted in Table 2). Data imputation is less biased than dropping cases in developing multivariable models.¹⁴

Illness episodes were then randomly assigned to a two thirds model-development and a one third model-validation sample. On the basis of the literature and clinical experience, we defined categories of variables that might relate to the presence or absence of pneumonia, such as lung findings (eg, crackles, wheezes), respiratory symptoms (eg, cough, sputum production), vital signs, findings of delirium (eg, acute confusion, decreased alertness), and laboratory findings. Restricting our focus to the development sample, we selected the best representatives of these groups on clinical and statistical grounds. For continuous variables, we considered the shape of the relationship to presence of pneumonia. For example, both very high and very low pulse rates predicted increased risk of pneumonia. In such cases, we considered several different ways to represent the variable in the model. We also limited the range of some variables to avoid undue influence of outliers (approximately the 1% most extreme values). For example, pulse rate above 140 was set equal to 140.

We then employed forward and backward stepwise logistic regression with possible or probable pneumonia (also referred to as positive x-ray results) as the dependent variable. For final model inclusion, we required variables to bear a plausible relationship to the diagnosis of pneumonia and meet a statistical significance criterion (a=.05).

To obtain final estimates of the relationship of each model variable to pneumonia probability, we considered adjustments for 2 kinds of correlation within our data: (1) individuals are nested within facilities, and (2) subjects could be represented by more than one episode.¹⁵ Using generalized estimating equations (GEE) in Proc Genmod in SAS software (SAS Institute, Cary, NC),¹⁶ we noted that the effect of facilities was minor, but the effect of repeat episodes by the same subject was more marked. Consequently, we used GEE to account for repeat episodes on subjects. To avoid unstable GEE estimates, we dropped 5 episodes in the development sample and 8 in the overall sample (episodes beyond the 5th and 6th per individual, respectively).

Using parameter estimates from the development sample, we tested the model’s discrimination and calibration in the validation sample.¹⁷ To assess discrimination, we used the c-statistic, which evaluates among all possible pairs of individuals whether those with higher predicted risk are more likely to die. The c-statistic is also equal to the area under the receiver operating characteristic curve. To assess calibration—agreement between observed and predicted mortality over the range of predicted risk—we used the Hosmer-Lemeshow goodness-of-fit statistic.¹⁸ We then used estimates fitted to the overall sample to develop a simple additive score to provide a clinically usable prediction rule. Statistical analyses were performed with SAS statistical software.¹⁶

Results

Project nurses performed 2592 evaluations. In 90% (2337), residents received chest x-rays either in the nursing home or on hospital transfer. In 3 additional cases crucial information was missing from nursing home records. This left for final analysis 2334 episodes in 1474 individuals Figure 1.

Fifty-five percent of radiographs were interpreted as negative, 12% showed possible pneumonia, and 33% showed probable pneumonia. Most nursing home residents with pneumonia had few presenting symptoms; 80% had 3 or fewer respiratory or general symptoms. However, only 7.5% of subjects evaluated had no respiratory symptoms. Table 2 shows the relationship of selected variables to radiographic findings of absent, possible, or probable pneumonia. Though a few signs and symptoms are more common in those with positive (possible or probable pneumonia) than negative chest x-ray results, most did not discriminate at all. Fever (temperature Ž38°C) was present in 44.4% of positives but only 28.5% of negatives (P=.001).

Multivariable Analysis and Prediction Score

Our GEE model to predict radiographic pneumonia includes 3 vital sign abnormalities (fever, rapid pulse, and rapid respiratory rate), 2 lung findings (presence of crackles and absence of wheezes), 2 potential indicators of delirium (somnolence or decreased alertness and acute confusion), and elevated white blood count. Table 3 reports GEE estimates for the entire sample. Though only exhibiting fair overall performance, the model did well at distinguishing subjects with a high probability of pneumonia. In the 20% of subjects with the highest predicted risks, more than two thirds had pneumonia.

For the full range of values, the model derived on the development sample showed a c-statistic of 0.672, which reduced to 0.632 in the validation sample. A value of 1.0 would indicate perfect discrimination between those who did and did not have radiographic pneumonia, while a value of 0.5 would indicate no better than chance discrimination. Model calibration was not acceptable in the validation sample (Hosmer-Lemeshow goodness-of-fit statistic, P=.008). Inspection suggested the disagreement between predicted and observed probability of pneumonia was primarily with lower-risk estimates.

Because the model performed relatively well at distinguishing subjects very likely to have pneumonia, we created a simple point system aimed at identifying such high-risk individuals. Table 4 shows the scoring system. For 33% of subjects (score Ž3), there was a 56% or higher probability of radiographic pneumonia. An additional 24% of subjects (score of 2) had 44% probability of radiographic pneumonia. However, even those with the lowest scores (-1 to 0, 15% of subjects) still had a 24% probability of pneumonia. The relationship between the score and the probability of radiographic evidence of pneumonia is shown in Figure W1.*

Discussion

In a large community-based sample, we considered presenting symptoms, signs, and laboratory findings associated with radiographic pneumonia. Individual findings discriminated poorly, and we could not separate out a very-low-risk group. However, our simple scoring system identified approximately one third to slightly more than one half with high probability of pneumonia—individuals who might be treated without a confirmatory chest x-ray. If our data are confirmed, they suggest a simple clinical strategy in patients with respiratory or general symptoms Table 1 that might suggest pneumonia: (1) if there are no respiratory symptoms, consider other conditions, such as a urinary tract infection, that might fully explain the symptoms; (2) obtain information to apply our symptom score Table 4; (3) for those with scores of 2 or higher (some might choose 3 instead), treat for pneumonia; (4) for those with scores of -1, 0, or 1, obtain a chest radiograph as a guide to treatment.

Considering individual findings, fever was significantly more common in pneumonia, but only 43% of those with possible or probable pneumonia had a temperature of at least 38°C. This reaffirms common wisdom and previous findings that fever is frequently absent in elderly people with pneumonia.^9,19 We also confirmed that few signs or symptoms are the norm for nursing home-acquired pneumonia.

Chest examination findings also do not adequately distinguish patients with and without pneumonia Table 2. Also, even expert physicians frequently differ on lung examination findings.²⁰ Nonetheless, presence of crackles and absence of wheezing contribute to our scoring system. Both findings are seen with multiple conditions, but in our data crackles are slightly more associated with pneumonia, while wheezing is more strongly associated with other diseases.

The other components of our scoring system are clinical factors commonly associated with pneumonia. Though none individually discriminates well between those with and without pneumonia Table 2, several combined serve to identify a high-risk group.

Four previous studies from emergency department or outpatient settings developed clinical prediction rules to identify pneumonia.^21-24 Criteria for identifying subjects varied substantially, and each rule has limited accuracy in predicting radiographic pneumonia.²⁰ We had adequate data to evaluate 3 of the rules.^21-23 As is usually the case when transporting a prediction rule to a new sample, none performed any better than our rule (data not shown). Our sample created the very difficult challenge for any prediction rule of a very high overall prevalence of pneumonia (45%). That made it unlikely that we could identify a low-risk group in whom x-ray studies could be readily forgone, but we were able to identify a highrisk group.

Limitations

Our findings are subject to several limitations. All facilities in our study were located in central or eastern Missouri, and not all physicians or eligible residents in those facilities participated. Compared with national data, we studied an unusually representative sample of nursing home residents from 36 facilities, including rural and urban locations. Also, in episodes excluded because of physician nonparticipation, residents were very similar to included residents in age, vital signs, and presenting symptoms (data available on request). More important, we lack an independent validation sample from a different cohort. Clinical prediction rules usually do not perform as well in independent samples. This is exemplified by the poor performance of the 3 rules we considered from other settings. Overall, our logistic model was only modest in discriminating and was not well calibrated for low-risk episodes in our reserved validation sample. Although we have developed a promising scoring system to identify residents with high probability of radiographic pneumonia, it needs to be validated in other samples of nursing home residents to determine its ultimate usefulness. For all these reasons, our results may not generalize.

Also, although we identified residents prospectively, project nurses were unable to evaluate 9.2% of residents before transfer to a hospital. Clinical findings abstracted from medical records, such as lung findings, may not have been complete. It is also possible that project nurses could have missed some important findings. However, our staff provided a higher level of expertise than is typically available in nursing homes. In fact, this may limit application of our findings. Nursing home staff vary widely in their ability to accurately examine residents or even identify illness. In many instances, facility staff had not obtained vital signs at the point when we identified a resident as ill enough to qualify for an evaluation.²⁵ Therefore, in many nursing homes, physicians may lack confidence to apply our rule without an evaluation by a physician, advanced practice nurse, or physician assistant.

Finally, determining whether subjects had pneumonia primarily depended on our classification of radiographic reports. Though radiographs generally included 2 views, many were portable films of variable quality, and frequently there was no previous radiograph for comparison. In some subjects with pneumonia, radiographic infiltrates might not yet have developed. Also, even under ideal conditions, radiologists commonly disagree on the presence of pneumonia.²⁶ Some subjects may have been misclassified. However, unless radiographic technique or interpretation was specifically related to clinical predictors, misclassification would simply diminish the relationship of predictors to pneumonia rather than creating a bias. We reviewed reports rather than radiographs, because that is the information usually available to clinicians faced with diagnosis and treatment decisions. We also paid special attention to avoiding any bias in the interpretations. All data were recorded before interpreting radiology reports and the interpretations were performed independent of clinical data. We also made special efforts to assure consistency in labeling radiology reports as possible, probable, or negative for pneumonia. When lack of agreement persisted, the study radiologist reinterpreted the actual films.

Conclusions

Most nursing home residents with pneumonia have few symptoms. We created a simple scoring to identify nursing home residents who have a high probability of radiographic pneumonia. If our results are confirmed, physicians might consider initiating treatment without an x-ray in such residents. Low scores do not rule out pneumonia, and most physicians would want to press for further diagnosis or treatment in this group.

Acknowledgments

This study was supported by the Agency for Healthcare Research and Quality (grant HS08551) and Dr Mehr’s Robert Wood Johnson Foundation Generalist Physician Faculty Scholars award. Dr Kruse was partially supported by an Institutional National Research Service Award (PE10038) from the Health Resources and Services Administration. Our project would not have been possible without the support of the many attending physicians, administrators, and staff of the involved nursing homes. Dr Clive Levine re-read more than 200 radiographs; Karen Davenport provided crucial administrative support; and Karen Madrone, MPA, assisted with manuscript preparation. Many other unnamed project staff also contributed.

Common medical problems, especially those that are self-limited or in their early phases, can be best studied in community practice settings where they are usually diagnosed and managed. Practice-based research provides one method to conduct studies of these problems. Often practice-based research physicians are linked together in practice-based research networks (PBRNs), thus forming, in effect, laboratories of community practices.^1-3

The methodologic limitations of these laboratories are of concern and have not been extensively explored. Although it has been adequately demonstrated that the patient populations and the problems addressed in participating practices are comparable to patients and problems in the general population,^4-6 the question of the selection bias of the clinicians has been raised.⁴

As research involvement can be a costly endeavor for the individual physician,⁷ participation in a research protocol—to some extent—may be related to the intensity of practice (ie, the volume of patients seen and services provided). It has been shown that high-volume practices differ from low-volume practices⁸ in that high-volume practices provide lower rates of preventive services and generate lower patient satisfaction. One may anticipate that physicians with more discretionary time (ie, fewer patients) may be better able to fully participate in research activities. There have been no direct studies of the impact of practice volume on the reporting of medical problems and compliance in research studies. This study, conducted as part of a larger Wisconsin Research Network (WReN) study of dyspepsia in primary care settings, is a first step in that direction.

Methods

Eighteen family physicians, making up the Practice-Based Research Group of WReN practices, volunteered to participate in a study of the initial management of dyspepsia in primary care.⁹ As part of the study protocol, participants were requested to record the number of adult patients presenting with dyspepsia and the total number of patients seen in their clinic for each week of the 12-month study. Dyspepsia was defined as pain in the upper abdomen lasting for at least 2 weeks and not attributable to cardiac or pulmonary disease or trauma. Data was collected for both initial and follow-up visits. Participants were instructed to complete a 1-page data instrument for each dyspeptic patient at the time of the visit. Each instrument contained 68 data elements and took up to 5 minutes to complete. Data forms were mailed to the study coordinator on a monthly basis. Data collection began on January 30, 1995, and continued through February 2, 1996.

An average weekly patient volume was calculated for each physician, as was the reported rate of dyspepsia visits in their practice. The patient volume was estimated for each physician by summing the weekly patient totals and dividing by the number of weeks during which the physician saw patients in the clinic and participated in the study. The reported rate of dyspepsia visits for each physician was estimated as the total number of patient visits reported meeting the study criteria for dyspepsia divided by the total number of patients seen during the study period.

Following completion of primary data collection, a demographic questionnaire was sent out to all 18 participants. The questionnaire distribution occurred approximately 4 months after data collection and during a chart review phase of the primary study. The chart review was performed by a research assistant and did not involve the participating physicians. One question, included to assess compliance with the study protocol, asked, “On a 10-point scale, how compliant were you at recording data for all qualifying dyspepsia patients during the weeks that you were involved with this study?” Responses were circled on a scale from 1 (poor) to 10 (perfect). Type of practice (solo, group multispecialty, or academic) was also obtained. Seventeen of the 18 questionnaires were completed and returned.

MINITAB was used for statistical analyses. Descriptive statistics were calculated for the outcome variables. Because data for reported rate of dyspepsia visits and compliance were not normally distributed, Spearman rank correlation (“ = 0.05) was used to test the hypotheses that practice volume, protocol compliance, and reported rate of dyspepsia visits were correlated. The one solo practitioner was placed with the group practice physicians because of a high level of similarity in all outcome variables. Because differences were noted among the practice types, the Kruskal-Wallis test was used to assess differences in patient volume, compliance, and reported rate of dyspepsia visits.

Results

The average participant in this study was a 46-year-old male physician who had been in practice for 17 years and saw 61.5 patients per week Table w1. Eight physicians were located in group practices, while 5 were in multispecialty and 3 were in academic practices. The mean reported rate of dyspepsia visits was 7.7 cases per 1000 patient visits. Initial dyspepsia visits accounted for 118 of the 231 reported visits for dyspepsia (0.51%), with a total of 45,337 patient visits recorded by participating physicians.

The average participant recorded visits over 43.2 weeks of the possible 53-week study (81.5% overall participation rate). The average self-reported compliance with the study protocol was 6.7 on a 10-point scale but with a very wide range (from 1 to 10). Significant differences among practice types were found in patient volume, reported rate of dyspepsia visits, and self-reported compliance Table 2. Participants from group practices had the highest patient volumes but the lowest rate of dyspepsia visits and compliance. Academic physicians saw the least number of patients but had the highest reported rate of dyspepsia visits and compliance.

Significant negative rank correlations were found to exist between patient volume and reported rate of dyspepsia visits (Figure 1: rs = -0.548; P .05) and between patient volume and compliance with protocol (Figure 2: rs = -0.490; P .05). A significant positive rank correlation was found between compliance with protocol and rate of dyspepsia visits (Figure 3 (: rs = 0.551; P .05). No significant correlation existed between the number of weeks of participation and patient volume (rs = -0.303), rate of dyspepsia visits (rs = 0.065), or compliance with protocol (rs = 0.415).

Discussion

Practice volume can have a significant effect on physicians’ reporting rates in practice-based studies. The rate of dyspepsia visits, as measured by the identification of patients meeting study criteria and having a completed data form, was negatively related to the number of patients seen per week by the physician. Practice volume appears to be linked to reporting by way of compliance. As an extension, it appears that physicians are generally accurate in self-assessment of their compliance with a protocol.

Although previous evaluations of PBRNs have demonstrated high levels of accuracy within reported data,¹⁰ the results reported here are somewhat disturbing. If other studies show similar results, the idea that PBRNs can assess prevalence of medical conditions could be called into question. Also, there may be a bias in the higher? volume practices for patients with more severe symptoms to be reported in preference to those with less “attention getting” symptoms, or in low-volume practices to seek out problems for which the patient did not seek attention. Consequently, even when a medical problem is identified, there may be patient selection bias toward those with more or less severe symptoms.

Additional burden and lack of practice support were common reasons for withdrawing from participation in PBRNs.¹¹ Overall participation and compliance with a research protocol, therefore, is likely related to the complexity of that protocol. While the reported rate of dyspepsia visits was negatively related to practice volume, the simple reporting of a weekly tally of patients seen in clinic was not. Consequently, compliance-sensitive measurements (eg, prevalence) may need simple time-efficient protocols. For example, full compliance with the protocol for the approximately 1050 physicians currently involved in the Centers for Disease Control and Prevention US Influenza Sentinel Physician Surveillance Network requires less than 3 minutes per week. This surveillance network for monitoring prevalence of influenza-like illness is a highly accurate, timely, and valued component of influenza surveillance.¹² Other enhancements for study protocols may include decreased periods for data gathering, use of intermittent reporting, and use of other office staff for case identification.

Limitations

This study is limited by a potential lack of generalizability. It is an observational study of physician behavior around a complex and relatively high-burden data collection instrument. There were no true standards regarding prevalence of dyspepsia at any location, thus allowing for the possibility that patient populations differed significantly among sites. Self-reported compliance with the research protocol was based on recall 4 months after the end of the data collection period. Also, some of the effect attributable to patient volume could alternatively result from the types of physicians involved in this study.

Academic physicians, with low practice volumes, may be more likely to be compliant with research protocols in general, regardless of their practice volumes. Because of the small sample size, however, this alternate hypothesis cannot be examined independently. With the exclusion of the academic physicians, relationships between the variables demonstrated the same trends, but the Spearman rank correlations were no longer significant (n = 14; patient volume vs rate: rs = -0.345; patient volume vs compliance: rs = -0.187; compliance vs rate: rs = 0.379).

This study does, however, challenge other investigators using PBRNs to revisit suitable data to determine similar patterns. Also, a simple assessment of participant compliance might prove to be an essential enhancement of future practice-based research.

Conclusions

Even encumbered with potential methodologic dilemmas, practice-based research studies may be the only way to approach many common medical issues in the context of the communities in which they occur.^1-3 For example, while selection bias in reporting of dyspepsia is clearly a problem in this example, the selection bias is still far less severe than it would be in the gastrointestinal specialty clinic of a referral center. Likewise, if nonreferred conditions are to be tracked over extensive periods of time, the use of community settings is essential, as was done with a recent longitudinal study of depression.¹³

Acknowledgments

Funding for this study was provided through a grant from the American Academy of Family Physicians. We thank the following participants of the WReN Practice-Based Research Group: R. Baldwin, E. Barr, D. Baumgardner, A. Berlage, M. Chin, D. Erickson, R. Erickson, G. Gay, M. Grajewski, D. Hahn, T. Hankey, D. Madlon-Kay, A. Marquis, E. Ott, D. Pine, and L. Radant.

Common medical problems, especially those that are self-limited or in their early phases, can be best studied in community practice settings where they are usually diagnosed and managed. Practice-based research provides one method to conduct studies of these problems. Often practice-based research physicians are linked together in practice-based research networks (PBRNs), thus forming, in effect, laboratories of community practices.^1-3

The methodologic limitations of these laboratories are of concern and have not been extensively explored. Although it has been adequately demonstrated that the patient populations and the problems addressed in participating practices are comparable to patients and problems in the general population,^4-6 the question of the selection bias of the clinicians has been raised.⁴

As research involvement can be a costly endeavor for the individual physician,⁷ participation in a research protocol—to some extent—may be related to the intensity of practice (ie, the volume of patients seen and services provided). It has been shown that high-volume practices differ from low-volume practices⁸ in that high-volume practices provide lower rates of preventive services and generate lower patient satisfaction. One may anticipate that physicians with more discretionary time (ie, fewer patients) may be better able to fully participate in research activities. There have been no direct studies of the impact of practice volume on the reporting of medical problems and compliance in research studies. This study, conducted as part of a larger Wisconsin Research Network (WReN) study of dyspepsia in primary care settings, is a first step in that direction.

Methods

Eighteen family physicians, making up the Practice-Based Research Group of WReN practices, volunteered to participate in a study of the initial management of dyspepsia in primary care.⁹ As part of the study protocol, participants were requested to record the number of adult patients presenting with dyspepsia and the total number of patients seen in their clinic for each week of the 12-month study. Dyspepsia was defined as pain in the upper abdomen lasting for at least 2 weeks and not attributable to cardiac or pulmonary disease or trauma. Data was collected for both initial and follow-up visits. Participants were instructed to complete a 1-page data instrument for each dyspeptic patient at the time of the visit. Each instrument contained 68 data elements and took up to 5 minutes to complete. Data forms were mailed to the study coordinator on a monthly basis. Data collection began on January 30, 1995, and continued through February 2, 1996.

An average weekly patient volume was calculated for each physician, as was the reported rate of dyspepsia visits in their practice. The patient volume was estimated for each physician by summing the weekly patient totals and dividing by the number of weeks during which the physician saw patients in the clinic and participated in the study. The reported rate of dyspepsia visits for each physician was estimated as the total number of patient visits reported meeting the study criteria for dyspepsia divided by the total number of patients seen during the study period.

Following completion of primary data collection, a demographic questionnaire was sent out to all 18 participants. The questionnaire distribution occurred approximately 4 months after data collection and during a chart review phase of the primary study. The chart review was performed by a research assistant and did not involve the participating physicians. One question, included to assess compliance with the study protocol, asked, “On a 10-point scale, how compliant were you at recording data for all qualifying dyspepsia patients during the weeks that you were involved with this study?” Responses were circled on a scale from 1 (poor) to 10 (perfect). Type of practice (solo, group multispecialty, or academic) was also obtained. Seventeen of the 18 questionnaires were completed and returned.

MINITAB was used for statistical analyses. Descriptive statistics were calculated for the outcome variables. Because data for reported rate of dyspepsia visits and compliance were not normally distributed, Spearman rank correlation (“ = 0.05) was used to test the hypotheses that practice volume, protocol compliance, and reported rate of dyspepsia visits were correlated. The one solo practitioner was placed with the group practice physicians because of a high level of similarity in all outcome variables. Because differences were noted among the practice types, the Kruskal-Wallis test was used to assess differences in patient volume, compliance, and reported rate of dyspepsia visits.

Results

The average participant in this study was a 46-year-old male physician who had been in practice for 17 years and saw 61.5 patients per week Table w1. Eight physicians were located in group practices, while 5 were in multispecialty and 3 were in academic practices. The mean reported rate of dyspepsia visits was 7.7 cases per 1000 patient visits. Initial dyspepsia visits accounted for 118 of the 231 reported visits for dyspepsia (0.51%), with a total of 45,337 patient visits recorded by participating physicians.

The average participant recorded visits over 43.2 weeks of the possible 53-week study (81.5% overall participation rate). The average self-reported compliance with the study protocol was 6.7 on a 10-point scale but with a very wide range (from 1 to 10). Significant differences among practice types were found in patient volume, reported rate of dyspepsia visits, and self-reported compliance Table 2. Participants from group practices had the highest patient volumes but the lowest rate of dyspepsia visits and compliance. Academic physicians saw the least number of patients but had the highest reported rate of dyspepsia visits and compliance.

Significant negative rank correlations were found to exist between patient volume and reported rate of dyspepsia visits (Figure 1: rs = -0.548; P .05) and between patient volume and compliance with protocol (Figure 2: rs = -0.490; P .05). A significant positive rank correlation was found between compliance with protocol and rate of dyspepsia visits (Figure 3 (: rs = 0.551; P .05). No significant correlation existed between the number of weeks of participation and patient volume (rs = -0.303), rate of dyspepsia visits (rs = 0.065), or compliance with protocol (rs = 0.415).

Discussion

Practice volume can have a significant effect on physicians’ reporting rates in practice-based studies. The rate of dyspepsia visits, as measured by the identification of patients meeting study criteria and having a completed data form, was negatively related to the number of patients seen per week by the physician. Practice volume appears to be linked to reporting by way of compliance. As an extension, it appears that physicians are generally accurate in self-assessment of their compliance with a protocol.

Although previous evaluations of PBRNs have demonstrated high levels of accuracy within reported data,¹⁰ the results reported here are somewhat disturbing. If other studies show similar results, the idea that PBRNs can assess prevalence of medical conditions could be called into question. Also, there may be a bias in the higher? volume practices for patients with more severe symptoms to be reported in preference to those with less “attention getting” symptoms, or in low-volume practices to seek out problems for which the patient did not seek attention. Consequently, even when a medical problem is identified, there may be patient selection bias toward those with more or less severe symptoms.

Additional burden and lack of practice support were common reasons for withdrawing from participation in PBRNs.¹¹ Overall participation and compliance with a research protocol, therefore, is likely related to the complexity of that protocol. While the reported rate of dyspepsia visits was negatively related to practice volume, the simple reporting of a weekly tally of patients seen in clinic was not. Consequently, compliance-sensitive measurements (eg, prevalence) may need simple time-efficient protocols. For example, full compliance with the protocol for the approximately 1050 physicians currently involved in the Centers for Disease Control and Prevention US Influenza Sentinel Physician Surveillance Network requires less than 3 minutes per week. This surveillance network for monitoring prevalence of influenza-like illness is a highly accurate, timely, and valued component of influenza surveillance.¹² Other enhancements for study protocols may include decreased periods for data gathering, use of intermittent reporting, and use of other office staff for case identification.

Limitations

This study is limited by a potential lack of generalizability. It is an observational study of physician behavior around a complex and relatively high-burden data collection instrument. There were no true standards regarding prevalence of dyspepsia at any location, thus allowing for the possibility that patient populations differed significantly among sites. Self-reported compliance with the research protocol was based on recall 4 months after the end of the data collection period. Also, some of the effect attributable to patient volume could alternatively result from the types of physicians involved in this study.

Academic physicians, with low practice volumes, may be more likely to be compliant with research protocols in general, regardless of their practice volumes. Because of the small sample size, however, this alternate hypothesis cannot be examined independently. With the exclusion of the academic physicians, relationships between the variables demonstrated the same trends, but the Spearman rank correlations were no longer significant (n = 14; patient volume vs rate: rs = -0.345; patient volume vs compliance: rs = -0.187; compliance vs rate: rs = 0.379).

This study does, however, challenge other investigators using PBRNs to revisit suitable data to determine similar patterns. Also, a simple assessment of participant compliance might prove to be an essential enhancement of future practice-based research.

Conclusions

Even encumbered with potential methodologic dilemmas, practice-based research studies may be the only way to approach many common medical issues in the context of the communities in which they occur.^1-3 For example, while selection bias in reporting of dyspepsia is clearly a problem in this example, the selection bias is still far less severe than it would be in the gastrointestinal specialty clinic of a referral center. Likewise, if nonreferred conditions are to be tracked over extensive periods of time, the use of community settings is essential, as was done with a recent longitudinal study of depression.¹³

Acknowledgments

Funding for this study was provided through a grant from the American Academy of Family Physicians. We thank the following participants of the WReN Practice-Based Research Group: R. Baldwin, E. Barr, D. Baumgardner, A. Berlage, M. Chin, D. Erickson, R. Erickson, G. Gay, M. Grajewski, D. Hahn, T. Hankey, D. Madlon-Kay, A. Marquis, E. Ott, D. Pine, and L. Radant.

Common medical problems, especially those that are self-limited or in their early phases, can be best studied in community practice settings where they are usually diagnosed and managed. Practice-based research provides one method to conduct studies of these problems. Often practice-based research physicians are linked together in practice-based research networks (PBRNs), thus forming, in effect, laboratories of community practices.^1-3

The methodologic limitations of these laboratories are of concern and have not been extensively explored. Although it has been adequately demonstrated that the patient populations and the problems addressed in participating practices are comparable to patients and problems in the general population,^4-6 the question of the selection bias of the clinicians has been raised.⁴

As research involvement can be a costly endeavor for the individual physician,⁷ participation in a research protocol—to some extent—may be related to the intensity of practice (ie, the volume of patients seen and services provided). It has been shown that high-volume practices differ from low-volume practices⁸ in that high-volume practices provide lower rates of preventive services and generate lower patient satisfaction. One may anticipate that physicians with more discretionary time (ie, fewer patients) may be better able to fully participate in research activities. There have been no direct studies of the impact of practice volume on the reporting of medical problems and compliance in research studies. This study, conducted as part of a larger Wisconsin Research Network (WReN) study of dyspepsia in primary care settings, is a first step in that direction.

Methods

Eighteen family physicians, making up the Practice-Based Research Group of WReN practices, volunteered to participate in a study of the initial management of dyspepsia in primary care.⁹ As part of the study protocol, participants were requested to record the number of adult patients presenting with dyspepsia and the total number of patients seen in their clinic for each week of the 12-month study. Dyspepsia was defined as pain in the upper abdomen lasting for at least 2 weeks and not attributable to cardiac or pulmonary disease or trauma. Data was collected for both initial and follow-up visits. Participants were instructed to complete a 1-page data instrument for each dyspeptic patient at the time of the visit. Each instrument contained 68 data elements and took up to 5 minutes to complete. Data forms were mailed to the study coordinator on a monthly basis. Data collection began on January 30, 1995, and continued through February 2, 1996.

An average weekly patient volume was calculated for each physician, as was the reported rate of dyspepsia visits in their practice. The patient volume was estimated for each physician by summing the weekly patient totals and dividing by the number of weeks during which the physician saw patients in the clinic and participated in the study. The reported rate of dyspepsia visits for each physician was estimated as the total number of patient visits reported meeting the study criteria for dyspepsia divided by the total number of patients seen during the study period.

Following completion of primary data collection, a demographic questionnaire was sent out to all 18 participants. The questionnaire distribution occurred approximately 4 months after data collection and during a chart review phase of the primary study. The chart review was performed by a research assistant and did not involve the participating physicians. One question, included to assess compliance with the study protocol, asked, “On a 10-point scale, how compliant were you at recording data for all qualifying dyspepsia patients during the weeks that you were involved with this study?” Responses were circled on a scale from 1 (poor) to 10 (perfect). Type of practice (solo, group multispecialty, or academic) was also obtained. Seventeen of the 18 questionnaires were completed and returned.

MINITAB was used for statistical analyses. Descriptive statistics were calculated for the outcome variables. Because data for reported rate of dyspepsia visits and compliance were not normally distributed, Spearman rank correlation (“ = 0.05) was used to test the hypotheses that practice volume, protocol compliance, and reported rate of dyspepsia visits were correlated. The one solo practitioner was placed with the group practice physicians because of a high level of similarity in all outcome variables. Because differences were noted among the practice types, the Kruskal-Wallis test was used to assess differences in patient volume, compliance, and reported rate of dyspepsia visits.

Results

The average participant in this study was a 46-year-old male physician who had been in practice for 17 years and saw 61.5 patients per week Table w1. Eight physicians were located in group practices, while 5 were in multispecialty and 3 were in academic practices. The mean reported rate of dyspepsia visits was 7.7 cases per 1000 patient visits. Initial dyspepsia visits accounted for 118 of the 231 reported visits for dyspepsia (0.51%), with a total of 45,337 patient visits recorded by participating physicians.

The average participant recorded visits over 43.2 weeks of the possible 53-week study (81.5% overall participation rate). The average self-reported compliance with the study protocol was 6.7 on a 10-point scale but with a very wide range (from 1 to 10). Significant differences among practice types were found in patient volume, reported rate of dyspepsia visits, and self-reported compliance Table 2. Participants from group practices had the highest patient volumes but the lowest rate of dyspepsia visits and compliance. Academic physicians saw the least number of patients but had the highest reported rate of dyspepsia visits and compliance.

Significant negative rank correlations were found to exist between patient volume and reported rate of dyspepsia visits (Figure 1: rs = -0.548; P .05) and between patient volume and compliance with protocol (Figure 2: rs = -0.490; P .05). A significant positive rank correlation was found between compliance with protocol and rate of dyspepsia visits (Figure 3 (: rs = 0.551; P .05). No significant correlation existed between the number of weeks of participation and patient volume (rs = -0.303), rate of dyspepsia visits (rs = 0.065), or compliance with protocol (rs = 0.415).

Discussion

Practice volume can have a significant effect on physicians’ reporting rates in practice-based studies. The rate of dyspepsia visits, as measured by the identification of patients meeting study criteria and having a completed data form, was negatively related to the number of patients seen per week by the physician. Practice volume appears to be linked to reporting by way of compliance. As an extension, it appears that physicians are generally accurate in self-assessment of their compliance with a protocol.

Although previous evaluations of PBRNs have demonstrated high levels of accuracy within reported data,¹⁰ the results reported here are somewhat disturbing. If other studies show similar results, the idea that PBRNs can assess prevalence of medical conditions could be called into question. Also, there may be a bias in the higher? volume practices for patients with more severe symptoms to be reported in preference to those with less “attention getting” symptoms, or in low-volume practices to seek out problems for which the patient did not seek attention. Consequently, even when a medical problem is identified, there may be patient selection bias toward those with more or less severe symptoms.

Additional burden and lack of practice support were common reasons for withdrawing from participation in PBRNs.¹¹ Overall participation and compliance with a research protocol, therefore, is likely related to the complexity of that protocol. While the reported rate of dyspepsia visits was negatively related to practice volume, the simple reporting of a weekly tally of patients seen in clinic was not. Consequently, compliance-sensitive measurements (eg, prevalence) may need simple time-efficient protocols. For example, full compliance with the protocol for the approximately 1050 physicians currently involved in the Centers for Disease Control and Prevention US Influenza Sentinel Physician Surveillance Network requires less than 3 minutes per week. This surveillance network for monitoring prevalence of influenza-like illness is a highly accurate, timely, and valued component of influenza surveillance.¹² Other enhancements for study protocols may include decreased periods for data gathering, use of intermittent reporting, and use of other office staff for case identification.

Limitations

This study is limited by a potential lack of generalizability. It is an observational study of physician behavior around a complex and relatively high-burden data collection instrument. There were no true standards regarding prevalence of dyspepsia at any location, thus allowing for the possibility that patient populations differed significantly among sites. Self-reported compliance with the research protocol was based on recall 4 months after the end of the data collection period. Also, some of the effect attributable to patient volume could alternatively result from the types of physicians involved in this study.

Academic physicians, with low practice volumes, may be more likely to be compliant with research protocols in general, regardless of their practice volumes. Because of the small sample size, however, this alternate hypothesis cannot be examined independently. With the exclusion of the academic physicians, relationships between the variables demonstrated the same trends, but the Spearman rank correlations were no longer significant (n = 14; patient volume vs rate: rs = -0.345; patient volume vs compliance: rs = -0.187; compliance vs rate: rs = 0.379).

This study does, however, challenge other investigators using PBRNs to revisit suitable data to determine similar patterns. Also, a simple assessment of participant compliance might prove to be an essential enhancement of future practice-based research.

Conclusions

Even encumbered with potential methodologic dilemmas, practice-based research studies may be the only way to approach many common medical issues in the context of the communities in which they occur.^1-3 For example, while selection bias in reporting of dyspepsia is clearly a problem in this example, the selection bias is still far less severe than it would be in the gastrointestinal specialty clinic of a referral center. Likewise, if nonreferred conditions are to be tracked over extensive periods of time, the use of community settings is essential, as was done with a recent longitudinal study of depression.¹³

Acknowledgments

Funding for this study was provided through a grant from the American Academy of Family Physicians. We thank the following participants of the WReN Practice-Based Research Group: R. Baldwin, E. Barr, D. Baumgardner, A. Berlage, M. Chin, D. Erickson, R. Erickson, G. Gay, M. Grajewski, D. Hahn, T. Hankey, D. Madlon-Kay, A. Marquis, E. Ott, D. Pine, and L. Radant.

Research in primary care is a growing field; the need for research on new diagnostic and therapeutic methods, the prognostic value of clinical signs and symptoms, and the effectiveness of clinical strategies in the population where these clinical contributions will be applied is well recognized.^1-5 The number and scale of research projects in primary care is expanding continuously, and research networks are being established.^6-8 This development, however, is putting increasing pressure on family physicians (FP) to actively participate in research. From an FP’s perspective there is a delicate balance between active research participation and efficient clinical practice.⁹ Proper planning to minimize paperwork and the delegation of research logistics to practice assistants can avoid many practical obstacles.^7,10,11

Motivating and recruiting FPs to participate is an essential step in conducting primary care research. Factors such as the clinical relevance of the research subject, personal interest in the topic, ownership, personal contact with the research group, and the required time investment have been shown to influence the participation of FPs.^5,11-15

Earlier studies have shown that higher-qualified physicians (with more continuing medical education training or a degree in research), physicians involved in under-or postgraduate training, FPs with research experience, and physicians working in well-organized practices (with more practice assistants and management protocols) are more interested in participating in research. Sex, the number of physicians per practice, and wages do not influence participation.¹⁵ The actual impact of financial incentives, however, is unclear. One study reported that incentives raised participation,¹⁶ while others could not confirm this.^12,15 Some have also drawn attention to the ethical and methodologic aspects of payment for research.^5,17

After successfully contracting with FPs to recruit patients for the study, the next task for a research group is to attain the maximal recruitment of patients. Unfortunately, less than half of the participating physicians actually recruit patients in the research project,^9,12,18 and often only a minority (20%)¹⁸ of the eligible patients are actually included. Our objective in this study was to identify practice and physician characteristics determining successful patient recruitment among FPs participating in a combined randomized clinical trial/cohort study on drug treatment of dyspepsia in the Netherlands.

Methods

Setting

Our data were compiled from a practice-based research network (PBRN) used for a primary care study of dyspepsia, the CIRANO study (Cisapride or Ranitidine in NonOrganic dyspepsia), which was conducted from 1996-1998 in the Netherlands. The CIRANO project consisted of 2 parts: a cohort study, in which dyspeptic patients were included and followed up for 1 year, and a randomized clinical trial in which patients selected from the cohort were treated with either an H2 blocker or a prokinetic drug.¹⁹ It was designed by the dyspepsia group of the Julius Centre, a group of primary care researchers who have been involved in gastrointestinal research and guideline development in the Netherlands for a number of years. Janssen Cilag Inc. was the sole sponsor of the project. The protocol was approved by the academic ethical committee, and data monitoring was done in accordance with the GCP guidelines (good clinical practice, a government protocol for conducting drug trials).

In the cohort study the workload for the participating physicians was small. After identification and inclusion by the FP, patients had to complete a validated dyspepsia symptom score, a quality of life questionnaire (COOP/WONCA chart), and a mental health state check list (GHQ 12). Also, the practice assistant performed a Helicobacter pylori whole- blood test. Follow-up of the patient was done by the research group. The FP workload in the clinical trial was heavier. After inclusion, the patients were randomized to one of the treatment arms, treated for 4 weeks, and followed up for 3 months. Patients were seen by the FP at inclusion and after 1 and 3 months.

Monitoring, data recording, and verification were performed by a clinical research organization under supervision of the research group. Data were stored and analyzed by the group.

FPs were given a financial incentive for each patient they included. It comprised a reimbursement for the extra practice time spent completing the research protocol. The estimated overall time investment for FPs was 2 hours with an additional 5 minutes for each patient included in the cohort and 1 additional hour for each patient included in the randomized clinical trial. Since the workload differed significantly, the reimbursement was higher for the patients included in the clinical trial ($25 per patient in the cohort, $70 for each patient in the randomized clinical trial). Also, project bulletins were distributed regularly during the course of the project.

All FPs in the academic network of the Utrecht University (2000 FPs, one third of all Dutch FPs) were invited by the academic research group to participate in the CIRANO study. Two hundred of them expressed interest in the study and asked for documentation. A total of 165 FPs finally signed the research contract.

Questionnaire

Five months after the CIRANO project was completed, all the participating physicians were sent an anonymous questionnaire containing 4 sections: (1) demographic and practice data, (2) initial motivation to participate, (3) evaluation of the logistics of the project, and (4) motivation to participate in future projects. The evaluation questions were Likert-type (a scale of 4 answer categories); motivation was analyzed by asking respondents to indicate the 3 most important reasons from a list of 8 Table 1. A reminder was sent 1 month after the first mailing. Data from the questionnaire, as well as the number of patients included in the study, were entered in SPSS for Windows software version 8.5 for analysis (SPSS, Inc; Chicago, Ill). Questionnaires that were not fully completed were excluded.

Analysis

To calculate odds ratios the number of recruited patients per FP was dichotomized, with cut-off points at the 25th percentile of inclusion: 0 to 4 versus 5 or more patients for inclusion in the cohort study, and 0 or 1 patient versus 2 or more for inclusion in the clinical trial. The association between demographic data, motivating factors and the number of recruited patients was calculated and expressed as an odds ratio (with 95% confidence interval [CI]). Factors that were thus associated with recruitment at a P level of less than 0.25 (so as not to exclude potentially important variables), together with 7 factors known from the literature to be relevant (sex, list size, years in practice, practice location, research experience, high specialization, and financial incentive-driven motivation) were entered in a logistic regression model.

Using a stepwise-backward procedure, determinants of maximal inclusion were analyzed and reported as adjusted odds ratio (with 95% CI). As the workload and the financial incentive for the clinical trial and cohort study differed significantly, we analyzed inclusion in the 2 parts of the research project separately.

Results

Of the 165 participating FPs, 132 returned the questionnaire; the response rate was 80%. Since 4 of the questionnaires were incomplete, data from 128 family physicians could be used for analysis. Most responders were men (87%), and half had been in practice for more than 5 years, primarily in semi-urban areas of the Netherlands (68%). Most (77%) were involved in other professional activities, such as vocational training, continuing medical education (CME), or activities of the College of Family Physicians (CFP) at the district level. Half the responders worked in a group practice, and more than 60% of the practices were “specialized” (defined as performing at least 4 of the following clinical activities in routine daily practice: minor surgery, Doppler studies, electrocardiography, cervical screening, intrauterine device insertion, diabetes protocol, or spirometry). Also, 57% of the participants had previous research experience.

The initial motivation for participation in the project varied Table 1. For the majority of the participants, the research topic and the participation of our academic research group were the most important factors. One third of the respondents considered participation a professional obligation, were attracted by the personal appeal by the research group, or were intrigued by the presentation of the project. The involvement of the sponsor and the financial incentive were important for only a minority in their decision to participate.

In general, the project was well evaluated; 80% of the respondents stated that the project had met their expectations (56% fully, 24% partially), and 60% noted that they would consider participation in a similar research project in the future. More specifically, the participants were satisfied with the quality of the correspondence, the newsletter, and the monitoring of the project Table 2. However, 47% of the participating FPs thought that the overall time investment in the project was too burdensome, and one third mentioned the negative impact the application of the GCP guidelines had on the workload of the project.

From September 1996 until January 1998 these 128 physicians recruited 793 patients in the cohort phase of the study (average = 6.3 per FP; standard deviation [SD]=6.6) and 527 in the clinical trial (average = 4.2 per FP; SD=4.9). A total of 15% of the FPs recruited no patients in the cohort study, while 59% recruited 4 or more patients. In comparison, 21% of the FPs did not recruit any patients in the clinical trial, and 65% recruited 2 or more patients.

In the univariate analysis only the factors “active in CME/CFP” and “motivation by the academic research group” were associated with the number of recruited patients. These associations occurred in both the cohort study and the randomized clinical trial Table 3. After entering these 2 factors were entered in a logistic model, together with 7 factors earlier reported relevant in the literature, multivariate analysis indicated that only the factor “motivation by the participation of the academic research group” predicted the number of patients recruited in cohort study (adjusted odds ratio [OR] = 3.5; 95% CI, 1.4-9.0) and clinical trial (adjusted OR = 2.9; 95% CI, 1.2-6.9).

Discussion

The combination of participation in research and daily clinical practice requires a major time investment. Even though FPs consider their participation thoroughly before a research project commences, the actual numbers of patients they recruit are often disappointing.

In this large randomized trial on dyspepsia in primary care we showed that those FPs whose motivation was driven by the participation of an academic department of family medicine in the research group recruited the most patients. They were 3 times more likely than their colleagues to recruit at least 2 patients for the clinical trial or 5 patients for the cohort. Other factors such as list (practice) size, involvement in professional CME/CFP activities, research experience, and financial incentive may have played a role in the FPs’ decision to participate in the research project but were not associated with the actual number of patients they recruited.

As in most projects, a minority of the participating physicians did not manage to recruit any patients. These colleagues either underestimated the time investment required, had second thoughts about the acceptability for their patients, or were disappointed by the planning and paperwork of the project.

Factors determining patient recruitment in primary care research have hardly been studied. Busy schedules, forgetfulness, poor patient compliance, and FP involvement in too many projects are a few of the reasons given to explain poor recruitment.^18,9

One third of all Dutch FPs were invited to participate in the dyspepsia study. Even though bias might have been caused by either the subject (dyspepsia physicians), the fact that half of the FPs had experience in research, or the fact that the participants were generally very active in numerous professional activities (active physicians), we think the results of our study can be generalized to the primary care setting in the Netherlands. Our conclusions may, however, require modification in other countries because of differences in practice organization and research climate in primary care.

Although there were various reasons for participation in the CIRANO study, they are consistent with earlier reports. FPs who participated in our dyspepsia project were mainly motivated by the subject and by the fact that the project was affiliated with our academic primary care research group. The motivation was not a matter of personal acquaintance, since most of the participants were not known to the members of the research group.

A substantial number of the participating colleagues also felt that participation was a professional obligation. This perception might have been induced by the fact that during the introduction of the project, special emphasis was put on the evidence missing from certain paragraphs of the Dutch guidelines on dyspepsia and on the need for primary care-based research to fill this gap. Although the research group felt that this was an important aspect of motivation, the evaluation showed that while it was an important reason to participate, it was not independently associated with patient recruitment. Only 10% of the participants stated that the financial incentive was a major reason to participate. Although this could be an unrealistic subjective statement, multivariate analysis confirmed that incentive-driven motivation was not related to the number of patients recruited. The fact that the results were the same for both the cohort study and the clinical trial might also be an indication that the amount of the incentive played a minor role in patient recruitment. This confirms earlier reports^12,13 that FPs probably do not participate in research for the money, although they do want a proper reimbursement for the time invested.

The fact that the majority of our study group were involved in professional education or organization confirms once again¹⁵ that active colleagues are the ones most motivated for research. Interest in research, however, obviously does not guarantee successful inclusion. Although a high level of practice organization and a high specialization in clinical activities have also reportedly been associated with optimal recruitment, we could not confirm this with our data.

Conclusions

Collaborators for primary care research projects should primarily be sought among the colleagues who are already active in different professional fields, and who have a strong affiliation with academic research. Successful participation is mainly determined by the initial motivation of the FP: Those who are motivated by the presence of an academic research group in the study recruit best. The research topic, the amount of the financial incentive, research experience, and other factors often suggested to influence patient recruitment are probably less important.

Research in primary care is a growing field; the need for research on new diagnostic and therapeutic methods, the prognostic value of clinical signs and symptoms, and the effectiveness of clinical strategies in the population where these clinical contributions will be applied is well recognized.^1-5 The number and scale of research projects in primary care is expanding continuously, and research networks are being established.^6-8 This development, however, is putting increasing pressure on family physicians (FP) to actively participate in research. From an FP’s perspective there is a delicate balance between active research participation and efficient clinical practice.⁹ Proper planning to minimize paperwork and the delegation of research logistics to practice assistants can avoid many practical obstacles.^7,10,11

Motivating and recruiting FPs to participate is an essential step in conducting primary care research. Factors such as the clinical relevance of the research subject, personal interest in the topic, ownership, personal contact with the research group, and the required time investment have been shown to influence the participation of FPs.^5,11-15

Earlier studies have shown that higher-qualified physicians (with more continuing medical education training or a degree in research), physicians involved in under-or postgraduate training, FPs with research experience, and physicians working in well-organized practices (with more practice assistants and management protocols) are more interested in participating in research. Sex, the number of physicians per practice, and wages do not influence participation.¹⁵ The actual impact of financial incentives, however, is unclear. One study reported that incentives raised participation,¹⁶ while others could not confirm this.^12,15 Some have also drawn attention to the ethical and methodologic aspects of payment for research.^5,17

After successfully contracting with FPs to recruit patients for the study, the next task for a research group is to attain the maximal recruitment of patients. Unfortunately, less than half of the participating physicians actually recruit patients in the research project,^9,12,18 and often only a minority (20%)¹⁸ of the eligible patients are actually included. Our objective in this study was to identify practice and physician characteristics determining successful patient recruitment among FPs participating in a combined randomized clinical trial/cohort study on drug treatment of dyspepsia in the Netherlands.

Methods

Setting

Our data were compiled from a practice-based research network (PBRN) used for a primary care study of dyspepsia, the CIRANO study (Cisapride or Ranitidine in NonOrganic dyspepsia), which was conducted from 1996-1998 in the Netherlands. The CIRANO project consisted of 2 parts: a cohort study, in which dyspeptic patients were included and followed up for 1 year, and a randomized clinical trial in which patients selected from the cohort were treated with either an H2 blocker or a prokinetic drug.¹⁹ It was designed by the dyspepsia group of the Julius Centre, a group of primary care researchers who have been involved in gastrointestinal research and guideline development in the Netherlands for a number of years. Janssen Cilag Inc. was the sole sponsor of the project. The protocol was approved by the academic ethical committee, and data monitoring was done in accordance with the GCP guidelines (good clinical practice, a government protocol for conducting drug trials).

In the cohort study the workload for the participating physicians was small. After identification and inclusion by the FP, patients had to complete a validated dyspepsia symptom score, a quality of life questionnaire (COOP/WONCA chart), and a mental health state check list (GHQ 12). Also, the practice assistant performed a Helicobacter pylori whole- blood test. Follow-up of the patient was done by the research group. The FP workload in the clinical trial was heavier. After inclusion, the patients were randomized to one of the treatment arms, treated for 4 weeks, and followed up for 3 months. Patients were seen by the FP at inclusion and after 1 and 3 months.

Monitoring, data recording, and verification were performed by a clinical research organization under supervision of the research group. Data were stored and analyzed by the group.

FPs were given a financial incentive for each patient they included. It comprised a reimbursement for the extra practice time spent completing the research protocol. The estimated overall time investment for FPs was 2 hours with an additional 5 minutes for each patient included in the cohort and 1 additional hour for each patient included in the randomized clinical trial. Since the workload differed significantly, the reimbursement was higher for the patients included in the clinical trial ($25 per patient in the cohort, $70 for each patient in the randomized clinical trial). Also, project bulletins were distributed regularly during the course of the project.

All FPs in the academic network of the Utrecht University (2000 FPs, one third of all Dutch FPs) were invited by the academic research group to participate in the CIRANO study. Two hundred of them expressed interest in the study and asked for documentation. A total of 165 FPs finally signed the research contract.

Questionnaire

Five months after the CIRANO project was completed, all the participating physicians were sent an anonymous questionnaire containing 4 sections: (1) demographic and practice data, (2) initial motivation to participate, (3) evaluation of the logistics of the project, and (4) motivation to participate in future projects. The evaluation questions were Likert-type (a scale of 4 answer categories); motivation was analyzed by asking respondents to indicate the 3 most important reasons from a list of 8 Table 1. A reminder was sent 1 month after the first mailing. Data from the questionnaire, as well as the number of patients included in the study, were entered in SPSS for Windows software version 8.5 for analysis (SPSS, Inc; Chicago, Ill). Questionnaires that were not fully completed were excluded.

Analysis

To calculate odds ratios the number of recruited patients per FP was dichotomized, with cut-off points at the 25th percentile of inclusion: 0 to 4 versus 5 or more patients for inclusion in the cohort study, and 0 or 1 patient versus 2 or more for inclusion in the clinical trial. The association between demographic data, motivating factors and the number of recruited patients was calculated and expressed as an odds ratio (with 95% confidence interval [CI]). Factors that were thus associated with recruitment at a P level of less than 0.25 (so as not to exclude potentially important variables), together with 7 factors known from the literature to be relevant (sex, list size, years in practice, practice location, research experience, high specialization, and financial incentive-driven motivation) were entered in a logistic regression model.

Using a stepwise-backward procedure, determinants of maximal inclusion were analyzed and reported as adjusted odds ratio (with 95% CI). As the workload and the financial incentive for the clinical trial and cohort study differed significantly, we analyzed inclusion in the 2 parts of the research project separately.

Results

Of the 165 participating FPs, 132 returned the questionnaire; the response rate was 80%. Since 4 of the questionnaires were incomplete, data from 128 family physicians could be used for analysis. Most responders were men (87%), and half had been in practice for more than 5 years, primarily in semi-urban areas of the Netherlands (68%). Most (77%) were involved in other professional activities, such as vocational training, continuing medical education (CME), or activities of the College of Family Physicians (CFP) at the district level. Half the responders worked in a group practice, and more than 60% of the practices were “specialized” (defined as performing at least 4 of the following clinical activities in routine daily practice: minor surgery, Doppler studies, electrocardiography, cervical screening, intrauterine device insertion, diabetes protocol, or spirometry). Also, 57% of the participants had previous research experience.

The initial motivation for participation in the project varied Table 1. For the majority of the participants, the research topic and the participation of our academic research group were the most important factors. One third of the respondents considered participation a professional obligation, were attracted by the personal appeal by the research group, or were intrigued by the presentation of the project. The involvement of the sponsor and the financial incentive were important for only a minority in their decision to participate.

In general, the project was well evaluated; 80% of the respondents stated that the project had met their expectations (56% fully, 24% partially), and 60% noted that they would consider participation in a similar research project in the future. More specifically, the participants were satisfied with the quality of the correspondence, the newsletter, and the monitoring of the project Table 2. However, 47% of the participating FPs thought that the overall time investment in the project was too burdensome, and one third mentioned the negative impact the application of the GCP guidelines had on the workload of the project.

From September 1996 until January 1998 these 128 physicians recruited 793 patients in the cohort phase of the study (average = 6.3 per FP; standard deviation [SD]=6.6) and 527 in the clinical trial (average = 4.2 per FP; SD=4.9). A total of 15% of the FPs recruited no patients in the cohort study, while 59% recruited 4 or more patients. In comparison, 21% of the FPs did not recruit any patients in the clinical trial, and 65% recruited 2 or more patients.

In the univariate analysis only the factors “active in CME/CFP” and “motivation by the academic research group” were associated with the number of recruited patients. These associations occurred in both the cohort study and the randomized clinical trial Table 3. After entering these 2 factors were entered in a logistic model, together with 7 factors earlier reported relevant in the literature, multivariate analysis indicated that only the factor “motivation by the participation of the academic research group” predicted the number of patients recruited in cohort study (adjusted odds ratio [OR] = 3.5; 95% CI, 1.4-9.0) and clinical trial (adjusted OR = 2.9; 95% CI, 1.2-6.9).

Discussion

The combination of participation in research and daily clinical practice requires a major time investment. Even though FPs consider their participation thoroughly before a research project commences, the actual numbers of patients they recruit are often disappointing.

In this large randomized trial on dyspepsia in primary care we showed that those FPs whose motivation was driven by the participation of an academic department of family medicine in the research group recruited the most patients. They were 3 times more likely than their colleagues to recruit at least 2 patients for the clinical trial or 5 patients for the cohort. Other factors such as list (practice) size, involvement in professional CME/CFP activities, research experience, and financial incentive may have played a role in the FPs’ decision to participate in the research project but were not associated with the actual number of patients they recruited.

As in most projects, a minority of the participating physicians did not manage to recruit any patients. These colleagues either underestimated the time investment required, had second thoughts about the acceptability for their patients, or were disappointed by the planning and paperwork of the project.

Factors determining patient recruitment in primary care research have hardly been studied. Busy schedules, forgetfulness, poor patient compliance, and FP involvement in too many projects are a few of the reasons given to explain poor recruitment.^18,9

One third of all Dutch FPs were invited to participate in the dyspepsia study. Even though bias might have been caused by either the subject (dyspepsia physicians), the fact that half of the FPs had experience in research, or the fact that the participants were generally very active in numerous professional activities (active physicians), we think the results of our study can be generalized to the primary care setting in the Netherlands. Our conclusions may, however, require modification in other countries because of differences in practice organization and research climate in primary care.

Although there were various reasons for participation in the CIRANO study, they are consistent with earlier reports. FPs who participated in our dyspepsia project were mainly motivated by the subject and by the fact that the project was affiliated with our academic primary care research group. The motivation was not a matter of personal acquaintance, since most of the participants were not known to the members of the research group.

A substantial number of the participating colleagues also felt that participation was a professional obligation. This perception might have been induced by the fact that during the introduction of the project, special emphasis was put on the evidence missing from certain paragraphs of the Dutch guidelines on dyspepsia and on the need for primary care-based research to fill this gap. Although the research group felt that this was an important aspect of motivation, the evaluation showed that while it was an important reason to participate, it was not independently associated with patient recruitment. Only 10% of the participants stated that the financial incentive was a major reason to participate. Although this could be an unrealistic subjective statement, multivariate analysis confirmed that incentive-driven motivation was not related to the number of patients recruited. The fact that the results were the same for both the cohort study and the clinical trial might also be an indication that the amount of the incentive played a minor role in patient recruitment. This confirms earlier reports^12,13 that FPs probably do not participate in research for the money, although they do want a proper reimbursement for the time invested.

The fact that the majority of our study group were involved in professional education or organization confirms once again¹⁵ that active colleagues are the ones most motivated for research. Interest in research, however, obviously does not guarantee successful inclusion. Although a high level of practice organization and a high specialization in clinical activities have also reportedly been associated with optimal recruitment, we could not confirm this with our data.

Conclusions

Collaborators for primary care research projects should primarily be sought among the colleagues who are already active in different professional fields, and who have a strong affiliation with academic research. Successful participation is mainly determined by the initial motivation of the FP: Those who are motivated by the presence of an academic research group in the study recruit best. The research topic, the amount of the financial incentive, research experience, and other factors often suggested to influence patient recruitment are probably less important.

Research in primary care is a growing field; the need for research on new diagnostic and therapeutic methods, the prognostic value of clinical signs and symptoms, and the effectiveness of clinical strategies in the population where these clinical contributions will be applied is well recognized.^1-5 The number and scale of research projects in primary care is expanding continuously, and research networks are being established.^6-8 This development, however, is putting increasing pressure on family physicians (FP) to actively participate in research. From an FP’s perspective there is a delicate balance between active research participation and efficient clinical practice.⁹ Proper planning to minimize paperwork and the delegation of research logistics to practice assistants can avoid many practical obstacles.^7,10,11

Motivating and recruiting FPs to participate is an essential step in conducting primary care research. Factors such as the clinical relevance of the research subject, personal interest in the topic, ownership, personal contact with the research group, and the required time investment have been shown to influence the participation of FPs.^5,11-15

Earlier studies have shown that higher-qualified physicians (with more continuing medical education training or a degree in research), physicians involved in under-or postgraduate training, FPs with research experience, and physicians working in well-organized practices (with more practice assistants and management protocols) are more interested in participating in research. Sex, the number of physicians per practice, and wages do not influence participation.¹⁵ The actual impact of financial incentives, however, is unclear. One study reported that incentives raised participation,¹⁶ while others could not confirm this.^12,15 Some have also drawn attention to the ethical and methodologic aspects of payment for research.^5,17

After successfully contracting with FPs to recruit patients for the study, the next task for a research group is to attain the maximal recruitment of patients. Unfortunately, less than half of the participating physicians actually recruit patients in the research project,^9,12,18 and often only a minority (20%)¹⁸ of the eligible patients are actually included. Our objective in this study was to identify practice and physician characteristics determining successful patient recruitment among FPs participating in a combined randomized clinical trial/cohort study on drug treatment of dyspepsia in the Netherlands.

Methods

Setting

Our data were compiled from a practice-based research network (PBRN) used for a primary care study of dyspepsia, the CIRANO study (Cisapride or Ranitidine in NonOrganic dyspepsia), which was conducted from 1996-1998 in the Netherlands. The CIRANO project consisted of 2 parts: a cohort study, in which dyspeptic patients were included and followed up for 1 year, and a randomized clinical trial in which patients selected from the cohort were treated with either an H2 blocker or a prokinetic drug.¹⁹ It was designed by the dyspepsia group of the Julius Centre, a group of primary care researchers who have been involved in gastrointestinal research and guideline development in the Netherlands for a number of years. Janssen Cilag Inc. was the sole sponsor of the project. The protocol was approved by the academic ethical committee, and data monitoring was done in accordance with the GCP guidelines (good clinical practice, a government protocol for conducting drug trials).

In the cohort study the workload for the participating physicians was small. After identification and inclusion by the FP, patients had to complete a validated dyspepsia symptom score, a quality of life questionnaire (COOP/WONCA chart), and a mental health state check list (GHQ 12). Also, the practice assistant performed a Helicobacter pylori whole- blood test. Follow-up of the patient was done by the research group. The FP workload in the clinical trial was heavier. After inclusion, the patients were randomized to one of the treatment arms, treated for 4 weeks, and followed up for 3 months. Patients were seen by the FP at inclusion and after 1 and 3 months.

Monitoring, data recording, and verification were performed by a clinical research organization under supervision of the research group. Data were stored and analyzed by the group.

FPs were given a financial incentive for each patient they included. It comprised a reimbursement for the extra practice time spent completing the research protocol. The estimated overall time investment for FPs was 2 hours with an additional 5 minutes for each patient included in the cohort and 1 additional hour for each patient included in the randomized clinical trial. Since the workload differed significantly, the reimbursement was higher for the patients included in the clinical trial ($25 per patient in the cohort, $70 for each patient in the randomized clinical trial). Also, project bulletins were distributed regularly during the course of the project.

All FPs in the academic network of the Utrecht University (2000 FPs, one third of all Dutch FPs) were invited by the academic research group to participate in the CIRANO study. Two hundred of them expressed interest in the study and asked for documentation. A total of 165 FPs finally signed the research contract.

Questionnaire

Five months after the CIRANO project was completed, all the participating physicians were sent an anonymous questionnaire containing 4 sections: (1) demographic and practice data, (2) initial motivation to participate, (3) evaluation of the logistics of the project, and (4) motivation to participate in future projects. The evaluation questions were Likert-type (a scale of 4 answer categories); motivation was analyzed by asking respondents to indicate the 3 most important reasons from a list of 8 Table 1. A reminder was sent 1 month after the first mailing. Data from the questionnaire, as well as the number of patients included in the study, were entered in SPSS for Windows software version 8.5 for analysis (SPSS, Inc; Chicago, Ill). Questionnaires that were not fully completed were excluded.

Analysis

To calculate odds ratios the number of recruited patients per FP was dichotomized, with cut-off points at the 25th percentile of inclusion: 0 to 4 versus 5 or more patients for inclusion in the cohort study, and 0 or 1 patient versus 2 or more for inclusion in the clinical trial. The association between demographic data, motivating factors and the number of recruited patients was calculated and expressed as an odds ratio (with 95% confidence interval [CI]). Factors that were thus associated with recruitment at a P level of less than 0.25 (so as not to exclude potentially important variables), together with 7 factors known from the literature to be relevant (sex, list size, years in practice, practice location, research experience, high specialization, and financial incentive-driven motivation) were entered in a logistic regression model.

Using a stepwise-backward procedure, determinants of maximal inclusion were analyzed and reported as adjusted odds ratio (with 95% CI). As the workload and the financial incentive for the clinical trial and cohort study differed significantly, we analyzed inclusion in the 2 parts of the research project separately.

Results

Of the 165 participating FPs, 132 returned the questionnaire; the response rate was 80%. Since 4 of the questionnaires were incomplete, data from 128 family physicians could be used for analysis. Most responders were men (87%), and half had been in practice for more than 5 years, primarily in semi-urban areas of the Netherlands (68%). Most (77%) were involved in other professional activities, such as vocational training, continuing medical education (CME), or activities of the College of Family Physicians (CFP) at the district level. Half the responders worked in a group practice, and more than 60% of the practices were “specialized” (defined as performing at least 4 of the following clinical activities in routine daily practice: minor surgery, Doppler studies, electrocardiography, cervical screening, intrauterine device insertion, diabetes protocol, or spirometry). Also, 57% of the participants had previous research experience.

The initial motivation for participation in the project varied Table 1. For the majority of the participants, the research topic and the participation of our academic research group were the most important factors. One third of the respondents considered participation a professional obligation, were attracted by the personal appeal by the research group, or were intrigued by the presentation of the project. The involvement of the sponsor and the financial incentive were important for only a minority in their decision to participate.

In general, the project was well evaluated; 80% of the respondents stated that the project had met their expectations (56% fully, 24% partially), and 60% noted that they would consider participation in a similar research project in the future. More specifically, the participants were satisfied with the quality of the correspondence, the newsletter, and the monitoring of the project Table 2. However, 47% of the participating FPs thought that the overall time investment in the project was too burdensome, and one third mentioned the negative impact the application of the GCP guidelines had on the workload of the project.

From September 1996 until January 1998 these 128 physicians recruited 793 patients in the cohort phase of the study (average = 6.3 per FP; standard deviation [SD]=6.6) and 527 in the clinical trial (average = 4.2 per FP; SD=4.9). A total of 15% of the FPs recruited no patients in the cohort study, while 59% recruited 4 or more patients. In comparison, 21% of the FPs did not recruit any patients in the clinical trial, and 65% recruited 2 or more patients.

In the univariate analysis only the factors “active in CME/CFP” and “motivation by the academic research group” were associated with the number of recruited patients. These associations occurred in both the cohort study and the randomized clinical trial Table 3. After entering these 2 factors were entered in a logistic model, together with 7 factors earlier reported relevant in the literature, multivariate analysis indicated that only the factor “motivation by the participation of the academic research group” predicted the number of patients recruited in cohort study (adjusted odds ratio [OR] = 3.5; 95% CI, 1.4-9.0) and clinical trial (adjusted OR = 2.9; 95% CI, 1.2-6.9).

Discussion

The combination of participation in research and daily clinical practice requires a major time investment. Even though FPs consider their participation thoroughly before a research project commences, the actual numbers of patients they recruit are often disappointing.

In this large randomized trial on dyspepsia in primary care we showed that those FPs whose motivation was driven by the participation of an academic department of family medicine in the research group recruited the most patients. They were 3 times more likely than their colleagues to recruit at least 2 patients for the clinical trial or 5 patients for the cohort. Other factors such as list (practice) size, involvement in professional CME/CFP activities, research experience, and financial incentive may have played a role in the FPs’ decision to participate in the research project but were not associated with the actual number of patients they recruited.

As in most projects, a minority of the participating physicians did not manage to recruit any patients. These colleagues either underestimated the time investment required, had second thoughts about the acceptability for their patients, or were disappointed by the planning and paperwork of the project.

Factors determining patient recruitment in primary care research have hardly been studied. Busy schedules, forgetfulness, poor patient compliance, and FP involvement in too many projects are a few of the reasons given to explain poor recruitment.^18,9

One third of all Dutch FPs were invited to participate in the dyspepsia study. Even though bias might have been caused by either the subject (dyspepsia physicians), the fact that half of the FPs had experience in research, or the fact that the participants were generally very active in numerous professional activities (active physicians), we think the results of our study can be generalized to the primary care setting in the Netherlands. Our conclusions may, however, require modification in other countries because of differences in practice organization and research climate in primary care.

Although there were various reasons for participation in the CIRANO study, they are consistent with earlier reports. FPs who participated in our dyspepsia project were mainly motivated by the subject and by the fact that the project was affiliated with our academic primary care research group. The motivation was not a matter of personal acquaintance, since most of the participants were not known to the members of the research group.

A substantial number of the participating colleagues also felt that participation was a professional obligation. This perception might have been induced by the fact that during the introduction of the project, special emphasis was put on the evidence missing from certain paragraphs of the Dutch guidelines on dyspepsia and on the need for primary care-based research to fill this gap. Although the research group felt that this was an important aspect of motivation, the evaluation showed that while it was an important reason to participate, it was not independently associated with patient recruitment. Only 10% of the participants stated that the financial incentive was a major reason to participate. Although this could be an unrealistic subjective statement, multivariate analysis confirmed that incentive-driven motivation was not related to the number of patients recruited. The fact that the results were the same for both the cohort study and the clinical trial might also be an indication that the amount of the incentive played a minor role in patient recruitment. This confirms earlier reports^12,13 that FPs probably do not participate in research for the money, although they do want a proper reimbursement for the time invested.

The fact that the majority of our study group were involved in professional education or organization confirms once again¹⁵ that active colleagues are the ones most motivated for research. Interest in research, however, obviously does not guarantee successful inclusion. Although a high level of practice organization and a high specialization in clinical activities have also reportedly been associated with optimal recruitment, we could not confirm this with our data.

Conclusions

Collaborators for primary care research projects should primarily be sought among the colleagues who are already active in different professional fields, and who have a strong affiliation with academic research. Successful participation is mainly determined by the initial motivation of the FP: Those who are motivated by the presence of an academic research group in the study recruit best. The research topic, the amount of the financial incentive, research experience, and other factors often suggested to influence patient recruitment are probably less important.

Clinicians, researchers, and policymakers now recognize that multiple competing demands1 and opportunities² are simultaneously affecting the physicians, staff, and patients within primary care practices. Our current understanding of outpatient practice is largely based on administrative databases, national surveys, and medical record reviews, with additional insights from surveys of patients or clinicians. These data generally are not designed to capture the richness of the content and context that is needed to better understand the realities and complexities of practice.^3-6 The underlying premise of The Prevention and Competing Demands in Primary Care (P&CD) Study is that efforts to change practice should be preceded by efforts to understand it.^2,7 The explicit goal of this study is to understand practice structure and process, including details of patients, physicians, staff, and clinical encounters; the practice as an organization; and its relationship to the larger community and health system.

In this paper we describe a dynamic observational multimethod assessment process (MAP) that can be used to understand the complex reality of primary care practice. MAP is based on a multimethod comparative case study design^8,9 that integrates elements of epidemiology with methods derived from the qualitative traditions of anthropology and sociology and relies most heavily on qualitative observation and interviewing methods. Studies of this type require an iterative data collection and analysis approach that evolves over time so that new methods can be introduced as the investigators gain a better understanding of important issues. A major strength of our study design was that it allowed hypotheses and insights gained from participants and from ongoing analyses to be integrated into the ongoing investigation.

The study’s primary research questions related to how practice characteristics affect preventive service delivery. Thus, the research design included: (1) an examination of the organizational contexts that support preventive services, (2) an examination of the competing demands imposed by carrying out clinical prevention and illness care in clinical encounters and in the practice, (3) a comparison of the approaches used by practices with high versus low intensity of preventive services delivered to eligible patients, and (4) an examination of approaches used to deliver different types of preventive services. Although the particular focus was on preventive services, the rich MAP allowed pursuit of other research topics that are presented in this issue of JFP.

This article describes the evolutionary methods of the P&CD study, focusing on how data were collected to ensure that sufficient details were available to understand a practice’s values, structures, and processes.*

Emergent research design

The P&CD study was conceived in 1994 to be an in-depth follow-up of insights from the Direct Observation of Primary Care (DOPC) Study that was just getting under way in northeastern Ohio.⁵ The DOPC Study provided a largely quantitative assessment of patients, physicians, encounters, and practices using patient questionnaires, physician surveys, medical record audits, and direct observation of clinical encounters using the Davis Observation Code.¹⁰ That study’s initial findings were presented in the May 1998 theme issue of JFP, and the study processes have recently been described.¹¹ Details of the DOPC methods have been published elsewhere.^4,5,12

Although the initial design allowed the DOPC research nurses to collect brief observational notes, the intensity of the quantitative data collection limited the scope of the study’s qualitative data for understanding details of the practice’s organization and the competing demands within clinical encounters. As a consequence, the P&CD study was designed to provide more in-depth description and understanding of the competing demands of family practice, and in particular, to evaluate factors affecting preventive services delivery using a comparative case study design and a MAP.

A key feature of the P&CD study design was an openness to the integration of emerging insights into the data collection protocol. For example, preliminary analyses of the DOPC data¹³ and other ongoing studies¹⁴ led to the discovery that complexity science was valuable for explaining the dynamics of office systems⁶ and needed to be incorporated into the design. (Complexity science is the study of systems that are characterized by nonlinear dynamics and emergent properties; it emphasizes the need to understand the interrelationships of the whole system and not just collect data about the parts.¹⁵) The investigators also developed new ways to display the relationships among physicians and staff in the practices using “practice genograms.”¹⁶ The practice genogram is a diagram of the functional and interpersonal relationships among the clinicians, support staff, and other people and organizations interacting with the practice. Throughout the project and consistent with the standards of qualitative research design,17 there were continued modifications and enhancements in the data collection and analysis strategies in response to insights that were emerging from ongoing analyses and interpretation of the data.

An important feature of the project was the development of an advisory committee of consultants and co-investigators that convened annually to provide multidisciplinary input, review results, and provide feedback. The advisory committee included academic representatives with expertise in nursing, health education, women’s health, minority health, and public policy. Two additional members were added to the project to provide expertise into the study of organizations as complex systems. The annual reviews by the advisory committee led to significant changes in the research design while the study was ongoing.

Practice Sample

Beginning in late 1996, we drew from 91 practices in Nebraska that had been randomly selected to participate in an earlier study on tobacco prevention and cessation.¹⁸ Initially a sample of 10 practices was purposefully selected^19,20 using an iterative process to represent a range in size (small and large), geographic location (urban, suburban, and rural), and rate of delivery of tobacco-related preventive services. Preliminary analyses of these 10 initial practices provided a summary of preventive health delivery strategies in primary care practices and a description of competing demands that enhanced or limited these strategies. To confirm or refute the emerging insights from the original 10 practices, 8 additional practices were selected for further data collection during the second and third years of the study. The sampling strategy in years 2 and 3 ensured that at least 2 practices each from several major regional hospital health systems were included and allowed us to assess emerging hypotheses about the importance of health system context for understanding community practices.

The practices were recruited by contacting one of the physicians to solicit participation; only those in which all family physicians in the practice agreed to participate were included in the study. Twenty-three practices were contacted; all physicians in 18 agreed to participate (78%).

Core Data Collection Methods

Data were collected by trained field researchers who spent 4 weeks or more taking notes at each practice while observing the practice and clinical encounters, conducting informal key informant interviews of staff, collecting office documents, and auditing charts of patients whose encounters were observed. Within each practice, data collection occurred in stages, with a short break after the initial week or 2 of observation to allow preliminary analyses to inform additional data collection.

Observations at the practice level were recorded in a combination of structured observational checklists, unstructured dictated field notes,^21-23 and key informant interviews.²⁴ Detailed floor plans of the practice were used to identify where particular activities occurred and where individual practice participants worked. Each day at the practice, the field researcher took short notes or “jottings” and dictated expanded field notes in the evening.²³ A template of topics was used periodically to ensure that important aspects of the practice were not being overlooked. The template included lists of features of the community, practice, staff, and patients that the researchers saw as important Figure W1.*¹³ A 3-page structured practice environment checklist was adapted from earlier work on the DOPC project¹¹ and included a wide range of practice characteristics and functions, including items such as the number and training of staff, counseling options offered, and management of telephone calls and referrals Figure W2a. This checklist also served as a detailed reminder to the research nurse of topics to be included in the field notes. Throughout the time the field researchers were recording field notes and filling in the checklist, they opportunistically asked clinicians or staff informal key informant questions for confirmation or clarification.

Consecutive patients for each clinician in a practice were approached with a goal of recruiting 30 patients who would consent to have the field researcher observe their visit. This generally required approaching 35 to 40 patients. Because some clinicians worked part-time or were not consistently in the practice, it was not always possible to observe 30 visits. After explaining the study and gaining signed informed consent from patients who agreed to participate, the field researchers observed the outpatient visit as unobtrusively as possible. A 1-page structured encounter checklist Figure W3 that was also modified from the DOPC study11 provided blanks for noting the reason for the visit, chief complaints, and final diagnoses, and for indicating whether any of approximately 100 preventive services were ordered or delivered. Space was provided at the bottom of the form for recording notes that were later used to dictate a re-creation of the encounter. At a later time, a chart audit was done on each observed patient’s medical record using a structure chart audit form Figure W4.

After the initial observational data were transcribed, a preliminary practice genogram¹⁶ was drawn, and an initial practice summary was written. The genogram of practice participants, roles, and relationships was initially diagrammed on a white board by a transdisciplinary research team by interviewing the field researcher about the current and past practice clinicians and staff and about the health system and community. The demographics of individuals were recorded, including age, sex, years with the practice, percentage of work effort, and job responsibilities. Additional details included functional and emotional relationships observed in the practice, such as who worked together and any obvious conflicts among members of the practice or with health system or community affiliations. This process enabled the investigators to identify areas of incomplete data so the field researchers could return to fill in missing details.

In addition to the key informant interviews done as part of the observation activities, more formal semistructured individual depth interviews were arranged with each clinician and many of the office staff.²⁵ These interviews consisted of a 30-minute to 1-hour narrative interview in which the respondent was asked open-ended questions designed to elicit in-depth responses Figure W5. Although the major focus of these interviews was on the delivery of preventive services, more general questions were included to understand perspectives on practice process. For example, respondents were asked: “Could you describe for me a typical day for you in this practice?” and “If you believed a change was needed regarding some specific delivery of a service within this office, could you describe the process you would go through to try to get it implemented?” These interviews were audiotaped and transcribed verbatim.

To supplement the observational and interview data, field researchers gathered existing documents and artifacts from the practice. Items such as blank charts and flow sheets, patient schedules, personnel lists, samples of patient education materials and handouts, mission and vision statements, and annual reports were collected and compiled in binders. In some practices, particularly those affiliated with hospital health systems, materials were also available from Web sites. All transcribed interviews and dictated field notes from the practice and encounter observations were imported into FolioViews 4.11 (NextPage, Provo, Utah), a text-base management software program that facilitates coding, searching, and retrieval of large computerized text files.²⁶

Emerging Design Decisions

Midway through data collection at the first practice, the advisory committee met to review the data and discuss any concerns. The advisory committee identified a number of emerging hypotheses related to complexity science concepts that were used to guide subsequent data collection and management. It was deemed particularly important to identify “attractors”—factors in the practice and in the larger environment that influenced the structure and function of the practice as an organization.⁶ For example, an attractor might be a particular burning interest of one of the physicians, an expectation of the local hospital systems, or a dominant demographic characteristic of the patients being served.¹³ An expanded systemic model of primary care Figure 1 was articulated that characterized 6 core areas for data collection: (1) patient perceptions and behavior, (2) physician perceptions and behavior, (3) encounter structures and processes, (4) practice structures and processes, (5) community characteristics, and (6) the larger health system. This model identified the need for additional data on the community context and patient experience. Checklists were revised and field researchers were asked to spend more time gathering data about the community. It was also apparent that accurate calculation of certain preventive service delivery rates would require patient input and a larger sample size. For example, a patient exit card

Figure W6 was developed to ascertain self-reported tobacco use status (for all patients) and use of obstetrics and gynecology services and history of hysterectomy (for women). These data were used to determine a patient’s eligibility for tobacco cessation counseling and Papanicolaou tests, respectively. Observation of 30 encounters with each clinician was done to increase the stability for calculating rates for common preventive service recommendations.

A larger issue emerged with the discovery that, after contributing data over the course of weeks or months, members of the practices desired feedback in a timely manner. Practice clinicians and staff were very interested in how they were doing and asked when they would be receiving a report of our results. They did not want to wait 3 years for the completion of the study. Although ongoing analyses were anticipated, these had primarily been designed to ensure completeness of data and to provide feedback to field researchers on areas where clarifications were needed. In response to the emergent desire for feedback, the team generated rapid-turnaround summary reports for each practice. A summary report template was designed to present the descriptive details of the practice, including the practice genogram and a summary of the strengths and weaknesses of the practice’s prevention approach. The final page in the report provided the practice with a series of questions or points for self-reflection that often included process questions, such as “How can this organization become a team?” or “How can this practice deliver preventive services more consistently?” These reports were shared interactively with practices at a debriefing meeting within 2 to 3 months of completing data collection at each practice.

The feedback meetings provided an important opportunity to check the validity of the researchers’ analyses by comparing them with the perspectives of the practice participants. In all the practices, the response was a strong overall validation of the research team’s interpretation of the practice and its structures and processes. During the feedback presentations, the practice physicians and staff consistently made comments like, “Wow, did you ever get us,” or “This is like looking in a mirror.” In a number of sessions, the participants mentioned that the report raised issues about which they were vaguely aware and that the findings were stimulating considerable self-reflection. In several practices, the physicians disclosed that they would be taking actions in the future to modify some of the deficiencies the reports uncovered.

The next modification came as the data were being collected simultaneously in the second and third practices. We realized that despite our efforts to be nondisruptive, participating in the project required extra effort on the part of the practice. Each practice therefore received partial compensation in the form of a $500 certificate for the purchase of books or equipment.

After completing data collection at several practices, it was apparent that patients’ perspectives were still under-represented and that this was limiting the understanding of the practice. To gain further insights into patients’ experiences, beginning with the sixth practice we adopted the patient path approach described by Pommerenke and Dietrich,²⁷ following patients from the time they walk into the practice until the time they leave, using a patient path form for recording activities at different stages of the visit Figure W7. Additional brief open-ended interviews were conducted in the waiting room or examination rooms while patients were waiting.

Although we had asked the research nurses to be more thorough in their descriptions of the community, data from the community and health system were still incomplete. This became even more pronounced when studying practices that were part of health systems. Therefore, beginning with the sixth practice, we included in-depth interviews with individuals from health systems (eg, regional managers and medical directors). A further refinement came with the use of community key informant trees, a systematic process of identifying and interviewing members of the local community surrounding the practice.^24,28 These interviews of patients, church leaders, and other individuals from the community began with the ninth practice.

Once all the modifications were incorporated, the final case study design provided data at each of the 6 levels as shown in the Table 1. Particularly detailed data were available at the clinician, encounter, and practice levels. For example, at the clinician level the data included perceptions of roles as ascertained through the in-depth interviews, as well as actual behaviors recorded in the encounter field notes and chart audits. Insights on the structures and process of the practice were obtained through unstructured observations of the practice, structured checklists, written documents, and interviews. Supporting data were collected on patients’ perspectives, the community, and the health system that provided contexts for the practice case studies.

Discussion

The complexity of primary care practices is best understood from multiple perspectives,²⁹ a principle that guided the initial selection of a multimethod comparative case study design for this investigation. The MAP that emerged from this comparative case study design has a number of strengths and weaknesses. A particular asset of the design was our ability to investigate specific phenomena within their context rather than isolated from it.²⁹ The design also encouraged the investigators to pursue emerging insights, thus informing multiple perspectives that might not have otherwise been considered, although this may be somewhat limited by the purposeful sampling strategy that focuses on maximizing information about a particular topic.

Limitations

A limitation for broader implementation of this research design is the intensity of data collection and analysis, which are difficult to accomplish without considerable resources and a research team with diverse skills. There might also be some concerns about the ways that the data collection process alters practice behavior; however, the prolonged observational time frame and the multiple data sources for “triangulation” are designed to limit any potential Hawthorne effect. That is, by collecting and systematically comparing data from multiple sources, including direct observation, different forms of interviews, and existing documents, the investigators were able to identify inconsistencies in patterns of behavior.^17,19

Although the data collection, analysis, and feedback process appeared to increase a practice’s self-reflection, our study limited the input of patient and practice participants in the design, analysis, and interpretation and thus does not approach the participatory research paradigm espoused by McCauley and colleagues.³⁰ Still, our study moved from being primarily observational descriptive research into a more collaborative and interventional project, in part at the request of the participants. This suggests a model and method for future research in the arena of health care process and outcome improvement with the practices as collaborators. The MAP characterized in this paper offers a means for simultaneously describing, understanding, and improving the richly complex and varied processes and outcomes of primary care. By more actively engaging the practices in the research process, the MAP also points toward a new more collaborative model of practice-based research.

Conclusions

The comprehensive data in the P&CD study provide a unique opportunity to understand and describe multiple perspectives from the clinician, patient, encounter, practice, community, and health system spheres. Each of the papers in this issue of JFP used some of these comprehensive data to study one or more of these spheres. For example, the encounter field notes were the primary source of data for exploring how “family” presents in encounters³¹Several of the authors used subsets of patients, including patients presenting with acute respiratory track infections,³³ smokers,³⁴ and frequent attenders.³⁵ These authors each supplemented the encounter field notes with data from the medical record reviews, medical record field notes, and patient exit cards. The complete data set including practice field notes, practice genograms, physician and staff interviews, office environment checklists, and encounter field notes were used to describe staff training, roles, and functions.³⁶ This is only a part of the research adventure available in this type of data. We hope many others will join in the excitement.

Acknowledgments

Our study was supported by a grant from the Agency for Healthcare Research and Quality (R01 HS08776) and a Research Center grant from the American Academy of Family Physicians. Drs Crabtree, Miller and Stange are associated with the Center for Research in Family Practice and Primary Care, Cleveland, New Brunswick, Allentown, and San Antonio. We are grateful to the physicians, staff, and patients from the 18 practices, without whose participation this study would not have been possible. We also wish to thank Connie Gibbs and Jen Rouse, who spent many hours in the practices collecting data, and Diane Dodendorf and Jason Lebsack, who spent countless hours coordinating transcription and data management activities, for their dedicated work. We are especially indebted to Mary McAndrews, who transcribed hundreds of taped interviews and dictated field notes. The ongoing analyses that ensured the quality and comprehensiveness of the data were made possible through the dedicated work of Helen McIlvain, PhD; Jeffrey Susman, MD; Virginia Aita, PhD; Kristine McVea, MD; Elisabeth Backer, MD; Paul Turner, PhD; and Louis Pol, PhD. Finally, we thank the members of the advisory committee: Valerie Gilchrist, MD; Paul Nutting, MD, MPH; Carlos Jaén, MD, PhD; Kurt Stange, MD, PhD; William Miller, MD, MA; Reuben McDaniel, PhD; and Ruth Anderson, RN, PhD.

Clinicians, researchers, and policymakers now recognize that multiple competing demands1 and opportunities² are simultaneously affecting the physicians, staff, and patients within primary care practices. Our current understanding of outpatient practice is largely based on administrative databases, national surveys, and medical record reviews, with additional insights from surveys of patients or clinicians. These data generally are not designed to capture the richness of the content and context that is needed to better understand the realities and complexities of practice.^3-6 The underlying premise of The Prevention and Competing Demands in Primary Care (P&CD) Study is that efforts to change practice should be preceded by efforts to understand it.^2,7 The explicit goal of this study is to understand practice structure and process, including details of patients, physicians, staff, and clinical encounters; the practice as an organization; and its relationship to the larger community and health system.

In this paper we describe a dynamic observational multimethod assessment process (MAP) that can be used to understand the complex reality of primary care practice. MAP is based on a multimethod comparative case study design^8,9 that integrates elements of epidemiology with methods derived from the qualitative traditions of anthropology and sociology and relies most heavily on qualitative observation and interviewing methods. Studies of this type require an iterative data collection and analysis approach that evolves over time so that new methods can be introduced as the investigators gain a better understanding of important issues. A major strength of our study design was that it allowed hypotheses and insights gained from participants and from ongoing analyses to be integrated into the ongoing investigation.

The study’s primary research questions related to how practice characteristics affect preventive service delivery. Thus, the research design included: (1) an examination of the organizational contexts that support preventive services, (2) an examination of the competing demands imposed by carrying out clinical prevention and illness care in clinical encounters and in the practice, (3) a comparison of the approaches used by practices with high versus low intensity of preventive services delivered to eligible patients, and (4) an examination of approaches used to deliver different types of preventive services. Although the particular focus was on preventive services, the rich MAP allowed pursuit of other research topics that are presented in this issue of JFP.

This article describes the evolutionary methods of the P&CD study, focusing on how data were collected to ensure that sufficient details were available to understand a practice’s values, structures, and processes.*

Emergent research design

The P&CD study was conceived in 1994 to be an in-depth follow-up of insights from the Direct Observation of Primary Care (DOPC) Study that was just getting under way in northeastern Ohio.⁵ The DOPC Study provided a largely quantitative assessment of patients, physicians, encounters, and practices using patient questionnaires, physician surveys, medical record audits, and direct observation of clinical encounters using the Davis Observation Code.¹⁰ That study’s initial findings were presented in the May 1998 theme issue of JFP, and the study processes have recently been described.¹¹ Details of the DOPC methods have been published elsewhere.^4,5,12

Although the initial design allowed the DOPC research nurses to collect brief observational notes, the intensity of the quantitative data collection limited the scope of the study’s qualitative data for understanding details of the practice’s organization and the competing demands within clinical encounters. As a consequence, the P&CD study was designed to provide more in-depth description and understanding of the competing demands of family practice, and in particular, to evaluate factors affecting preventive services delivery using a comparative case study design and a MAP.

A key feature of the P&CD study design was an openness to the integration of emerging insights into the data collection protocol. For example, preliminary analyses of the DOPC data¹³ and other ongoing studies¹⁴ led to the discovery that complexity science was valuable for explaining the dynamics of office systems⁶ and needed to be incorporated into the design. (Complexity science is the study of systems that are characterized by nonlinear dynamics and emergent properties; it emphasizes the need to understand the interrelationships of the whole system and not just collect data about the parts.¹⁵) The investigators also developed new ways to display the relationships among physicians and staff in the practices using “practice genograms.”¹⁶ The practice genogram is a diagram of the functional and interpersonal relationships among the clinicians, support staff, and other people and organizations interacting with the practice. Throughout the project and consistent with the standards of qualitative research design,17 there were continued modifications and enhancements in the data collection and analysis strategies in response to insights that were emerging from ongoing analyses and interpretation of the data.

An important feature of the project was the development of an advisory committee of consultants and co-investigators that convened annually to provide multidisciplinary input, review results, and provide feedback. The advisory committee included academic representatives with expertise in nursing, health education, women’s health, minority health, and public policy. Two additional members were added to the project to provide expertise into the study of organizations as complex systems. The annual reviews by the advisory committee led to significant changes in the research design while the study was ongoing.

Practice Sample

Beginning in late 1996, we drew from 91 practices in Nebraska that had been randomly selected to participate in an earlier study on tobacco prevention and cessation.¹⁸ Initially a sample of 10 practices was purposefully selected^19,20 using an iterative process to represent a range in size (small and large), geographic location (urban, suburban, and rural), and rate of delivery of tobacco-related preventive services. Preliminary analyses of these 10 initial practices provided a summary of preventive health delivery strategies in primary care practices and a description of competing demands that enhanced or limited these strategies. To confirm or refute the emerging insights from the original 10 practices, 8 additional practices were selected for further data collection during the second and third years of the study. The sampling strategy in years 2 and 3 ensured that at least 2 practices each from several major regional hospital health systems were included and allowed us to assess emerging hypotheses about the importance of health system context for understanding community practices.

The practices were recruited by contacting one of the physicians to solicit participation; only those in which all family physicians in the practice agreed to participate were included in the study. Twenty-three practices were contacted; all physicians in 18 agreed to participate (78%).

Core Data Collection Methods

Data were collected by trained field researchers who spent 4 weeks or more taking notes at each practice while observing the practice and clinical encounters, conducting informal key informant interviews of staff, collecting office documents, and auditing charts of patients whose encounters were observed. Within each practice, data collection occurred in stages, with a short break after the initial week or 2 of observation to allow preliminary analyses to inform additional data collection.

Observations at the practice level were recorded in a combination of structured observational checklists, unstructured dictated field notes,^21-23 and key informant interviews.²⁴ Detailed floor plans of the practice were used to identify where particular activities occurred and where individual practice participants worked. Each day at the practice, the field researcher took short notes or “jottings” and dictated expanded field notes in the evening.²³ A template of topics was used periodically to ensure that important aspects of the practice were not being overlooked. The template included lists of features of the community, practice, staff, and patients that the researchers saw as important Figure W1.*¹³ A 3-page structured practice environment checklist was adapted from earlier work on the DOPC project¹¹ and included a wide range of practice characteristics and functions, including items such as the number and training of staff, counseling options offered, and management of telephone calls and referrals Figure W2a. This checklist also served as a detailed reminder to the research nurse of topics to be included in the field notes. Throughout the time the field researchers were recording field notes and filling in the checklist, they opportunistically asked clinicians or staff informal key informant questions for confirmation or clarification.

Consecutive patients for each clinician in a practice were approached with a goal of recruiting 30 patients who would consent to have the field researcher observe their visit. This generally required approaching 35 to 40 patients. Because some clinicians worked part-time or were not consistently in the practice, it was not always possible to observe 30 visits. After explaining the study and gaining signed informed consent from patients who agreed to participate, the field researchers observed the outpatient visit as unobtrusively as possible. A 1-page structured encounter checklist Figure W3 that was also modified from the DOPC study11 provided blanks for noting the reason for the visit, chief complaints, and final diagnoses, and for indicating whether any of approximately 100 preventive services were ordered or delivered. Space was provided at the bottom of the form for recording notes that were later used to dictate a re-creation of the encounter. At a later time, a chart audit was done on each observed patient’s medical record using a structure chart audit form Figure W4.

After the initial observational data were transcribed, a preliminary practice genogram¹⁶ was drawn, and an initial practice summary was written. The genogram of practice participants, roles, and relationships was initially diagrammed on a white board by a transdisciplinary research team by interviewing the field researcher about the current and past practice clinicians and staff and about the health system and community. The demographics of individuals were recorded, including age, sex, years with the practice, percentage of work effort, and job responsibilities. Additional details included functional and emotional relationships observed in the practice, such as who worked together and any obvious conflicts among members of the practice or with health system or community affiliations. This process enabled the investigators to identify areas of incomplete data so the field researchers could return to fill in missing details.

In addition to the key informant interviews done as part of the observation activities, more formal semistructured individual depth interviews were arranged with each clinician and many of the office staff.²⁵ These interviews consisted of a 30-minute to 1-hour narrative interview in which the respondent was asked open-ended questions designed to elicit in-depth responses Figure W5. Although the major focus of these interviews was on the delivery of preventive services, more general questions were included to understand perspectives on practice process. For example, respondents were asked: “Could you describe for me a typical day for you in this practice?” and “If you believed a change was needed regarding some specific delivery of a service within this office, could you describe the process you would go through to try to get it implemented?” These interviews were audiotaped and transcribed verbatim.

To supplement the observational and interview data, field researchers gathered existing documents and artifacts from the practice. Items such as blank charts and flow sheets, patient schedules, personnel lists, samples of patient education materials and handouts, mission and vision statements, and annual reports were collected and compiled in binders. In some practices, particularly those affiliated with hospital health systems, materials were also available from Web sites. All transcribed interviews and dictated field notes from the practice and encounter observations were imported into FolioViews 4.11 (NextPage, Provo, Utah), a text-base management software program that facilitates coding, searching, and retrieval of large computerized text files.²⁶

Emerging Design Decisions

Midway through data collection at the first practice, the advisory committee met to review the data and discuss any concerns. The advisory committee identified a number of emerging hypotheses related to complexity science concepts that were used to guide subsequent data collection and management. It was deemed particularly important to identify “attractors”—factors in the practice and in the larger environment that influenced the structure and function of the practice as an organization.⁶ For example, an attractor might be a particular burning interest of one of the physicians, an expectation of the local hospital systems, or a dominant demographic characteristic of the patients being served.¹³ An expanded systemic model of primary care Figure 1 was articulated that characterized 6 core areas for data collection: (1) patient perceptions and behavior, (2) physician perceptions and behavior, (3) encounter structures and processes, (4) practice structures and processes, (5) community characteristics, and (6) the larger health system. This model identified the need for additional data on the community context and patient experience. Checklists were revised and field researchers were asked to spend more time gathering data about the community. It was also apparent that accurate calculation of certain preventive service delivery rates would require patient input and a larger sample size. For example, a patient exit card

Figure W6 was developed to ascertain self-reported tobacco use status (for all patients) and use of obstetrics and gynecology services and history of hysterectomy (for women). These data were used to determine a patient’s eligibility for tobacco cessation counseling and Papanicolaou tests, respectively. Observation of 30 encounters with each clinician was done to increase the stability for calculating rates for common preventive service recommendations.

A larger issue emerged with the discovery that, after contributing data over the course of weeks or months, members of the practices desired feedback in a timely manner. Practice clinicians and staff were very interested in how they were doing and asked when they would be receiving a report of our results. They did not want to wait 3 years for the completion of the study. Although ongoing analyses were anticipated, these had primarily been designed to ensure completeness of data and to provide feedback to field researchers on areas where clarifications were needed. In response to the emergent desire for feedback, the team generated rapid-turnaround summary reports for each practice. A summary report template was designed to present the descriptive details of the practice, including the practice genogram and a summary of the strengths and weaknesses of the practice’s prevention approach. The final page in the report provided the practice with a series of questions or points for self-reflection that often included process questions, such as “How can this organization become a team?” or “How can this practice deliver preventive services more consistently?” These reports were shared interactively with practices at a debriefing meeting within 2 to 3 months of completing data collection at each practice.

The feedback meetings provided an important opportunity to check the validity of the researchers’ analyses by comparing them with the perspectives of the practice participants. In all the practices, the response was a strong overall validation of the research team’s interpretation of the practice and its structures and processes. During the feedback presentations, the practice physicians and staff consistently made comments like, “Wow, did you ever get us,” or “This is like looking in a mirror.” In a number of sessions, the participants mentioned that the report raised issues about which they were vaguely aware and that the findings were stimulating considerable self-reflection. In several practices, the physicians disclosed that they would be taking actions in the future to modify some of the deficiencies the reports uncovered.

The next modification came as the data were being collected simultaneously in the second and third practices. We realized that despite our efforts to be nondisruptive, participating in the project required extra effort on the part of the practice. Each practice therefore received partial compensation in the form of a $500 certificate for the purchase of books or equipment.

After completing data collection at several practices, it was apparent that patients’ perspectives were still under-represented and that this was limiting the understanding of the practice. To gain further insights into patients’ experiences, beginning with the sixth practice we adopted the patient path approach described by Pommerenke and Dietrich,²⁷ following patients from the time they walk into the practice until the time they leave, using a patient path form for recording activities at different stages of the visit Figure W7. Additional brief open-ended interviews were conducted in the waiting room or examination rooms while patients were waiting.

Although we had asked the research nurses to be more thorough in their descriptions of the community, data from the community and health system were still incomplete. This became even more pronounced when studying practices that were part of health systems. Therefore, beginning with the sixth practice, we included in-depth interviews with individuals from health systems (eg, regional managers and medical directors). A further refinement came with the use of community key informant trees, a systematic process of identifying and interviewing members of the local community surrounding the practice.^24,28 These interviews of patients, church leaders, and other individuals from the community began with the ninth practice.

Once all the modifications were incorporated, the final case study design provided data at each of the 6 levels as shown in the Table 1. Particularly detailed data were available at the clinician, encounter, and practice levels. For example, at the clinician level the data included perceptions of roles as ascertained through the in-depth interviews, as well as actual behaviors recorded in the encounter field notes and chart audits. Insights on the structures and process of the practice were obtained through unstructured observations of the practice, structured checklists, written documents, and interviews. Supporting data were collected on patients’ perspectives, the community, and the health system that provided contexts for the practice case studies.

Discussion

The complexity of primary care practices is best understood from multiple perspectives,²⁹ a principle that guided the initial selection of a multimethod comparative case study design for this investigation. The MAP that emerged from this comparative case study design has a number of strengths and weaknesses. A particular asset of the design was our ability to investigate specific phenomena within their context rather than isolated from it.²⁹ The design also encouraged the investigators to pursue emerging insights, thus informing multiple perspectives that might not have otherwise been considered, although this may be somewhat limited by the purposeful sampling strategy that focuses on maximizing information about a particular topic.

Limitations

A limitation for broader implementation of this research design is the intensity of data collection and analysis, which are difficult to accomplish without considerable resources and a research team with diverse skills. There might also be some concerns about the ways that the data collection process alters practice behavior; however, the prolonged observational time frame and the multiple data sources for “triangulation” are designed to limit any potential Hawthorne effect. That is, by collecting and systematically comparing data from multiple sources, including direct observation, different forms of interviews, and existing documents, the investigators were able to identify inconsistencies in patterns of behavior.^17,19

Although the data collection, analysis, and feedback process appeared to increase a practice’s self-reflection, our study limited the input of patient and practice participants in the design, analysis, and interpretation and thus does not approach the participatory research paradigm espoused by McCauley and colleagues.³⁰ Still, our study moved from being primarily observational descriptive research into a more collaborative and interventional project, in part at the request of the participants. This suggests a model and method for future research in the arena of health care process and outcome improvement with the practices as collaborators. The MAP characterized in this paper offers a means for simultaneously describing, understanding, and improving the richly complex and varied processes and outcomes of primary care. By more actively engaging the practices in the research process, the MAP also points toward a new more collaborative model of practice-based research.

Conclusions

The comprehensive data in the P&CD study provide a unique opportunity to understand and describe multiple perspectives from the clinician, patient, encounter, practice, community, and health system spheres. Each of the papers in this issue of JFP used some of these comprehensive data to study one or more of these spheres. For example, the encounter field notes were the primary source of data for exploring how “family” presents in encounters³¹Several of the authors used subsets of patients, including patients presenting with acute respiratory track infections,³³ smokers,³⁴ and frequent attenders.³⁵ These authors each supplemented the encounter field notes with data from the medical record reviews, medical record field notes, and patient exit cards. The complete data set including practice field notes, practice genograms, physician and staff interviews, office environment checklists, and encounter field notes were used to describe staff training, roles, and functions.³⁶ This is only a part of the research adventure available in this type of data. We hope many others will join in the excitement.

Acknowledgments

Our study was supported by a grant from the Agency for Healthcare Research and Quality (R01 HS08776) and a Research Center grant from the American Academy of Family Physicians. Drs Crabtree, Miller and Stange are associated with the Center for Research in Family Practice and Primary Care, Cleveland, New Brunswick, Allentown, and San Antonio. We are grateful to the physicians, staff, and patients from the 18 practices, without whose participation this study would not have been possible. We also wish to thank Connie Gibbs and Jen Rouse, who spent many hours in the practices collecting data, and Diane Dodendorf and Jason Lebsack, who spent countless hours coordinating transcription and data management activities, for their dedicated work. We are especially indebted to Mary McAndrews, who transcribed hundreds of taped interviews and dictated field notes. The ongoing analyses that ensured the quality and comprehensiveness of the data were made possible through the dedicated work of Helen McIlvain, PhD; Jeffrey Susman, MD; Virginia Aita, PhD; Kristine McVea, MD; Elisabeth Backer, MD; Paul Turner, PhD; and Louis Pol, PhD. Finally, we thank the members of the advisory committee: Valerie Gilchrist, MD; Paul Nutting, MD, MPH; Carlos Jaén, MD, PhD; Kurt Stange, MD, PhD; William Miller, MD, MA; Reuben McDaniel, PhD; and Ruth Anderson, RN, PhD.

Clinicians, researchers, and policymakers now recognize that multiple competing demands1 and opportunities² are simultaneously affecting the physicians, staff, and patients within primary care practices. Our current understanding of outpatient practice is largely based on administrative databases, national surveys, and medical record reviews, with additional insights from surveys of patients or clinicians. These data generally are not designed to capture the richness of the content and context that is needed to better understand the realities and complexities of practice.^3-6 The underlying premise of The Prevention and Competing Demands in Primary Care (P&CD) Study is that efforts to change practice should be preceded by efforts to understand it.^2,7 The explicit goal of this study is to understand practice structure and process, including details of patients, physicians, staff, and clinical encounters; the practice as an organization; and its relationship to the larger community and health system.

In this paper we describe a dynamic observational multimethod assessment process (MAP) that can be used to understand the complex reality of primary care practice. MAP is based on a multimethod comparative case study design^8,9 that integrates elements of epidemiology with methods derived from the qualitative traditions of anthropology and sociology and relies most heavily on qualitative observation and interviewing methods. Studies of this type require an iterative data collection and analysis approach that evolves over time so that new methods can be introduced as the investigators gain a better understanding of important issues. A major strength of our study design was that it allowed hypotheses and insights gained from participants and from ongoing analyses to be integrated into the ongoing investigation.

The study’s primary research questions related to how practice characteristics affect preventive service delivery. Thus, the research design included: (1) an examination of the organizational contexts that support preventive services, (2) an examination of the competing demands imposed by carrying out clinical prevention and illness care in clinical encounters and in the practice, (3) a comparison of the approaches used by practices with high versus low intensity of preventive services delivered to eligible patients, and (4) an examination of approaches used to deliver different types of preventive services. Although the particular focus was on preventive services, the rich MAP allowed pursuit of other research topics that are presented in this issue of JFP.

This article describes the evolutionary methods of the P&CD study, focusing on how data were collected to ensure that sufficient details were available to understand a practice’s values, structures, and processes.*

Emergent research design

The P&CD study was conceived in 1994 to be an in-depth follow-up of insights from the Direct Observation of Primary Care (DOPC) Study that was just getting under way in northeastern Ohio.⁵ The DOPC Study provided a largely quantitative assessment of patients, physicians, encounters, and practices using patient questionnaires, physician surveys, medical record audits, and direct observation of clinical encounters using the Davis Observation Code.¹⁰ That study’s initial findings were presented in the May 1998 theme issue of JFP, and the study processes have recently been described.¹¹ Details of the DOPC methods have been published elsewhere.^4,5,12

Although the initial design allowed the DOPC research nurses to collect brief observational notes, the intensity of the quantitative data collection limited the scope of the study’s qualitative data for understanding details of the practice’s organization and the competing demands within clinical encounters. As a consequence, the P&CD study was designed to provide more in-depth description and understanding of the competing demands of family practice, and in particular, to evaluate factors affecting preventive services delivery using a comparative case study design and a MAP.

A key feature of the P&CD study design was an openness to the integration of emerging insights into the data collection protocol. For example, preliminary analyses of the DOPC data¹³ and other ongoing studies¹⁴ led to the discovery that complexity science was valuable for explaining the dynamics of office systems⁶ and needed to be incorporated into the design. (Complexity science is the study of systems that are characterized by nonlinear dynamics and emergent properties; it emphasizes the need to understand the interrelationships of the whole system and not just collect data about the parts.¹⁵) The investigators also developed new ways to display the relationships among physicians and staff in the practices using “practice genograms.”¹⁶ The practice genogram is a diagram of the functional and interpersonal relationships among the clinicians, support staff, and other people and organizations interacting with the practice. Throughout the project and consistent with the standards of qualitative research design,17 there were continued modifications and enhancements in the data collection and analysis strategies in response to insights that were emerging from ongoing analyses and interpretation of the data.

An important feature of the project was the development of an advisory committee of consultants and co-investigators that convened annually to provide multidisciplinary input, review results, and provide feedback. The advisory committee included academic representatives with expertise in nursing, health education, women’s health, minority health, and public policy. Two additional members were added to the project to provide expertise into the study of organizations as complex systems. The annual reviews by the advisory committee led to significant changes in the research design while the study was ongoing.

Practice Sample

Beginning in late 1996, we drew from 91 practices in Nebraska that had been randomly selected to participate in an earlier study on tobacco prevention and cessation.¹⁸ Initially a sample of 10 practices was purposefully selected^19,20 using an iterative process to represent a range in size (small and large), geographic location (urban, suburban, and rural), and rate of delivery of tobacco-related preventive services. Preliminary analyses of these 10 initial practices provided a summary of preventive health delivery strategies in primary care practices and a description of competing demands that enhanced or limited these strategies. To confirm or refute the emerging insights from the original 10 practices, 8 additional practices were selected for further data collection during the second and third years of the study. The sampling strategy in years 2 and 3 ensured that at least 2 practices each from several major regional hospital health systems were included and allowed us to assess emerging hypotheses about the importance of health system context for understanding community practices.

The practices were recruited by contacting one of the physicians to solicit participation; only those in which all family physicians in the practice agreed to participate were included in the study. Twenty-three practices were contacted; all physicians in 18 agreed to participate (78%).

Core Data Collection Methods

Data were collected by trained field researchers who spent 4 weeks or more taking notes at each practice while observing the practice and clinical encounters, conducting informal key informant interviews of staff, collecting office documents, and auditing charts of patients whose encounters were observed. Within each practice, data collection occurred in stages, with a short break after the initial week or 2 of observation to allow preliminary analyses to inform additional data collection.

Observations at the practice level were recorded in a combination of structured observational checklists, unstructured dictated field notes,^21-23 and key informant interviews.²⁴ Detailed floor plans of the practice were used to identify where particular activities occurred and where individual practice participants worked. Each day at the practice, the field researcher took short notes or “jottings” and dictated expanded field notes in the evening.²³ A template of topics was used periodically to ensure that important aspects of the practice were not being overlooked. The template included lists of features of the community, practice, staff, and patients that the researchers saw as important Figure W1.*¹³ A 3-page structured practice environment checklist was adapted from earlier work on the DOPC project¹¹ and included a wide range of practice characteristics and functions, including items such as the number and training of staff, counseling options offered, and management of telephone calls and referrals Figure W2a. This checklist also served as a detailed reminder to the research nurse of topics to be included in the field notes. Throughout the time the field researchers were recording field notes and filling in the checklist, they opportunistically asked clinicians or staff informal key informant questions for confirmation or clarification.

Consecutive patients for each clinician in a practice were approached with a goal of recruiting 30 patients who would consent to have the field researcher observe their visit. This generally required approaching 35 to 40 patients. Because some clinicians worked part-time or were not consistently in the practice, it was not always possible to observe 30 visits. After explaining the study and gaining signed informed consent from patients who agreed to participate, the field researchers observed the outpatient visit as unobtrusively as possible. A 1-page structured encounter checklist Figure W3 that was also modified from the DOPC study11 provided blanks for noting the reason for the visit, chief complaints, and final diagnoses, and for indicating whether any of approximately 100 preventive services were ordered or delivered. Space was provided at the bottom of the form for recording notes that were later used to dictate a re-creation of the encounter. At a later time, a chart audit was done on each observed patient’s medical record using a structure chart audit form Figure W4.

After the initial observational data were transcribed, a preliminary practice genogram¹⁶ was drawn, and an initial practice summary was written. The genogram of practice participants, roles, and relationships was initially diagrammed on a white board by a transdisciplinary research team by interviewing the field researcher about the current and past practice clinicians and staff and about the health system and community. The demographics of individuals were recorded, including age, sex, years with the practice, percentage of work effort, and job responsibilities. Additional details included functional and emotional relationships observed in the practice, such as who worked together and any obvious conflicts among members of the practice or with health system or community affiliations. This process enabled the investigators to identify areas of incomplete data so the field researchers could return to fill in missing details.

In addition to the key informant interviews done as part of the observation activities, more formal semistructured individual depth interviews were arranged with each clinician and many of the office staff.²⁵ These interviews consisted of a 30-minute to 1-hour narrative interview in which the respondent was asked open-ended questions designed to elicit in-depth responses Figure W5. Although the major focus of these interviews was on the delivery of preventive services, more general questions were included to understand perspectives on practice process. For example, respondents were asked: “Could you describe for me a typical day for you in this practice?” and “If you believed a change was needed regarding some specific delivery of a service within this office, could you describe the process you would go through to try to get it implemented?” These interviews were audiotaped and transcribed verbatim.

To supplement the observational and interview data, field researchers gathered existing documents and artifacts from the practice. Items such as blank charts and flow sheets, patient schedules, personnel lists, samples of patient education materials and handouts, mission and vision statements, and annual reports were collected and compiled in binders. In some practices, particularly those affiliated with hospital health systems, materials were also available from Web sites. All transcribed interviews and dictated field notes from the practice and encounter observations were imported into FolioViews 4.11 (NextPage, Provo, Utah), a text-base management software program that facilitates coding, searching, and retrieval of large computerized text files.²⁶

Emerging Design Decisions

Midway through data collection at the first practice, the advisory committee met to review the data and discuss any concerns. The advisory committee identified a number of emerging hypotheses related to complexity science concepts that were used to guide subsequent data collection and management. It was deemed particularly important to identify “attractors”—factors in the practice and in the larger environment that influenced the structure and function of the practice as an organization.⁶ For example, an attractor might be a particular burning interest of one of the physicians, an expectation of the local hospital systems, or a dominant demographic characteristic of the patients being served.¹³ An expanded systemic model of primary care Figure 1 was articulated that characterized 6 core areas for data collection: (1) patient perceptions and behavior, (2) physician perceptions and behavior, (3) encounter structures and processes, (4) practice structures and processes, (5) community characteristics, and (6) the larger health system. This model identified the need for additional data on the community context and patient experience. Checklists were revised and field researchers were asked to spend more time gathering data about the community. It was also apparent that accurate calculation of certain preventive service delivery rates would require patient input and a larger sample size. For example, a patient exit card

Figure W6 was developed to ascertain self-reported tobacco use status (for all patients) and use of obstetrics and gynecology services and history of hysterectomy (for women). These data were used to determine a patient’s eligibility for tobacco cessation counseling and Papanicolaou tests, respectively. Observation of 30 encounters with each clinician was done to increase the stability for calculating rates for common preventive service recommendations.

A larger issue emerged with the discovery that, after contributing data over the course of weeks or months, members of the practices desired feedback in a timely manner. Practice clinicians and staff were very interested in how they were doing and asked when they would be receiving a report of our results. They did not want to wait 3 years for the completion of the study. Although ongoing analyses were anticipated, these had primarily been designed to ensure completeness of data and to provide feedback to field researchers on areas where clarifications were needed. In response to the emergent desire for feedback, the team generated rapid-turnaround summary reports for each practice. A summary report template was designed to present the descriptive details of the practice, including the practice genogram and a summary of the strengths and weaknesses of the practice’s prevention approach. The final page in the report provided the practice with a series of questions or points for self-reflection that often included process questions, such as “How can this organization become a team?” or “How can this practice deliver preventive services more consistently?” These reports were shared interactively with practices at a debriefing meeting within 2 to 3 months of completing data collection at each practice.

The feedback meetings provided an important opportunity to check the validity of the researchers’ analyses by comparing them with the perspectives of the practice participants. In all the practices, the response was a strong overall validation of the research team’s interpretation of the practice and its structures and processes. During the feedback presentations, the practice physicians and staff consistently made comments like, “Wow, did you ever get us,” or “This is like looking in a mirror.” In a number of sessions, the participants mentioned that the report raised issues about which they were vaguely aware and that the findings were stimulating considerable self-reflection. In several practices, the physicians disclosed that they would be taking actions in the future to modify some of the deficiencies the reports uncovered.

The next modification came as the data were being collected simultaneously in the second and third practices. We realized that despite our efforts to be nondisruptive, participating in the project required extra effort on the part of the practice. Each practice therefore received partial compensation in the form of a $500 certificate for the purchase of books or equipment.

After completing data collection at several practices, it was apparent that patients’ perspectives were still under-represented and that this was limiting the understanding of the practice. To gain further insights into patients’ experiences, beginning with the sixth practice we adopted the patient path approach described by Pommerenke and Dietrich,²⁷ following patients from the time they walk into the practice until the time they leave, using a patient path form for recording activities at different stages of the visit Figure W7. Additional brief open-ended interviews were conducted in the waiting room or examination rooms while patients were waiting.

Although we had asked the research nurses to be more thorough in their descriptions of the community, data from the community and health system were still incomplete. This became even more pronounced when studying practices that were part of health systems. Therefore, beginning with the sixth practice, we included in-depth interviews with individuals from health systems (eg, regional managers and medical directors). A further refinement came with the use of community key informant trees, a systematic process of identifying and interviewing members of the local community surrounding the practice.^24,28 These interviews of patients, church leaders, and other individuals from the community began with the ninth practice.

Once all the modifications were incorporated, the final case study design provided data at each of the 6 levels as shown in the Table 1. Particularly detailed data were available at the clinician, encounter, and practice levels. For example, at the clinician level the data included perceptions of roles as ascertained through the in-depth interviews, as well as actual behaviors recorded in the encounter field notes and chart audits. Insights on the structures and process of the practice were obtained through unstructured observations of the practice, structured checklists, written documents, and interviews. Supporting data were collected on patients’ perspectives, the community, and the health system that provided contexts for the practice case studies.

Discussion

The complexity of primary care practices is best understood from multiple perspectives,²⁹ a principle that guided the initial selection of a multimethod comparative case study design for this investigation. The MAP that emerged from this comparative case study design has a number of strengths and weaknesses. A particular asset of the design was our ability to investigate specific phenomena within their context rather than isolated from it.²⁹ The design also encouraged the investigators to pursue emerging insights, thus informing multiple perspectives that might not have otherwise been considered, although this may be somewhat limited by the purposeful sampling strategy that focuses on maximizing information about a particular topic.

Limitations

A limitation for broader implementation of this research design is the intensity of data collection and analysis, which are difficult to accomplish without considerable resources and a research team with diverse skills. There might also be some concerns about the ways that the data collection process alters practice behavior; however, the prolonged observational time frame and the multiple data sources for “triangulation” are designed to limit any potential Hawthorne effect. That is, by collecting and systematically comparing data from multiple sources, including direct observation, different forms of interviews, and existing documents, the investigators were able to identify inconsistencies in patterns of behavior.^17,19

Although the data collection, analysis, and feedback process appeared to increase a practice’s self-reflection, our study limited the input of patient and practice participants in the design, analysis, and interpretation and thus does not approach the participatory research paradigm espoused by McCauley and colleagues.³⁰ Still, our study moved from being primarily observational descriptive research into a more collaborative and interventional project, in part at the request of the participants. This suggests a model and method for future research in the arena of health care process and outcome improvement with the practices as collaborators. The MAP characterized in this paper offers a means for simultaneously describing, understanding, and improving the richly complex and varied processes and outcomes of primary care. By more actively engaging the practices in the research process, the MAP also points toward a new more collaborative model of practice-based research.

Conclusions

The comprehensive data in the P&CD study provide a unique opportunity to understand and describe multiple perspectives from the clinician, patient, encounter, practice, community, and health system spheres. Each of the papers in this issue of JFP used some of these comprehensive data to study one or more of these spheres. For example, the encounter field notes were the primary source of data for exploring how “family” presents in encounters³¹Several of the authors used subsets of patients, including patients presenting with acute respiratory track infections,³³ smokers,³⁴ and frequent attenders.³⁵ These authors each supplemented the encounter field notes with data from the medical record reviews, medical record field notes, and patient exit cards. The complete data set including practice field notes, practice genograms, physician and staff interviews, office environment checklists, and encounter field notes were used to describe staff training, roles, and functions.³⁶ This is only a part of the research adventure available in this type of data. We hope many others will join in the excitement.

Acknowledgments

Our study was supported by a grant from the Agency for Healthcare Research and Quality (R01 HS08776) and a Research Center grant from the American Academy of Family Physicians. Drs Crabtree, Miller and Stange are associated with the Center for Research in Family Practice and Primary Care, Cleveland, New Brunswick, Allentown, and San Antonio. We are grateful to the physicians, staff, and patients from the 18 practices, without whose participation this study would not have been possible. We also wish to thank Connie Gibbs and Jen Rouse, who spent many hours in the practices collecting data, and Diane Dodendorf and Jason Lebsack, who spent countless hours coordinating transcription and data management activities, for their dedicated work. We are especially indebted to Mary McAndrews, who transcribed hundreds of taped interviews and dictated field notes. The ongoing analyses that ensured the quality and comprehensiveness of the data were made possible through the dedicated work of Helen McIlvain, PhD; Jeffrey Susman, MD; Virginia Aita, PhD; Kristine McVea, MD; Elisabeth Backer, MD; Paul Turner, PhD; and Louis Pol, PhD. Finally, we thank the members of the advisory committee: Valerie Gilchrist, MD; Paul Nutting, MD, MPH; Carlos Jaén, MD, PhD; Kurt Stange, MD, PhD; William Miller, MD, MA; Reuben McDaniel, PhD; and Ruth Anderson, RN, PhD.

One major focus of health services research and quality improvement efforts is to identify and reduce variation.^1-4 Standardization is the approach usually offered to minimize variation, thus reducing errors and increasing quality.^5,6 These interventions are often based on an industrial quality improvement paradigm⁷ using linear interventions that assume that inputs reliably lead to proportionate responses.⁸ These interventions include re-engineering and expanded information systems.^9,10 If the application of linear Newtonian views is correct, then standardization is the key to quality improvement, and effective practices will look much alike. The search for and attempt to implement best practice guidelines^11-14 are examples of efforts to bring practices into conformity and to establish process standards for best behavior. However, the search for simple, easily transportable interventions has not been as successful as traditional logic might suggest.^15-19

Emerging views of organizations derived from complexity science bring the key understanding that practices are more than commodity-delivering businesses—they are complex adaptive systems.²⁰ These systems involve connected participants interacting in ways that generate the spontaneous emergence of new structures and behaviors. In complex adaptive systems, we expect to see variation in practice patterns, even when the outcomes of practices are similar.

In a previous issue of JFP,²¹ we proposed a model of primary care practices as complex adaptive systems and suggested implications and strategies for change. Since then we have begun applying this theoretical framework to other studies designed to understand and advance generalist practice. Our present purpose is to advance the application of complexity science to understanding and improving primary care practices and their co-evolving health care systems.

The theory application process

The 3 studies that this theory application builds on began with the Direct Observation of Primary Care (DOPC) study, a 3-year (1994-1997) multimethod descriptive investigation of the content of 4454 patient visits to 138 family physicians in 84 family practices.^22,23 One of the outcomes of this study was a model for understanding change in family practice based on complexity science.²¹ Subsequently, The Prevention and Competing Demands in Primary Care Study (PCDPC) was a 3-year (1996-1999) in-depth descriptive case study of 1637 outpatient visits to 56 clinicians in18 family practices purposefully sampled to include diversity in geographic location, size of practice, and intensity of delivery of preventive services.²⁴ We also implemented a 4-year (1997-2001) multimethod clinical trial, the Study to Enhance Prevention by Understanding and Practice (STEP-UP), to understand and improve the delivery of preventive health services in 77 family practices.²⁵

We conducted an explicit theory application and refinement process consisting of 10 meetings to analyze data from DOPC, PCDPC, and STEP-UP, informed by a literature review of complexity science. The developing model was explicitly tested in 3 Nebraska family practices in 1999 in an effort to improve diabetes care.²⁶ The resulting specific theory application to family practices was then evaluated using 2 cases from these studies.

Application of complexity science to family practice

Synthesizing our observations of the family practices and the literature review, we developed the following theoretical model. Family practices are local professional complex adaptive systems with the primary purpose of seeing patients for everyday health concerns to assist “them” in getting on with their daily lives. “Them” refers to the patients and their families, the clinicians, and the office staff. Practice leaders and managers usually describe their practices in terms of efficiency, productivity, adherence to standards of care, and patient satisfaction. Increasingly, practices function in interactive ways with managers or owners from local health care systems. Still, these practices behaved more like complex adaptive systems operating within a professional milieu than like businesses.²⁷

Complex adaptive systems are like a family reunion; they are dynamic-bounded webs of diverse agents interacting nonlinearly.²⁸ Dynamic refers to the continual presence of multiple interactions and their accompanying surprises, challenges, and responses, both within the system and between the system (eg, practices) and its environment. Bounded refers to the defining purpose or intent of the system (eg, to deliver health care to local patients). The metaphor of a web characterizes the multiple interconnections of the system. The agents in practices include clinicians, office staff, and patients, and can also include pharmaceutical representatives, health care system administrators, and others. Agents have the capacity to exchange information, learn, and adjust their behavior. No individual agent can ever know or understand everything that is occurring. The nonlinear relationships among the agents are the result of ongoing feedback loops and mean that small changes can lead to large effects and big changes can lead to small effects. For example, the introduction of a small medical record stamp to identify smokers in one practice leads to a dramatic increase in smoking counseling, while a major quality improvement program in another practice results in minimal change.

What makes an organization “professional” is the application of specialized values and expertise to address difficult problems and uncertainty. These values and skills are acquired through specialized training and are created by the larger social context.²⁹ It is the socially defined professional values and expertise of the family physician that are applied in their daily activities. A co-evolving professional world of the health system and payer manager is increasingly interacting with the physician professional value system.

Each family practice is unique because of 5 features:

1. History and initial conditions,including any explicit or implicit mission and the underlying priorities for the practice.
2. Particular agents and their unique styles and interests.
3. The pattern of nonlinear interactions among agents.
4. The local fitness landscape(ie, the practice’s ecological niche) and its particular expectations, community values, competitive issues, and ecology.
5. Regional and global influences, such as larger health care systems, finances and regulations, and culture.

The local fitness landscape, a complexity science term from evolutionary biology, specifically refers to the local terrain and all the many complex adaptive systems, from microbes to organizations, that seek their own purpose and niche within that terrain. Biological evolution and technological evolution are processes attempting to optimize systems riddled with conflicting constraints.³⁰ Each family practice must evolve by attempting to optimize the entire package of services it delivers. This evolution must account for all the other competing and cooperating health care services and local resources, such as economic conditions, availability of insurance types, and the particular local disease and illness epidemiology.

How all of these factors manifest in a particular family practice at any given time and over time can be understood using 3 complexity science properties: self-organization, emergence, and co-evolution. Self-organization refers to the spontaneous development of structures and forms of behavior in systems characterized by multiple feedback loops and nonlinear dynamics. These structures are a function of the patterns of relationships among agents. Everything changes in response to and as a result of everything else, with each complex adaptive system seeking a better position in its local fitness landscape—a niche where it can prosper and survive.³¹ In this setting of ongoing co-evolution, both competition and collaboration become strategies for workable solutions. As the agents of any complex adaptive system interact, novelty and surprise continuously emerge in unpredictable ways. For example, a new successful approach to scheduling is introduced by an unassuming receptionist. This emergence creates a system that is greater than the sum of its parts; it is what cannot be understood through a reductionist (one problem at a time) examination of the practice.³²

A particular family practice is the unique self-organized system that emerges when particular physicians and staff (agents) come together in particular ways with particular goals, preferences, and priorities (initial conditions) within a particular community setting (local fitness landscape) given specific regional and global influences. At any future point, this practice is the unique self-organized system that has emerged through co-evolution with all the other systems in the local fitness landscape.

The result is much variation between and within family practices. Practices have much in common, however, because of their common goal of seeing patients to assist them with their everyday health problems in a shared cultural and historical context. From that perspective, variation in family practices is inevitable and a powerful source of creative possibility, value, and good clinical practice. Practices use 2 strategies for successfully enhancing that creativity: sensemaking and improvisation. Sensemaking is a social activity that requires interaction among agents.^33,34 People must come to have some notion of “Who am I,” “Why am I here,” and “What is going on around me?” Improvisation is a strategy for dealing with surprise in complex adaptive systems. Improvisation can be described as intuition guiding action in a spontaneous way.³⁵ Intuition is not a random guess at what to do, but the result of using high levels of expertise to act in the moment.^27,36

Among the many practices in the studies we are currently analyzing, multiple areas of variation are observed. These include differences in charting systems, clinical care decisions, scheduling, billing and coding procedures, staff relationships, and management and clinical styles. Sometimes these variations provide an adaptive advantage, but often not, and it is seldom clear in advance which will be true. Inflexible standardization, however, is often poorly responsive to the needs of different practices’ diverse agents and to the almost constant situations of uncertainty, contextual uniqueness, and surprise that occur in the practices.

Case studies

To illustrate the application of complexity science–based sensemaking to family practice, we present 2 case studies.

We selected 2 practices that had high quality of care as measured by delivery of preventive health services and patient satisfaction. One case takes advantage of the longitudinal data from DOPC and STEP-UP, and the other makes use of the more in-depth cross-sectional data from PCDPC. The cases were also selected to assure maximum variation in location and affiliation, and homogeneity in practice size (4-8 clinicians). The names have been changed to protect confidentiality.

Franchise Family Practice

History/Initial Conditions
Franchise is one of several primary care offices created by the Health Salute Corporation in affluent suburban areas of intense competition for market share growth. The corporate intent for this practice is to be productive and profitable. Two family physicians, a pediatrician, and a nurse practitioner were brought in from other practices, and several of their staff members followed. They all agree that their mission is to be the best practice in the Health Salute Corporation. Their identity is to capture market share through better efficiency, a mechanistic approach (scientific and standardized), and a friendly and caring attitude.

Agents and Patterns of Interaction
The practice manager’s daily attire in stockings and heels sets the tone for interactions, which are formal and professional. A small core of staff is dedicated to this practice, but there is often temporary help from other Health Salute offices during busy times. The patient population of predominantly mobile, insured, 2 working parent families tends to value convenience over relationships. The physicians seem to have little emotional investment in this particular practice, place, or each other. Conflicts are minimized and usually covered over with humor.

Local Fitness Landscape
There is no clear sense of community in this new suburb. Franchise is located in the heart of “minivan land,” an unrolling suburban carpet. The 2 competing systems are a major threat and are constantly being discussed. It is very clear that the survival of Franchise is dependent on success in the marketplace as determined by Health Salute.

Regional/Global Influences
Managed care has a strong presence, with much pressure to implement multiple practice guidelines, frequent chart audits, and different formularies.

Self-Organization
In many respects, Franchise Family Practice comes close to fulfilling its mission. Franchise is friendly, fun loving, and clean. It is a high performer at delivering preventive health services and is full of glitz and protocols. There are multiple systems in place for all phases of practice operation, and the manager sees they are working.

Emergence
Despite this managed order, surprises, problems, uncertainty, and complexities keep arising on a daily basis. Occasionally individuals respond creatively, but more often they stick to the protocols and generate even more trouble. There are frequent staff meetings where common problems are discussed. Many different solutions emerge in these discussions, but the final resolution is usually based on what the practice management thinks Health Salute would want. Even in this intensely structured practice, multiple competing demands, power distributions, and interpersonal battles are being simultaneously worked out on a daily basis.

Co-Evolution
As the suburbs grew, more practices were opened. The original practice in the area was soon challenged by Franchise and then by another competitor. Each of the 3 practices often acted or reacted in response to the others. Approximately a year after our research ended, Franchise Family Practice was closed by Health Salute because of inadequate profitability, and within a few months the second competitor also closed its practice.

Dusty Garden Family Practice

History/Initial Conditions
Dusty Garden began as a pioneering model for community-oriented primary care in an economically impoverished urban area. The practice was created with a focus on the patient in this underserved community. Envisioned by its founding family physician and practice manager, this practice was established in close collaboration with a community board. Survival is dependent on the ability to obtain funding for many poorly reimbursed services.

Agents and Patterns of Interaction
Dusty Garden has a dense and diverse web of complex interdependence. During the first few years of our research, most practice staff members came from the community. The practice grew from 4 to 6 family physicians and 2 nurse practitioners, and there was also much staff turnover. Dusty Garden was often a stepping stone for some clinicians, a chance to work in an “idealistic place” before going on to other things. However, the leadership has remained stable.

Local Fitness Landscape
The health care needs in the community were great, and the practice responded by growing rapidly, at times exceeding resources. At the same time, the local market consolidated into 2 competing hospital systems, with nearly all practices officially aligned with one of them. Necessary external funding also became more difficult to obtain.

Regional/Global Influences
Patients were represented by a mix of insurers. Uninsured or underinsured patients were cared for under a sliding-scale reimbursement scheme. There was also a perceived need to demonstrate successes and quality of care, and to place more emphasis on productivity and efficiency.

Self-Organization
The original practice was located in a dusty and cluttered building. It was difficult to tell who was responsible for what, but a shared sense of purpose gave the practice a family feel. Conflicts were evident but quickly resolved by frank discussions and a shared commitment to the practice mission. Schedules were constantly being disrupted by responding to patients and staff members’ diverse needs. In spite of this seeming chaos, Dusty Garden was an exemplar at delivering preventive services. This was accomplished by several dedicated clinicians and practice systems that involved the active participation of multiple personnel.

Co-Evolution
Significant change was occurring, and the practice was pushed to divide or grow in response to increasing patient demand and community need. They chose to grow and move into a much larger, newer, and more functional building down the street, resulting in a set of unanticipated consequences as both they and the fitness landscape changed. The new facility was more accessible and visible to a demographically different set of patients. Because of the changes in the local health care system, Dusty Garden also felt compelled to develop a relationship with the academic hospital system.

Emergence
What emerged was an organization where staff were isolated in large functionally differentiated spaces. The greater practice size demanded more specialization, and patterns of relationships dramatically changed. The change altered the number and specifics of the agents; many of the community-based staff left; and there was frequent turnover among newly hired and overwhelmed front office staff. The change also altered the number and character of the interactions per agent. Still, the vision remains strong, and many meetings are occurring to restore a “new” sense of practice community. Preventive service delivery rates remain high. The leadership initially responded to these changes with efforts aimed at greater standardization, but these are now being balanced by paying more attention to what solutions are being improvised “on the ground.”

Case study analysis

A comparative analysis of these 2 cases provides the following insights based on a complexity science view of the world:

1. Each practice was performing well using the delivery of preventive health and patient satisfaction as proxies for total practice performance.
2. The practices differed from each other in critical ways that seem to be at odds with traditional “best practices” thinking.
3. The practices were similar in that they had each organized themselves, coevolved, and emerged as a function of the nonlinear interdependencies among agents and the local fitness landscape, not solely as a function of some externally imposed script.
4. Each practice engaged in sensemaking activity to understand its unfolding world.
5. Each practice engaged in improvisational behavior as a strategy for developing strategic and tactical responses to its unfolding world.
6. Variation was often a source of strength, not a sign of bad practice.

Discussion

The traditional,^2-4 largely unsubstantiated,^37-40 view is that the best way to improve care is to eliminate variation. A view of family practice informed by complexity science suggests otherwise. In complex adaptive systems, agents in the practices create responses to changing circumstances—they improvise, or play practice jazz. Jazz players are often seen as role models of sensemaking and improvisational behavior.²⁸ They know a general musical structure, and within that they create jazz. Bad jazz occurs when one person plays what the others cannot make sense of and build on. All the players have an interdependent responsibility to create good jazz. When good jazz players hear something unexpected, they make sense of it and improvise. Dealing with the uncertain nature of complex adaptive systems involves thinking in terms of making sense of what is emerging. How can I improvise to use whatever happens to further the system’s development? It involves building on emergent characteristics of the complex adaptive system to develop patterns of social interaction⁴¹ among agents that give them confidence in each other, lead to small wins, and enhance the capacity to learn from unpredicted events.⁴²

Nevertheless, differentiating desirable from undesirable variation is an opportunity to learn from our history, and an opportunity to improve our practice jazz.^37,41,43 Small changes can have large results in some settings, while large efforts may lead to meager results in others. Complexity theory offers a framework for understanding these phenomena in family practice, and lays the groundwork for future research. On the basis of the proposed theoretical model, we hypothesize that it is critical to differentiate the variations that are sources of error from the variations due to the dynamics of relationships. From the perspective of complexity science, relational variation is linked to diversity among agents and represents constructive and adaptive variations and emergent behavior within an ever-changing and unpredictable local fitness landscape. From this perspective, the goals are to eliminate error through development of better systems of operation and to reduce confusion and poor judgment by improving sensemaking and communication. It is also important to enhance desirable variation by developing the skills of the relationship-centered clinical method,^44-46 improvisation, and reflective practice.^47-50

Sensemaking may be enhanced by considering the 4 ways of knowing health and health care,⁵¹ which include understanding: (1) the clinician; (2) the patient, family, and community; (3) systems; and (4) scientific evidence about disease and treatment. Judgments about the variation can be made within each way of knowing. Desirable variation due to clinician factors should build on each clinician’s unique skills and values, and compensate for or improve weaknesses. Local adaptation of objective evidence and the development of unique approaches to meeting the needs of patients in their personal and local context represent potentially desirable sources of variation. Evidence-based medicine provides a basis for reducing variation on the basis of scientific knowledge developed from studies of groups of individuals. Systems that integrate scientific evidence with the unique needs of patients, families, and communities and the specific talents of clinicians represent an opportunity for interventions that both reduce variation from known effective health care approaches and increase variability that personalizes care. Complexity science can help us to look for the inter-relationships among these different ways of knowing and to recognize what is not knowable or controllable. Yet, complexity science represents only a partial answer to efforts to integrate these diverse perspectives. There is a need for additional theoretical work to develop approaches that both include and transcend current ways of thinking.^52,53

Family practices are systems co-evolving within fitness landscapes where there is a continual need for sensemaking and improvisation. This is particularly true during the current period of rapid change and co-evolution of practices with a rapidly changing health care system. Excessive standardization with the goal of trying to maximize each part is as potentially problematic⁵¹ as variation from scientific evidence. Like a healthy body, a healthy practice represents a balance of the generalizable and the particular. The result is tension between the local, the regulatory, and the universal—and between patient, professional, societal, and ecological expectations. We believe the principles of complexity science explain why linear quality improvement interventions (one disease at a time) often have limited effect and poor transportability.^{15,16,19,54-56} These principles may also explain why countries with a higher proportion of primary care services have better population health status^36,57,58 despite the repeated observation that specialists do better at following disease guidelines and improving disease specific outcomes.^59-61 It is never just about the specific; it is about the specific in relation to the whole, and the whole is always more than the sum of the specifics. Good primary care serves all members of the community well with the resources available. Further application of complexity science to understand these paradoxes will require more quality longitudinal data in multiple practices and broad integrative measures of the process and outcomes of care.^51,62

Conclusions

Family physicians are told to implement guidelines, to diagnose and treat in specific ways, and to eliminate variation in practice. Our study using complexity science suggests that this is only part of the story. Family practices are systems that self-organize, reveal emergent behavior, and co-evolve. Successful practices are those that minimize errors, make good sense of what is happening, and effectively improvise to make good practice jazz. Seeking to eliminate error by dampening all variation through the imposition of excessive standardization and external controls is unlikely to be sustainably effective and is likely to have long-term negative consequences. We encourage all family practice staff members to become knowledgeable of practice guidelines and evidence-based practice; these are some of the core skills of good patient care.⁶³ Using these core skills to implement flexible, locally meaningful systems may reduce error. Also, efforts to change and improve future practice are best served by focusing on improving care as a whole and on developing the skills of reflective practice and relationship-centered care.⁵¹ We encourage policymakers to acknowledge the potential benefits of some kinds of variation and to support its healthy evolution.

Acknowledgments

The data used in our paper came from studies supported by a grant (1RO1 HS08776) from the Agency for Health Care Policy and Research (now the Agency for Healthcare Research and Quality) and grants from the National Cancer Institute (1RO1 CA60862 and 2RO1 CA60862). A A grant from the Center for Research in Family Practice and Primary Care, Cleveland, New Brunswick, Allentown, and San Antonio, supported communications, analyses, and writing. We are grateful to the practices participating in the research on which our paper was based.

One major focus of health services research and quality improvement efforts is to identify and reduce variation.^1-4 Standardization is the approach usually offered to minimize variation, thus reducing errors and increasing quality.^5,6 These interventions are often based on an industrial quality improvement paradigm⁷ using linear interventions that assume that inputs reliably lead to proportionate responses.⁸ These interventions include re-engineering and expanded information systems.^9,10 If the application of linear Newtonian views is correct, then standardization is the key to quality improvement, and effective practices will look much alike. The search for and attempt to implement best practice guidelines^11-14 are examples of efforts to bring practices into conformity and to establish process standards for best behavior. However, the search for simple, easily transportable interventions has not been as successful as traditional logic might suggest.^15-19

Emerging views of organizations derived from complexity science bring the key understanding that practices are more than commodity-delivering businesses—they are complex adaptive systems.²⁰ These systems involve connected participants interacting in ways that generate the spontaneous emergence of new structures and behaviors. In complex adaptive systems, we expect to see variation in practice patterns, even when the outcomes of practices are similar.

In a previous issue of JFP,²¹ we proposed a model of primary care practices as complex adaptive systems and suggested implications and strategies for change. Since then we have begun applying this theoretical framework to other studies designed to understand and advance generalist practice. Our present purpose is to advance the application of complexity science to understanding and improving primary care practices and their co-evolving health care systems.

The theory application process

The 3 studies that this theory application builds on began with the Direct Observation of Primary Care (DOPC) study, a 3-year (1994-1997) multimethod descriptive investigation of the content of 4454 patient visits to 138 family physicians in 84 family practices.^22,23 One of the outcomes of this study was a model for understanding change in family practice based on complexity science.²¹ Subsequently, The Prevention and Competing Demands in Primary Care Study (PCDPC) was a 3-year (1996-1999) in-depth descriptive case study of 1637 outpatient visits to 56 clinicians in18 family practices purposefully sampled to include diversity in geographic location, size of practice, and intensity of delivery of preventive services.²⁴ We also implemented a 4-year (1997-2001) multimethod clinical trial, the Study to Enhance Prevention by Understanding and Practice (STEP-UP), to understand and improve the delivery of preventive health services in 77 family practices.²⁵

We conducted an explicit theory application and refinement process consisting of 10 meetings to analyze data from DOPC, PCDPC, and STEP-UP, informed by a literature review of complexity science. The developing model was explicitly tested in 3 Nebraska family practices in 1999 in an effort to improve diabetes care.²⁶ The resulting specific theory application to family practices was then evaluated using 2 cases from these studies.

Application of complexity science to family practice

Synthesizing our observations of the family practices and the literature review, we developed the following theoretical model. Family practices are local professional complex adaptive systems with the primary purpose of seeing patients for everyday health concerns to assist “them” in getting on with their daily lives. “Them” refers to the patients and their families, the clinicians, and the office staff. Practice leaders and managers usually describe their practices in terms of efficiency, productivity, adherence to standards of care, and patient satisfaction. Increasingly, practices function in interactive ways with managers or owners from local health care systems. Still, these practices behaved more like complex adaptive systems operating within a professional milieu than like businesses.²⁷

Complex adaptive systems are like a family reunion; they are dynamic-bounded webs of diverse agents interacting nonlinearly.²⁸ Dynamic refers to the continual presence of multiple interactions and their accompanying surprises, challenges, and responses, both within the system and between the system (eg, practices) and its environment. Bounded refers to the defining purpose or intent of the system (eg, to deliver health care to local patients). The metaphor of a web characterizes the multiple interconnections of the system. The agents in practices include clinicians, office staff, and patients, and can also include pharmaceutical representatives, health care system administrators, and others. Agents have the capacity to exchange information, learn, and adjust their behavior. No individual agent can ever know or understand everything that is occurring. The nonlinear relationships among the agents are the result of ongoing feedback loops and mean that small changes can lead to large effects and big changes can lead to small effects. For example, the introduction of a small medical record stamp to identify smokers in one practice leads to a dramatic increase in smoking counseling, while a major quality improvement program in another practice results in minimal change.

What makes an organization “professional” is the application of specialized values and expertise to address difficult problems and uncertainty. These values and skills are acquired through specialized training and are created by the larger social context.²⁹ It is the socially defined professional values and expertise of the family physician that are applied in their daily activities. A co-evolving professional world of the health system and payer manager is increasingly interacting with the physician professional value system.

Each family practice is unique because of 5 features:

1. History and initial conditions,including any explicit or implicit mission and the underlying priorities for the practice.
2. Particular agents and their unique styles and interests.
3. The pattern of nonlinear interactions among agents.
4. The local fitness landscape(ie, the practice’s ecological niche) and its particular expectations, community values, competitive issues, and ecology.
5. Regional and global influences, such as larger health care systems, finances and regulations, and culture.

The local fitness landscape, a complexity science term from evolutionary biology, specifically refers to the local terrain and all the many complex adaptive systems, from microbes to organizations, that seek their own purpose and niche within that terrain. Biological evolution and technological evolution are processes attempting to optimize systems riddled with conflicting constraints.³⁰ Each family practice must evolve by attempting to optimize the entire package of services it delivers. This evolution must account for all the other competing and cooperating health care services and local resources, such as economic conditions, availability of insurance types, and the particular local disease and illness epidemiology.

How all of these factors manifest in a particular family practice at any given time and over time can be understood using 3 complexity science properties: self-organization, emergence, and co-evolution. Self-organization refers to the spontaneous development of structures and forms of behavior in systems characterized by multiple feedback loops and nonlinear dynamics. These structures are a function of the patterns of relationships among agents. Everything changes in response to and as a result of everything else, with each complex adaptive system seeking a better position in its local fitness landscape—a niche where it can prosper and survive.³¹ In this setting of ongoing co-evolution, both competition and collaboration become strategies for workable solutions. As the agents of any complex adaptive system interact, novelty and surprise continuously emerge in unpredictable ways. For example, a new successful approach to scheduling is introduced by an unassuming receptionist. This emergence creates a system that is greater than the sum of its parts; it is what cannot be understood through a reductionist (one problem at a time) examination of the practice.³²

A particular family practice is the unique self-organized system that emerges when particular physicians and staff (agents) come together in particular ways with particular goals, preferences, and priorities (initial conditions) within a particular community setting (local fitness landscape) given specific regional and global influences. At any future point, this practice is the unique self-organized system that has emerged through co-evolution with all the other systems in the local fitness landscape.

The result is much variation between and within family practices. Practices have much in common, however, because of their common goal of seeing patients to assist them with their everyday health problems in a shared cultural and historical context. From that perspective, variation in family practices is inevitable and a powerful source of creative possibility, value, and good clinical practice. Practices use 2 strategies for successfully enhancing that creativity: sensemaking and improvisation. Sensemaking is a social activity that requires interaction among agents.^33,34 People must come to have some notion of “Who am I,” “Why am I here,” and “What is going on around me?” Improvisation is a strategy for dealing with surprise in complex adaptive systems. Improvisation can be described as intuition guiding action in a spontaneous way.³⁵ Intuition is not a random guess at what to do, but the result of using high levels of expertise to act in the moment.^27,36

Among the many practices in the studies we are currently analyzing, multiple areas of variation are observed. These include differences in charting systems, clinical care decisions, scheduling, billing and coding procedures, staff relationships, and management and clinical styles. Sometimes these variations provide an adaptive advantage, but often not, and it is seldom clear in advance which will be true. Inflexible standardization, however, is often poorly responsive to the needs of different practices’ diverse agents and to the almost constant situations of uncertainty, contextual uniqueness, and surprise that occur in the practices.

Case studies

To illustrate the application of complexity science–based sensemaking to family practice, we present 2 case studies.

We selected 2 practices that had high quality of care as measured by delivery of preventive health services and patient satisfaction. One case takes advantage of the longitudinal data from DOPC and STEP-UP, and the other makes use of the more in-depth cross-sectional data from PCDPC. The cases were also selected to assure maximum variation in location and affiliation, and homogeneity in practice size (4-8 clinicians). The names have been changed to protect confidentiality.

Franchise Family Practice

History/Initial Conditions
Franchise is one of several primary care offices created by the Health Salute Corporation in affluent suburban areas of intense competition for market share growth. The corporate intent for this practice is to be productive and profitable. Two family physicians, a pediatrician, and a nurse practitioner were brought in from other practices, and several of their staff members followed. They all agree that their mission is to be the best practice in the Health Salute Corporation. Their identity is to capture market share through better efficiency, a mechanistic approach (scientific and standardized), and a friendly and caring attitude.

Agents and Patterns of Interaction
The practice manager’s daily attire in stockings and heels sets the tone for interactions, which are formal and professional. A small core of staff is dedicated to this practice, but there is often temporary help from other Health Salute offices during busy times. The patient population of predominantly mobile, insured, 2 working parent families tends to value convenience over relationships. The physicians seem to have little emotional investment in this particular practice, place, or each other. Conflicts are minimized and usually covered over with humor.

Local Fitness Landscape
There is no clear sense of community in this new suburb. Franchise is located in the heart of “minivan land,” an unrolling suburban carpet. The 2 competing systems are a major threat and are constantly being discussed. It is very clear that the survival of Franchise is dependent on success in the marketplace as determined by Health Salute.

Regional/Global Influences
Managed care has a strong presence, with much pressure to implement multiple practice guidelines, frequent chart audits, and different formularies.

Self-Organization
In many respects, Franchise Family Practice comes close to fulfilling its mission. Franchise is friendly, fun loving, and clean. It is a high performer at delivering preventive health services and is full of glitz and protocols. There are multiple systems in place for all phases of practice operation, and the manager sees they are working.

Emergence
Despite this managed order, surprises, problems, uncertainty, and complexities keep arising on a daily basis. Occasionally individuals respond creatively, but more often they stick to the protocols and generate even more trouble. There are frequent staff meetings where common problems are discussed. Many different solutions emerge in these discussions, but the final resolution is usually based on what the practice management thinks Health Salute would want. Even in this intensely structured practice, multiple competing demands, power distributions, and interpersonal battles are being simultaneously worked out on a daily basis.

Co-Evolution
As the suburbs grew, more practices were opened. The original practice in the area was soon challenged by Franchise and then by another competitor. Each of the 3 practices often acted or reacted in response to the others. Approximately a year after our research ended, Franchise Family Practice was closed by Health Salute because of inadequate profitability, and within a few months the second competitor also closed its practice.

Dusty Garden Family Practice

History/Initial Conditions
Dusty Garden began as a pioneering model for community-oriented primary care in an economically impoverished urban area. The practice was created with a focus on the patient in this underserved community. Envisioned by its founding family physician and practice manager, this practice was established in close collaboration with a community board. Survival is dependent on the ability to obtain funding for many poorly reimbursed services.

Agents and Patterns of Interaction
Dusty Garden has a dense and diverse web of complex interdependence. During the first few years of our research, most practice staff members came from the community. The practice grew from 4 to 6 family physicians and 2 nurse practitioners, and there was also much staff turnover. Dusty Garden was often a stepping stone for some clinicians, a chance to work in an “idealistic place” before going on to other things. However, the leadership has remained stable.

Local Fitness Landscape
The health care needs in the community were great, and the practice responded by growing rapidly, at times exceeding resources. At the same time, the local market consolidated into 2 competing hospital systems, with nearly all practices officially aligned with one of them. Necessary external funding also became more difficult to obtain.

Regional/Global Influences
Patients were represented by a mix of insurers. Uninsured or underinsured patients were cared for under a sliding-scale reimbursement scheme. There was also a perceived need to demonstrate successes and quality of care, and to place more emphasis on productivity and efficiency.

Self-Organization
The original practice was located in a dusty and cluttered building. It was difficult to tell who was responsible for what, but a shared sense of purpose gave the practice a family feel. Conflicts were evident but quickly resolved by frank discussions and a shared commitment to the practice mission. Schedules were constantly being disrupted by responding to patients and staff members’ diverse needs. In spite of this seeming chaos, Dusty Garden was an exemplar at delivering preventive services. This was accomplished by several dedicated clinicians and practice systems that involved the active participation of multiple personnel.

Co-Evolution
Significant change was occurring, and the practice was pushed to divide or grow in response to increasing patient demand and community need. They chose to grow and move into a much larger, newer, and more functional building down the street, resulting in a set of unanticipated consequences as both they and the fitness landscape changed. The new facility was more accessible and visible to a demographically different set of patients. Because of the changes in the local health care system, Dusty Garden also felt compelled to develop a relationship with the academic hospital system.

Emergence
What emerged was an organization where staff were isolated in large functionally differentiated spaces. The greater practice size demanded more specialization, and patterns of relationships dramatically changed. The change altered the number and specifics of the agents; many of the community-based staff left; and there was frequent turnover among newly hired and overwhelmed front office staff. The change also altered the number and character of the interactions per agent. Still, the vision remains strong, and many meetings are occurring to restore a “new” sense of practice community. Preventive service delivery rates remain high. The leadership initially responded to these changes with efforts aimed at greater standardization, but these are now being balanced by paying more attention to what solutions are being improvised “on the ground.”

Case study analysis

A comparative analysis of these 2 cases provides the following insights based on a complexity science view of the world:

1. Each practice was performing well using the delivery of preventive health and patient satisfaction as proxies for total practice performance.
2. The practices differed from each other in critical ways that seem to be at odds with traditional “best practices” thinking.
3. The practices were similar in that they had each organized themselves, coevolved, and emerged as a function of the nonlinear interdependencies among agents and the local fitness landscape, not solely as a function of some externally imposed script.
4. Each practice engaged in sensemaking activity to understand its unfolding world.
5. Each practice engaged in improvisational behavior as a strategy for developing strategic and tactical responses to its unfolding world.
6. Variation was often a source of strength, not a sign of bad practice.

Discussion

The traditional,^2-4 largely unsubstantiated,^37-40 view is that the best way to improve care is to eliminate variation. A view of family practice informed by complexity science suggests otherwise. In complex adaptive systems, agents in the practices create responses to changing circumstances—they improvise, or play practice jazz. Jazz players are often seen as role models of sensemaking and improvisational behavior.²⁸ They know a general musical structure, and within that they create jazz. Bad jazz occurs when one person plays what the others cannot make sense of and build on. All the players have an interdependent responsibility to create good jazz. When good jazz players hear something unexpected, they make sense of it and improvise. Dealing with the uncertain nature of complex adaptive systems involves thinking in terms of making sense of what is emerging. How can I improvise to use whatever happens to further the system’s development? It involves building on emergent characteristics of the complex adaptive system to develop patterns of social interaction⁴¹ among agents that give them confidence in each other, lead to small wins, and enhance the capacity to learn from unpredicted events.⁴²

Nevertheless, differentiating desirable from undesirable variation is an opportunity to learn from our history, and an opportunity to improve our practice jazz.^37,41,43 Small changes can have large results in some settings, while large efforts may lead to meager results in others. Complexity theory offers a framework for understanding these phenomena in family practice, and lays the groundwork for future research. On the basis of the proposed theoretical model, we hypothesize that it is critical to differentiate the variations that are sources of error from the variations due to the dynamics of relationships. From the perspective of complexity science, relational variation is linked to diversity among agents and represents constructive and adaptive variations and emergent behavior within an ever-changing and unpredictable local fitness landscape. From this perspective, the goals are to eliminate error through development of better systems of operation and to reduce confusion and poor judgment by improving sensemaking and communication. It is also important to enhance desirable variation by developing the skills of the relationship-centered clinical method,^44-46 improvisation, and reflective practice.^47-50

Sensemaking may be enhanced by considering the 4 ways of knowing health and health care,⁵¹ which include understanding: (1) the clinician; (2) the patient, family, and community; (3) systems; and (4) scientific evidence about disease and treatment. Judgments about the variation can be made within each way of knowing. Desirable variation due to clinician factors should build on each clinician’s unique skills and values, and compensate for or improve weaknesses. Local adaptation of objective evidence and the development of unique approaches to meeting the needs of patients in their personal and local context represent potentially desirable sources of variation. Evidence-based medicine provides a basis for reducing variation on the basis of scientific knowledge developed from studies of groups of individuals. Systems that integrate scientific evidence with the unique needs of patients, families, and communities and the specific talents of clinicians represent an opportunity for interventions that both reduce variation from known effective health care approaches and increase variability that personalizes care. Complexity science can help us to look for the inter-relationships among these different ways of knowing and to recognize what is not knowable or controllable. Yet, complexity science represents only a partial answer to efforts to integrate these diverse perspectives. There is a need for additional theoretical work to develop approaches that both include and transcend current ways of thinking.^52,53

Family practices are systems co-evolving within fitness landscapes where there is a continual need for sensemaking and improvisation. This is particularly true during the current period of rapid change and co-evolution of practices with a rapidly changing health care system. Excessive standardization with the goal of trying to maximize each part is as potentially problematic⁵¹ as variation from scientific evidence. Like a healthy body, a healthy practice represents a balance of the generalizable and the particular. The result is tension between the local, the regulatory, and the universal—and between patient, professional, societal, and ecological expectations. We believe the principles of complexity science explain why linear quality improvement interventions (one disease at a time) often have limited effect and poor transportability.^{15,16,19,54-56} These principles may also explain why countries with a higher proportion of primary care services have better population health status^36,57,58 despite the repeated observation that specialists do better at following disease guidelines and improving disease specific outcomes.^59-61 It is never just about the specific; it is about the specific in relation to the whole, and the whole is always more than the sum of the specifics. Good primary care serves all members of the community well with the resources available. Further application of complexity science to understand these paradoxes will require more quality longitudinal data in multiple practices and broad integrative measures of the process and outcomes of care.^51,62

Conclusions

Family physicians are told to implement guidelines, to diagnose and treat in specific ways, and to eliminate variation in practice. Our study using complexity science suggests that this is only part of the story. Family practices are systems that self-organize, reveal emergent behavior, and co-evolve. Successful practices are those that minimize errors, make good sense of what is happening, and effectively improvise to make good practice jazz. Seeking to eliminate error by dampening all variation through the imposition of excessive standardization and external controls is unlikely to be sustainably effective and is likely to have long-term negative consequences. We encourage all family practice staff members to become knowledgeable of practice guidelines and evidence-based practice; these are some of the core skills of good patient care.⁶³ Using these core skills to implement flexible, locally meaningful systems may reduce error. Also, efforts to change and improve future practice are best served by focusing on improving care as a whole and on developing the skills of reflective practice and relationship-centered care.⁵¹ We encourage policymakers to acknowledge the potential benefits of some kinds of variation and to support its healthy evolution.

Acknowledgments

The data used in our paper came from studies supported by a grant (1RO1 HS08776) from the Agency for Health Care Policy and Research (now the Agency for Healthcare Research and Quality) and grants from the National Cancer Institute (1RO1 CA60862 and 2RO1 CA60862). A A grant from the Center for Research in Family Practice and Primary Care, Cleveland, New Brunswick, Allentown, and San Antonio, supported communications, analyses, and writing. We are grateful to the practices participating in the research on which our paper was based.

One major focus of health services research and quality improvement efforts is to identify and reduce variation.^1-4 Standardization is the approach usually offered to minimize variation, thus reducing errors and increasing quality.^5,6 These interventions are often based on an industrial quality improvement paradigm⁷ using linear interventions that assume that inputs reliably lead to proportionate responses.⁸ These interventions include re-engineering and expanded information systems.^9,10 If the application of linear Newtonian views is correct, then standardization is the key to quality improvement, and effective practices will look much alike. The search for and attempt to implement best practice guidelines^11-14 are examples of efforts to bring practices into conformity and to establish process standards for best behavior. However, the search for simple, easily transportable interventions has not been as successful as traditional logic might suggest.^15-19

Emerging views of organizations derived from complexity science bring the key understanding that practices are more than commodity-delivering businesses—they are complex adaptive systems.²⁰ These systems involve connected participants interacting in ways that generate the spontaneous emergence of new structures and behaviors. In complex adaptive systems, we expect to see variation in practice patterns, even when the outcomes of practices are similar.

In a previous issue of JFP,²¹ we proposed a model of primary care practices as complex adaptive systems and suggested implications and strategies for change. Since then we have begun applying this theoretical framework to other studies designed to understand and advance generalist practice. Our present purpose is to advance the application of complexity science to understanding and improving primary care practices and their co-evolving health care systems.

The theory application process

The 3 studies that this theory application builds on began with the Direct Observation of Primary Care (DOPC) study, a 3-year (1994-1997) multimethod descriptive investigation of the content of 4454 patient visits to 138 family physicians in 84 family practices.^22,23 One of the outcomes of this study was a model for understanding change in family practice based on complexity science.²¹ Subsequently, The Prevention and Competing Demands in Primary Care Study (PCDPC) was a 3-year (1996-1999) in-depth descriptive case study of 1637 outpatient visits to 56 clinicians in18 family practices purposefully sampled to include diversity in geographic location, size of practice, and intensity of delivery of preventive services.²⁴ We also implemented a 4-year (1997-2001) multimethod clinical trial, the Study to Enhance Prevention by Understanding and Practice (STEP-UP), to understand and improve the delivery of preventive health services in 77 family practices.²⁵

We conducted an explicit theory application and refinement process consisting of 10 meetings to analyze data from DOPC, PCDPC, and STEP-UP, informed by a literature review of complexity science. The developing model was explicitly tested in 3 Nebraska family practices in 1999 in an effort to improve diabetes care.²⁶ The resulting specific theory application to family practices was then evaluated using 2 cases from these studies.

Application of complexity science to family practice

Synthesizing our observations of the family practices and the literature review, we developed the following theoretical model. Family practices are local professional complex adaptive systems with the primary purpose of seeing patients for everyday health concerns to assist “them” in getting on with their daily lives. “Them” refers to the patients and their families, the clinicians, and the office staff. Practice leaders and managers usually describe their practices in terms of efficiency, productivity, adherence to standards of care, and patient satisfaction. Increasingly, practices function in interactive ways with managers or owners from local health care systems. Still, these practices behaved more like complex adaptive systems operating within a professional milieu than like businesses.²⁷

Complex adaptive systems are like a family reunion; they are dynamic-bounded webs of diverse agents interacting nonlinearly.²⁸ Dynamic refers to the continual presence of multiple interactions and their accompanying surprises, challenges, and responses, both within the system and between the system (eg, practices) and its environment. Bounded refers to the defining purpose or intent of the system (eg, to deliver health care to local patients). The metaphor of a web characterizes the multiple interconnections of the system. The agents in practices include clinicians, office staff, and patients, and can also include pharmaceutical representatives, health care system administrators, and others. Agents have the capacity to exchange information, learn, and adjust their behavior. No individual agent can ever know or understand everything that is occurring. The nonlinear relationships among the agents are the result of ongoing feedback loops and mean that small changes can lead to large effects and big changes can lead to small effects. For example, the introduction of a small medical record stamp to identify smokers in one practice leads to a dramatic increase in smoking counseling, while a major quality improvement program in another practice results in minimal change.

What makes an organization “professional” is the application of specialized values and expertise to address difficult problems and uncertainty. These values and skills are acquired through specialized training and are created by the larger social context.²⁹ It is the socially defined professional values and expertise of the family physician that are applied in their daily activities. A co-evolving professional world of the health system and payer manager is increasingly interacting with the physician professional value system.

Each family practice is unique because of 5 features:

1. History and initial conditions,including any explicit or implicit mission and the underlying priorities for the practice.
2. Particular agents and their unique styles and interests.
3. The pattern of nonlinear interactions among agents.
4. The local fitness landscape(ie, the practice’s ecological niche) and its particular expectations, community values, competitive issues, and ecology.
5. Regional and global influences, such as larger health care systems, finances and regulations, and culture.

The local fitness landscape, a complexity science term from evolutionary biology, specifically refers to the local terrain and all the many complex adaptive systems, from microbes to organizations, that seek their own purpose and niche within that terrain. Biological evolution and technological evolution are processes attempting to optimize systems riddled with conflicting constraints.³⁰ Each family practice must evolve by attempting to optimize the entire package of services it delivers. This evolution must account for all the other competing and cooperating health care services and local resources, such as economic conditions, availability of insurance types, and the particular local disease and illness epidemiology.

How all of these factors manifest in a particular family practice at any given time and over time can be understood using 3 complexity science properties: self-organization, emergence, and co-evolution. Self-organization refers to the spontaneous development of structures and forms of behavior in systems characterized by multiple feedback loops and nonlinear dynamics. These structures are a function of the patterns of relationships among agents. Everything changes in response to and as a result of everything else, with each complex adaptive system seeking a better position in its local fitness landscape—a niche where it can prosper and survive.³¹ In this setting of ongoing co-evolution, both competition and collaboration become strategies for workable solutions. As the agents of any complex adaptive system interact, novelty and surprise continuously emerge in unpredictable ways. For example, a new successful approach to scheduling is introduced by an unassuming receptionist. This emergence creates a system that is greater than the sum of its parts; it is what cannot be understood through a reductionist (one problem at a time) examination of the practice.³²

A particular family practice is the unique self-organized system that emerges when particular physicians and staff (agents) come together in particular ways with particular goals, preferences, and priorities (initial conditions) within a particular community setting (local fitness landscape) given specific regional and global influences. At any future point, this practice is the unique self-organized system that has emerged through co-evolution with all the other systems in the local fitness landscape.

The result is much variation between and within family practices. Practices have much in common, however, because of their common goal of seeing patients to assist them with their everyday health problems in a shared cultural and historical context. From that perspective, variation in family practices is inevitable and a powerful source of creative possibility, value, and good clinical practice. Practices use 2 strategies for successfully enhancing that creativity: sensemaking and improvisation. Sensemaking is a social activity that requires interaction among agents.^33,34 People must come to have some notion of “Who am I,” “Why am I here,” and “What is going on around me?” Improvisation is a strategy for dealing with surprise in complex adaptive systems. Improvisation can be described as intuition guiding action in a spontaneous way.³⁵ Intuition is not a random guess at what to do, but the result of using high levels of expertise to act in the moment.^27,36

Among the many practices in the studies we are currently analyzing, multiple areas of variation are observed. These include differences in charting systems, clinical care decisions, scheduling, billing and coding procedures, staff relationships, and management and clinical styles. Sometimes these variations provide an adaptive advantage, but often not, and it is seldom clear in advance which will be true. Inflexible standardization, however, is often poorly responsive to the needs of different practices’ diverse agents and to the almost constant situations of uncertainty, contextual uniqueness, and surprise that occur in the practices.

Case studies

To illustrate the application of complexity science–based sensemaking to family practice, we present 2 case studies.

We selected 2 practices that had high quality of care as measured by delivery of preventive health services and patient satisfaction. One case takes advantage of the longitudinal data from DOPC and STEP-UP, and the other makes use of the more in-depth cross-sectional data from PCDPC. The cases were also selected to assure maximum variation in location and affiliation, and homogeneity in practice size (4-8 clinicians). The names have been changed to protect confidentiality.

Franchise Family Practice

History/Initial Conditions
Franchise is one of several primary care offices created by the Health Salute Corporation in affluent suburban areas of intense competition for market share growth. The corporate intent for this practice is to be productive and profitable. Two family physicians, a pediatrician, and a nurse practitioner were brought in from other practices, and several of their staff members followed. They all agree that their mission is to be the best practice in the Health Salute Corporation. Their identity is to capture market share through better efficiency, a mechanistic approach (scientific and standardized), and a friendly and caring attitude.

Agents and Patterns of Interaction
The practice manager’s daily attire in stockings and heels sets the tone for interactions, which are formal and professional. A small core of staff is dedicated to this practice, but there is often temporary help from other Health Salute offices during busy times. The patient population of predominantly mobile, insured, 2 working parent families tends to value convenience over relationships. The physicians seem to have little emotional investment in this particular practice, place, or each other. Conflicts are minimized and usually covered over with humor.

Local Fitness Landscape
There is no clear sense of community in this new suburb. Franchise is located in the heart of “minivan land,” an unrolling suburban carpet. The 2 competing systems are a major threat and are constantly being discussed. It is very clear that the survival of Franchise is dependent on success in the marketplace as determined by Health Salute.

Regional/Global Influences
Managed care has a strong presence, with much pressure to implement multiple practice guidelines, frequent chart audits, and different formularies.

Self-Organization
In many respects, Franchise Family Practice comes close to fulfilling its mission. Franchise is friendly, fun loving, and clean. It is a high performer at delivering preventive health services and is full of glitz and protocols. There are multiple systems in place for all phases of practice operation, and the manager sees they are working.

Emergence
Despite this managed order, surprises, problems, uncertainty, and complexities keep arising on a daily basis. Occasionally individuals respond creatively, but more often they stick to the protocols and generate even more trouble. There are frequent staff meetings where common problems are discussed. Many different solutions emerge in these discussions, but the final resolution is usually based on what the practice management thinks Health Salute would want. Even in this intensely structured practice, multiple competing demands, power distributions, and interpersonal battles are being simultaneously worked out on a daily basis.

Co-Evolution
As the suburbs grew, more practices were opened. The original practice in the area was soon challenged by Franchise and then by another competitor. Each of the 3 practices often acted or reacted in response to the others. Approximately a year after our research ended, Franchise Family Practice was closed by Health Salute because of inadequate profitability, and within a few months the second competitor also closed its practice.

Dusty Garden Family Practice

History/Initial Conditions
Dusty Garden began as a pioneering model for community-oriented primary care in an economically impoverished urban area. The practice was created with a focus on the patient in this underserved community. Envisioned by its founding family physician and practice manager, this practice was established in close collaboration with a community board. Survival is dependent on the ability to obtain funding for many poorly reimbursed services.

Agents and Patterns of Interaction
Dusty Garden has a dense and diverse web of complex interdependence. During the first few years of our research, most practice staff members came from the community. The practice grew from 4 to 6 family physicians and 2 nurse practitioners, and there was also much staff turnover. Dusty Garden was often a stepping stone for some clinicians, a chance to work in an “idealistic place” before going on to other things. However, the leadership has remained stable.

Local Fitness Landscape
The health care needs in the community were great, and the practice responded by growing rapidly, at times exceeding resources. At the same time, the local market consolidated into 2 competing hospital systems, with nearly all practices officially aligned with one of them. Necessary external funding also became more difficult to obtain.

Regional/Global Influences
Patients were represented by a mix of insurers. Uninsured or underinsured patients were cared for under a sliding-scale reimbursement scheme. There was also a perceived need to demonstrate successes and quality of care, and to place more emphasis on productivity and efficiency.

Self-Organization
The original practice was located in a dusty and cluttered building. It was difficult to tell who was responsible for what, but a shared sense of purpose gave the practice a family feel. Conflicts were evident but quickly resolved by frank discussions and a shared commitment to the practice mission. Schedules were constantly being disrupted by responding to patients and staff members’ diverse needs. In spite of this seeming chaos, Dusty Garden was an exemplar at delivering preventive services. This was accomplished by several dedicated clinicians and practice systems that involved the active participation of multiple personnel.

Co-Evolution
Significant change was occurring, and the practice was pushed to divide or grow in response to increasing patient demand and community need. They chose to grow and move into a much larger, newer, and more functional building down the street, resulting in a set of unanticipated consequences as both they and the fitness landscape changed. The new facility was more accessible and visible to a demographically different set of patients. Because of the changes in the local health care system, Dusty Garden also felt compelled to develop a relationship with the academic hospital system.

Emergence
What emerged was an organization where staff were isolated in large functionally differentiated spaces. The greater practice size demanded more specialization, and patterns of relationships dramatically changed. The change altered the number and specifics of the agents; many of the community-based staff left; and there was frequent turnover among newly hired and overwhelmed front office staff. The change also altered the number and character of the interactions per agent. Still, the vision remains strong, and many meetings are occurring to restore a “new” sense of practice community. Preventive service delivery rates remain high. The leadership initially responded to these changes with efforts aimed at greater standardization, but these are now being balanced by paying more attention to what solutions are being improvised “on the ground.”

Case study analysis

A comparative analysis of these 2 cases provides the following insights based on a complexity science view of the world:

1. Each practice was performing well using the delivery of preventive health and patient satisfaction as proxies for total practice performance.
2. The practices differed from each other in critical ways that seem to be at odds with traditional “best practices” thinking.
3. The practices were similar in that they had each organized themselves, coevolved, and emerged as a function of the nonlinear interdependencies among agents and the local fitness landscape, not solely as a function of some externally imposed script.
4. Each practice engaged in sensemaking activity to understand its unfolding world.
5. Each practice engaged in improvisational behavior as a strategy for developing strategic and tactical responses to its unfolding world.
6. Variation was often a source of strength, not a sign of bad practice.

Discussion

The traditional,^2-4 largely unsubstantiated,^37-40 view is that the best way to improve care is to eliminate variation. A view of family practice informed by complexity science suggests otherwise. In complex adaptive systems, agents in the practices create responses to changing circumstances—they improvise, or play practice jazz. Jazz players are often seen as role models of sensemaking and improvisational behavior.²⁸ They know a general musical structure, and within that they create jazz. Bad jazz occurs when one person plays what the others cannot make sense of and build on. All the players have an interdependent responsibility to create good jazz. When good jazz players hear something unexpected, they make sense of it and improvise. Dealing with the uncertain nature of complex adaptive systems involves thinking in terms of making sense of what is emerging. How can I improvise to use whatever happens to further the system’s development? It involves building on emergent characteristics of the complex adaptive system to develop patterns of social interaction⁴¹ among agents that give them confidence in each other, lead to small wins, and enhance the capacity to learn from unpredicted events.⁴²

Nevertheless, differentiating desirable from undesirable variation is an opportunity to learn from our history, and an opportunity to improve our practice jazz.^37,41,43 Small changes can have large results in some settings, while large efforts may lead to meager results in others. Complexity theory offers a framework for understanding these phenomena in family practice, and lays the groundwork for future research. On the basis of the proposed theoretical model, we hypothesize that it is critical to differentiate the variations that are sources of error from the variations due to the dynamics of relationships. From the perspective of complexity science, relational variation is linked to diversity among agents and represents constructive and adaptive variations and emergent behavior within an ever-changing and unpredictable local fitness landscape. From this perspective, the goals are to eliminate error through development of better systems of operation and to reduce confusion and poor judgment by improving sensemaking and communication. It is also important to enhance desirable variation by developing the skills of the relationship-centered clinical method,^44-46 improvisation, and reflective practice.^47-50

Sensemaking may be enhanced by considering the 4 ways of knowing health and health care,⁵¹ which include understanding: (1) the clinician; (2) the patient, family, and community; (3) systems; and (4) scientific evidence about disease and treatment. Judgments about the variation can be made within each way of knowing. Desirable variation due to clinician factors should build on each clinician’s unique skills and values, and compensate for or improve weaknesses. Local adaptation of objective evidence and the development of unique approaches to meeting the needs of patients in their personal and local context represent potentially desirable sources of variation. Evidence-based medicine provides a basis for reducing variation on the basis of scientific knowledge developed from studies of groups of individuals. Systems that integrate scientific evidence with the unique needs of patients, families, and communities and the specific talents of clinicians represent an opportunity for interventions that both reduce variation from known effective health care approaches and increase variability that personalizes care. Complexity science can help us to look for the inter-relationships among these different ways of knowing and to recognize what is not knowable or controllable. Yet, complexity science represents only a partial answer to efforts to integrate these diverse perspectives. There is a need for additional theoretical work to develop approaches that both include and transcend current ways of thinking.^52,53

Family practices are systems co-evolving within fitness landscapes where there is a continual need for sensemaking and improvisation. This is particularly true during the current period of rapid change and co-evolution of practices with a rapidly changing health care system. Excessive standardization with the goal of trying to maximize each part is as potentially problematic⁵¹ as variation from scientific evidence. Like a healthy body, a healthy practice represents a balance of the generalizable and the particular. The result is tension between the local, the regulatory, and the universal—and between patient, professional, societal, and ecological expectations. We believe the principles of complexity science explain why linear quality improvement interventions (one disease at a time) often have limited effect and poor transportability.^{15,16,19,54-56} These principles may also explain why countries with a higher proportion of primary care services have better population health status^36,57,58 despite the repeated observation that specialists do better at following disease guidelines and improving disease specific outcomes.^59-61 It is never just about the specific; it is about the specific in relation to the whole, and the whole is always more than the sum of the specifics. Good primary care serves all members of the community well with the resources available. Further application of complexity science to understand these paradoxes will require more quality longitudinal data in multiple practices and broad integrative measures of the process and outcomes of care.^51,62

Conclusions

Family physicians are told to implement guidelines, to diagnose and treat in specific ways, and to eliminate variation in practice. Our study using complexity science suggests that this is only part of the story. Family practices are systems that self-organize, reveal emergent behavior, and co-evolve. Successful practices are those that minimize errors, make good sense of what is happening, and effectively improvise to make good practice jazz. Seeking to eliminate error by dampening all variation through the imposition of excessive standardization and external controls is unlikely to be sustainably effective and is likely to have long-term negative consequences. We encourage all family practice staff members to become knowledgeable of practice guidelines and evidence-based practice; these are some of the core skills of good patient care.⁶³ Using these core skills to implement flexible, locally meaningful systems may reduce error. Also, efforts to change and improve future practice are best served by focusing on improving care as a whole and on developing the skills of reflective practice and relationship-centered care.⁵¹ We encourage policymakers to acknowledge the potential benefits of some kinds of variation and to support its healthy evolution.

Acknowledgments

The data used in our paper came from studies supported by a grant (1RO1 HS08776) from the Agency for Health Care Policy and Research (now the Agency for Healthcare Research and Quality) and grants from the National Cancer Institute (1RO1 CA60862 and 2RO1 CA60862). A A grant from the Center for Research in Family Practice and Primary Care, Cleveland, New Brunswick, Allentown, and San Antonio, supported communications, analyses, and writing. We are grateful to the practices participating in the research on which our paper was based.

Primary care practices have been called America’s de facto mental health network, with more than two thirds of mental health disorders treated in the primary care sector.¹ Up to 40% of primary care patients have a mental health problem,² and 19% of outpatients report significant emotional distress during the previous 4 weeks.³ However, the detection and treatment rates of these problems are low.^3-6

Thus, although the clinical philosophy of primary care professionals suggests that mental health care is an integral part of practice,^7-9 there is an apparent discrepancy between these espoused ideals and usual clinical practice.^3,5,10-11 Explanations of these findings include the reluctance of primary care physicians to label their patients and their use of observation and informal counseling as initial treatment efforts.^11-13 The competing demands of practice, lack of resources, inadequate reimbursement, and various organizational factors such as mental health carve-outs also profoundly influence management.^14-16 Using cluster analysis, Roter and colleagues¹⁷ found 5 distinct communication patterns between patients with ongoing medical problems and their physicians, ranging from narrowly biomedical to consumerist.¹⁷ Robinson and Roter^18-19 found that patients are likely to respond to direct inquiry by physicians about psychosocial distress and that physicians often briefly counsel their patients in return. Callahan and coworkers³ demonstrated that recent emotional distress and mental health problems have an important impact on encounter activities (eg, more time on history taking and counseling). Despite these investigations, a robust model of physicians’ response to emotional distress remains incompletely characterized.

We sought to develop a typology of physicians’ reactions to and management of patients’ mental health problems and emotional distress. Our findings can help clinicians identify their own style and consider ways of meeting particular patient needs that may be better suited to an alternative approach.

Methods

Detailed descriptive field notes of outpatient visits were collected as part of a large multimethod comparative case study of 18 midwestern family practices. Trained field researchers spent 4 weeks or more in each practice and directly observed the practice environment and 30 outpatient visits with each clinician in the practice. While observing the outpatient visits, the field researcher took chronological notes of what was occurring during the encounters. These notes were later used to dictate detailed descriptions of each encounter. Although there were differences in the style of reporting among the observers, the quality of data was consistent. Details of the design and data collection can be found elsewhere in this issue.²⁰

Two family physician researchers, 3 family therapists, and a medical anthropologist reviewed encounters from a purposeful sample of family physicians. Initially, encounters from 3 physicians representing diverse practice approaches (as assessed globally by a research nurse collecting the primary data) were reviewed. The goals were to understand the depth and detail of the data and to develop initial hypotheses, an organizational schema, and a crude overview of the presentation of and physician response to mental health issues. The management of mental health issues and emotional distress was then explored in a purposeful sample of physicians selected to maximize variation in sex, type, and location of practice; ethnicity; and age. By the nature of this qualitative study (without access to an independent gold standard for diagnosis of mental disorders), a broad definition of mental health problems was used, encompassing emotional distress and psychological problems. On the basis of the preliminary review of field note data, the research group identified that patients were presenting with emotional issues when they found a reported change in affect, a verbal report of an emotional issue, a somatic complaint often associated with emotional distress, or a follow-up visit for an expressed mental health issue (eg, refill of an antidepressant). This working definition was reached in the preliminary phase of the study, and through discussion a consensus was reached on the mental health aspects of each encounter.

Physicians were our unit of analysis, and the authors reviewed every outpatient visit available from each of the 13 physicians selected from the larger sample. The research team members used an editing organizing style for analysis,²¹ individually highlighted text they believed to be relevant, and made interpretive notes or observations in the margins.²² The research team then engaged in detailed discussions of the encounter transcripts. Particular attention was given to the total context of the encounter, recognizing other potential competing demands within the visit. The goal of this lengthy process was to reach consensus about what was important and how it should be interpreted. After discussing every encounter of a given physician, a summary case narrative was prepared and consensus reached about key themes for that physician.

After completing this initial review, matrices (eg, variations in patient management by practice location, physician age, and sex) were constructed to visualize other emergent patterns and facilitate comparisons across cases.²³ Additional physicians were reviewed to search for confirming and disconfirming evidence (eg, did management vary by physician ethnicity?) until saturation was reached (ie, until no further novel information or themes were identified). This required the review of outpatient visits from 13 physicians. One of the primary research nurses who conducted the participant observation provided input that ensured a full diversity of physicians was considered. She also served as an additional check on interpretation of the primary data. Finally, overall theses common to all physicians were identified and important variations in management noted. Thus, we began by looking at individual physicians’ responses within each encounter, developed a coherent description of each physician’s modus operandi, and then identified overarching themes describing broad approaches to emotional distress and mental health issues.

Results

The 379 patient visits to 13 physicians represented a diverse sample of practice and encounter types (Table 1, Table 2). Although the chief complaints of many patients did not overtly appear to relate to a mental health condition or emotional distress, many patients’ emotional concerns presented within the context of an acute or chronic medical condition. All physicians had many encounters in which both overt and more covert emotional concerns and mental health issues emerged.

Physician Responses Within Encounters

The research team noted a wide range of physician reactions to patients presenting with emotional distress or potential mental health problems. During the physician-patient interaction, physicians apparently either recognized the emotional component of the encounter or did not. If emotional distress was recognized, physicians appeared to either actively ignore this problem, gloss over or triage it, or actively manage the distress. These phenomena are illustrated in Figure 1 and will be described in more detail.

Recognition
Not all emotional and mental health issues were apparently recognized. Such missed opportunities were identified with all the participant physicians, even among physicians who were consistently more attentive to addressing mental health problems. For example, during a follow-up visit with a middle-aged man with abdominal tenderness a computed tomography scan had disclosed a renal mass. The patient’s wife asked numerous questions about possible depression and anxiety in her alcohol-using husband. The physician did not pursue any of these concerns.

However, a minority of physicians actively asked about mental health problems. This “active case finding” often capitalized on the physician’s previous knowledge of the patient’s social situation or personal issues. In one encounter focusing on breast cancer follow-up, the physician asked a woman how she was interacting with her spouse after a mastectomy. In instances such as this, active case finding was part of the chatting that opened or ended an encounter, particularly among physician and patients who were familiar with each other. Physicians in this sample neither used screening instruments (eg, the Primary Care Evaluation of Mental Disorders or the Zung mental health scales) nor routinely inquired about suicidal ideation, even in their seemingly most severely depressed patients.

Gloss-Over/Triage
In some instances, the physician apparently understood the impact of a situation but seemed to gloss over the issues. During a health care maintenance visit, a woman reported that she had a miscarriage 3 months earlier. The physician asked, “Is this a good thing or a sad thing?” The patient stated that it was a sad thing, because they were looking forward to the birth. There was no further probing into how the patient and family were dealing with the miscarriage.

In other encounters, physicians clearly seemed to recognize the psychologic implications of an encounter but chose to postpone management. These physicians appeared to triage certain cases based on time, competing demands, or perhaps their own ability to weather another challenging patient. For example, in the case of a patient with arm pain seeking workers’ compensation, the patient noted, “the pain (after doing some minor chores) was simply not worth it.” The physician did not pursue this cue further, but rather concentrated on the scheduling of magnetic resonance imaging, an electromyogram, and a follow-up appointment. However, the physician later related to the nurse researcher her understanding of the impact of this problem and acknowledged the patient’s discomfort. Thus, this physician apparently “triaged” this issue to a later date.

Management
Although only a minority of physicians actively sought cases of emotional distress in these encounters, most actively managed mental health problems. Prompted by the patient’s presentation, physicians followed up on “leads” to potential mental health issues, including: a mother who discussed the death of her daughter, a woman with menstrual irregularity, and marital and financial stress. Such encounters demonstrated physicians being sensitive to the underlying psychosocial issues in their patients’ lives.

The management response appeared to be predicated on the physician’s philosophy (biomedical vs holistic) and skill level (basic vs more advanced). In some instances, physicians appeared to spend considerable time on mental health issues with patients but apparently ran out of tools to deal with their problems effectively. This situation was most evident with patients who were substance abusers, who had chronic pain, those seeking workers’ compensation, and individuals with vague or multiple somatic symptoms.

A 4-Quadrant Typology of Physicians

A 4-quadrant typology of physicians emerged based on their philosophy and skill, as ascertained from the patient encounters Figure 2. Philosophically, physicians were on a continuum of being biomedically to biopsychosocially inclined, with each exhibiting a discernable dominant philosophy. Biomedically oriented physicians concentrated on the medical aspects of care and minimally explored the psychosocial milieu of the patient. Biopsychosocially oriented physicians addressed the patients’ emotional, physical, social, and sometimes spiritual wellbeing. Regardless of management approach (biomedical vs biopsychosocial), physicians demonstrated varying levels of competence in dealing with emotional distress.

Most physicians used “basic” skills—empathy, encouragement, small talk, use of silence, direct advice giving, and superficial education—to address their patients’ mental health problems. In some encounters the use of simple strategies was seemingly appropriate and effective; only occasionally were more advanced skills used. Such advanced skills ranged from effectively setting an agenda and soliciting the patient’s perspective to the use of more challenging interviewing skills, such as confrontation, implementing behavioral prescriptions, navigating referrals for skeptical patients, and mental health referrals that were part of a carefully developed treatment plan.

By combining the philosophy and skill dimensions, a 4-quadrant typology of physicians was apparent: the Technician, the Friend, the Detective, and the Healer. The Technician was medically oriented, dispensing medications and direct advice. Encounters were problem focused, and at times the physician appeared to be abrupt, ignorant of clear emotional distress, and not patient centered. In an encounter for follow-up of anxiety, one Technician told a patient complaining of neurologic symptoms that they might be stress related but still referred her to a neurologist. When she said, “This is really a frustrating way to feel,” he responded with, “Well, a neurologist deals with this,” and gave her samples of paroxetine, checked her for a sinus infection, and ended the encounter. Another patient seeing this physician for a complaint of depressive symptoms was identified without any discussion of underlying psychosocial issues; fluoxetine was dispensed in an encounter lasting less than 5 minutes.

The Friend was a biopsychosocially oriented physician with basic skills. One Friend extensively explored the patient’s background, concerns, and spiritual dimensions of illness. Encounters were long and tangential. A diverse array of topics was explored in a patient-centered fashion. However, only very basic counseling and management skills were ever observed with this physician. Direct advice was common, and conflict appeared to be avoided. A metaphor emerged of friends having coffee together.

Friends did not always appear to deliver care that optimally managed mental health issues. In some instances so many issues were discussed that the physician appeared to have difficulty setting an agenda for the visit and prioritizing problems. For example, for a patient just discharged after hospitalization for severe depression, there was no explicit discussion of depressive symptoms or suicidal ideation, despite a lengthy encounter.

The Detective was usually biomedically focused but when the occasion warranted, this type of physician demonstrated an impressive breadth of detective skills. For example, one Detective appeared most comfortable providing focused, snappy, medically oriented care. But she was alert to cues of emotional distress and demonstrated appropriate use of self-disclosure and confrontation in managing a patient with depression. In short, she was usually able to provide solutions for each case while focused on more biomedical issues.

The Healer used a full breadth of biopsychosocial skills, integrated most aspects of care seamlessly, and appeared comfortable with both strictly biomedical and psychosocial dimensions of care. One Healer regularly sought signs of emotional distress and exhibited an impressive range of skills in dealing with such problems as substance abuse and pain syndromes. For example, he astutely linked a patient’s stressful lifestyle with current somatic symptoms. In another encounter, with a woman with high blood pressure and weight gain, he assessed the possible biopsychosocial causes of the problem (etiologic stressors, sleep habits, relationship issues, diet changes, and depression, and probed about any anniversaries of a major stressor). However, even this Healer appeared to occasionally consciously temporize or triage emotional and mental health issues, such as when working with a patient with low back pain who was resistant to the treatment plan. During another encounter, he appeared to avoid the emotional implications of a diagnosis of venereal disease.

Thus, physicians addressed psychological problems in a variety of ways—from a strictly biomedical model to a more holistic fashion. Physicians also demonstrated a wide range of skills—from very basic to quite advanced, and applied these skills differently with different patients in different situations. Although a given provider’s performance often varied among encounters, most physicians appeared to have a preferred practice philosophy and singular skill set that they regularly used during patient visits.

Discussion

As in previous studies,^10-15 we found that not all physicians appeared comfortable, trained, adept, or motivated to make sense of the emotional distress presented by patients. Parallel to the findings of Roter and colleages¹⁷ with regard to general communication patterns of primary care physicians, a typology of physician responses to emotional distress emerged from our data. The framework of encounters (recognition, triage, and management) and 4-quadrant physician typology that surfaced from this study helps clarify how physicians respond to emotional distress. Each of the approaches in this typology is likely to have pros and cons for meeting different patient needs for mental health and general medical care.

Understanding physicians’ predominant styles based on their philosophy and skill set can have 2 important uses. First, physicians can reflect and seek feedback on their own style. Patient needs that may be less well met by this style can then be identified and alternate ways of meeting these needs pursued. Second, clinicians and continuing medical education providers can use this typology to design educational approaches. This education should focus on expanding clinician flexibility and increasing insight into when to use what approach. The outcomes and tradeoffs in effectiveness, efficiency, and integration of care remain important areas for future research.²⁴

Given the constraints on time, personal energy, and apparent competition between chronic physical and mental health problems, physician behaviors can be viewed as an understandable adaptation to the realities of a busy family practice.^11-15-24-28 Although we have documented significant variation in counseling skills among family physicians, there is no data to suggest that expansion of these skills would necessarily improve patient outcomes.²⁹ The effect of a long-term relationship and its quality between patients and a family physician on patient mental health outcomes remains unexplored and is a fruitful area for further research. Also, it is important to recognize that physicians are not homogeneous in their personality, philosophy, and skills and that patients self-select the kind of physician that best fits their own personality and style. Different approaches are likely to be functional for diverse clinicians with varied patients and situations.²⁴

Limitations

Our study has important limitations, including its sampling, design, and lack of a reference standard for mental health conditions. This qualitative research, by its very nature, is not based on a random population sample and is therefore not generalizable in the traditional quantitative sense. Its generalizability lies in the resonance it generates among primary care physicians and patients who recognize these patterns from their own experiences. Also, the findings are consistent with our existing understanding of competing demands^9,28,30 and physician communication strategies.^17-19 To the extent to which midwestern physicians and patients do not reflect the ethnic and socioeconomic diversity of other parts of the country, these findings may also be limited. Future research should attempt to include diversity. Patients’ emotional distress may be communicated in other ways besides speech or may not be communicated at all, so the direct observation approach we used cannot always correctly infer patients’ unexpressed mental health needs or physicians’ assessment of the situation. Because the data were cross-sectional, it is not possible to determine what had occurred in previous visits in a longitudinal management strategy. Nevertheless, the richness of the field note data provided an excellent detailed view of a large sample of visits. Finally, the lack of a reference standard for diagnosing mental health conditions does not alter the main findings of this study—a typology of physicians’ responses to emotional distress within their practices.

In trying to understand and improve the treatment of mental health issues, many previous researchers have focused on improving physician knowledge and dissemination of guidelines; such efforts have been disappointing when used alone.^31,32 Other investigators have sought to improve the interviewing skills of physicians, and while modestly successful, these studies have been limited in scope, length of follow-up, and ability to be replicated widely.^18,19,33,34 Other approaches have included collaborative management and quality improvement efforts; while successful, such interventions may be difficult to replicate in the usual physician practice setting without substantial external resources.^35-38

Conclusions

The chasm between ideal care of mental health disorders and actual practice may be narrower than mental health professionals would have us believe, and it is certainly bridgeable. It is possible to have better outcomes for medical conditions, improved patient and provider satisfaction, and reduced costs of care.^39,40 By studying the exemplary physicians found in real world practices—as found in this study and others—we might better understand that combination of inclination, skill, and setting that promotes quality cost-effective care. We found that mental health care, while sporadically and diversely attended to in outpatient visits, is often integrated with care of the diverse medical, social, and family problems that constitute primary care. Irrespective of differences in philosophy, training, or interest, however, structural and economic issues still appear to severely limit the ability of even willing family physicians to practice coherent integrated primary care.41 It is therefore important for the field as a whole to provide feasible strategies for promoting recognition and treatment of mental health issues by diverse clinicians and patients in usual practice settings.

Acknowledgments

Our study was supported by a grant from the Agency for Healthcare Research and Quality (R01 HS08776) and a research center grant from the American Academy of Family Physicians. The authors are grateful to the physicians, staff, and patients from the 18 practices, without whose participation this study would not have been possible. We also wish to acknowledge the dedicated work of Connie Gibbs and Jen Rouse, who spent countless hours collecting data; Diane Dodendorf and Jason Lebsack, who coordinated transcription and data management; and Mary McAndrews, who transcribed hundreds of taped interviews and dictated field notes. We would also like to thank Kurt C. Stange, MD, PhD, for reviewing earlier drafts of our manuscript.