Conducting The Direct Observation of Primary Care Study Insights from the Process of Conducting Multimethod Transdisciplinary Research in Community Practice

The Direct Observation of Primary Care (DOPC) Study has contributed to the understanding of family practice and has fostered the development of new primary care research methods^1-6 and theoretical perspectives.^7-10 The study’s findings have important implications for improving patient care^11-23 and developing policies^9,24-31 that maximize the impact of a generalist patient-centered approach toward the health of individuals, families, and communities.⁹ This study has spawned a large portfolio of related inquiry, including an in-depth qualitative study of family practices, a multimethod community practice intervention trial, and a new family practice research center.

The DOPC Study story represents a unique confluence of ideas, people, and opportunities. However, we hope that readers might glean insights relevant to their lines of inquiry and that this article will stimulate the continued development of a unique primary care and family practice research agenda.*

The Dopc Story

Concept Development

In 1988, family practice researchers Kurt Stange and Stephen Zyzanski collaborated on a paper about the benefits of integrating quantitative and qualitative research methods.³² While writing a second manuscript on the topic, they invited Benjamin Crabtree and William Miller (emerging experts on the application of qualitative research methods to primary care) to collaborate.³³

At the same time, a group of family physicians and researchers affiliated with Case Western Reserve University in Cleveland, Ohio, was attempting to develop an innovative approach to improving clinical preventive service delivery in practice. With a grant from the Ohio Academy of Family Physicians Foundation, they conducted a survey on family physician agreement with United States Preventive Services Task Force recommendations.^34-36 The survey findings, conversations with respondents, and a review of the literature led to the conclusion that current approaches to improving clinical preventive service delivery were limited by a lack of understanding of the true nature of family practice³⁷ and that efforts to improve practice should be preceded by efforts to understand practice.^38,39 Before designing an intervention study, further insight into the “black box” of real world family practice was needed.^2,40

The research group, which now included family practice academicians, clinicians, and methodologists, began exploring the development of a research network backward from the traditionally successful models used by the Ambulatory Sentinel Practice Network (ASPN) and other networks.^41-44 Rather than developing the infrastructure around a network of clinicians that performs research by gathering data, this new network was developed around a large descriptive study of the content and context of family practice. Funding for a specific study would be easier to obtain than start-up costs for a practice-based research infrastructure. Also, given the demands of clinical practice,⁴⁵ a well-funded study in a regional network could collect more extensive data than a study conducted by individual practices, providing opportunities for spinoff studies and other clinician-initiated inquiries. The research team decided to explore research opportunities with national funding agencies, with close communications and extensive input from local practicing family physicians.

During this time, a series of primary care research conferences sponsored by the Agency for Health Care Policy and Research (AHCPR) provided fertile ground for exploring research ideas and methods from a multidisciplinary perspective. At one conference, research team member Carlos Jaén (an epidemiologist and family physician), team leader Kurt Stange, and Paul Nutting (director of primary care research at AHCPR at the time) discussed the research network and project. The recognition emerged that many worthwhile primary care activities, including preventive service delivery, are not carried out during patient visits because of the competing demands imposed by other activities. This competing demands mode¹⁷ of preventive service delivery and primary care provided an important initial theoretical framework for what would become the DOPC Study.

Research Design

The research team began refining study questions and developing methods. A critical event occurred during a discussion of methods for measuring the content of outpatient family practice visits. Jason Chao, a family practice academician, enumerated these methods:“…chart review, patient questionnaire, billing data. One could do direct observation, but you can’t do that.” As everyone nodded agreement, his colleague Robert Kelly interrupted, “Why not? Why can’t you do direct observation?” The group listed many good reasons: intrusiveness, unacceptability to patients and clinicians, expense, and the potential to bias behavior. However, the question “Why not?” remained and created a shared sense that direct observation of real world family practices represented an opportunity to make a unique contribution. The group decided to include direct observation as a major measurement technique and to add a methodologic goal of establishing the validity and reliability of nonobservational techniques for assessing the content of outpatient medical practice. An additional advance occurred with the publication of the Davis Observation Code (DOC)⁴⁶ that classified patient visits into 20 different behavioral codes measured in 15-second intervals. Lead author Edward Callahan agreed to become a collaborator.

Limited existing research on the content of community primary care practice meant that the group would have difficulty in anticipating all content areas worth measuring and questions worth asking before immersing themselves in community family practice settings. Therefore, Drs Crabtree and Miller were asked to join the team to design a multimethod approach that integrated quantitative and qualitative methods.⁴⁷ Project design was pursued further in research team meetings, telephone conversations, and interactions with out-of-town collaborators during national professional meetings. These face-to-face meetings were essential to developing the trust, communication, and shared vision necessary for a transdisciplinary multimethod study.

Conversations with local family physicians soon revealed that preventive service delivery, although an important aspect of family practice, was not a sufficiently compelling research question to engage a new practice-based research network. A broader focus, such as the content of family practice, would engage the largest number of clinicians and be less likely to bias clinician behavior during direct observation. At the suggestion of practicing family physician Michael Rabovsky, in whose office the protocol was being pilot tested, the study was expanded to address the Medicare Resource-Based Relative Value System (RBRVS)-based billing system. Health economist Daniel Dunn, who helped develop the RBRVS,⁴⁸ was invited to participate.

Based on discussions with practicing family physicians, a strategy was developed for recruiting practice-based research network members. Members of the Ohio Academy of Family Physicians (OAFP) in Northeast Ohio were targeted to facilitate easy meeting of practices and travel of study teams to practice sites.⁴⁵ A letter describing the study and proposed network was sent to all 531 OAFP active members in the area. A total of 138 physicians responded and formed the fledgling Research Association of Practicing Physicians (RAPP) network. A working relationship was established with the NorthEast Ohio Network (NEON), a practice-based research network of 6 community residency training sites affiliated with the NorthEast Ohio Universities Colleges of Medicine, directed by William Gillanders (and later Valerie Gilchrist). NEON physicians were trained in National Ambulatory Medical Care Survey (NAMCS)⁴⁹ data collection techniques and provided the opportunity to evaluate the validity of the NAMCS methods compared with direct observation. These development activities were supported by considerable in-kind contributions of investigator time from the participants’ institutions.

Pursuit of Funding

A research concept paper was sent to the AHCPR for feedback. The response indicated that intervention studies were more compatible with funding priorities than the proposed observational study. The critique also pointed out “fatal flaws” engendered by direct observation methods and expressed skepticism that community physicians would allow such observation of their practices. These concerns were addressed with pilot data and a strengthened argument about the need for efforts to understand practice before attempting to change it. An investigator-initiated (R01) grant application was submitted to the AHCPR. A secondary assignment to the National Cancer Institute (NCI) was requested because of the clinical preventive service delivery focus and the important potential of understanding family practice and competing demands for the subsequent design of interventions to enhance cancer prevention and control.

The initial application was favorably reviewed and received a priority score near the funding line. In response to advice from NCI and AHCPR program officers, the research team allowed the application to be considered for funding during 3 upcoming NCI council meetings. Regular letters to research network members kept them informed of the funding status. After 1 year of narrowly missing the funding line, the grant application was revised and resubmitted in response to the scientific review committee’s critique, with increased emphasis on the implications of the study for cancer prevention and control. It was funded by the NCI, with an additional grant from the Robert Wood Johnson Foundation Generalist Physician Faculty Scholars Program, to develop communication and clinician-initiated research in RAPP and additional methodologic and descriptive aims.

Planning and Conduct of Fieldwork

In 1994, more than 3 years after the idea was conceived, the board of directors of the 138-member research network (RAPP) was formally activated. The board’s 14 network volunteer members, several of whom helped develop the study and the network, were active in planning the practical implementation and refinement of the study protocol. When a board member suggested that they review the details of the direct observation measures before the study, the board concluded that as study participants they should not be involved in planning study measures, to avoid biasing their behavior during direct observation.

With funding, 2 logistic aspects of the study gained importance. First, it had been 2 years since the research network was formed. Whether physicians had retained their commitment to participate was unknown. That concern was laid to rest, however, when the vast majority of physicians expressed continued interest in the study. In retrospect, the 2-year struggle to obtain funding helped bond the network and create a sense of ownership and allegiance to the project.

The second major logistic issue was the need to recruit 8 research nurses. Job requirements included excellent interpersonal skills, sensitivity to the demands of real world community family practice, attention to detail in collecting reliable and valid quantitative data from multiple measures, an open-minded observational ability to simultaneously collect qualitative data, willingness to drive to multiple and sometimes distant sites, and interest in a 1-year job at a university salary. Hiring 8 nurses who could meet these requirements and start the 2-month training process on the same date seemed unrealistic at best. Yet, because of word of mouth advertising, the excitement generated by the study, the recent termination of another research project at the university, and the excellent reputation of the department, 8 highly qualified research nurses were found.

During their 8-week training, the research nurses were enlisted as true partners. They helped refine the research protocol and instruments, and items were added to the measures to reflect their interests. Using videotaped encounters, Dr Callahan instructed the nurses in applying the DOC, and they took the lead in adapting it for the study. As the immensity of the quantitative data collection requirements grew, Drs Miller and Crabtree scaled back the qualitative data collection protocol. They trained the research nurses in observational techniques and in how to dictate ethnographic field notes to record unanticipated findings, provide rich descriptions of quantitatively measured variables, and critique the study methods’ accuracy in capturing the phenomena under study.⁵⁰

Details of the data collection procedure have been reported elsewhere.^1,2 Briefly, teams of 2 research nurses spent 1 day observing patient care by the 138 participating RAPP members. One nurse obtained verbal informed consent from patients in the waiting room and distributed patient exit questionnaires. The other nurse accompanied the physician, directly observing consecutive visits by consenting patients and recording observations using the DOC and a direct observation checklist. Typically, the nurses exchanged roles after lunch. They returned on a subsequent day to perform medical record reviews for each other’s observed visits and to collect billing data. On the basis of observation and brief interviews with key informants, the research nurses completed a practice environment checklist. They dictated ethnographic field notes immediately after leaving the practice.¹⁰

During the course of the fieldwork, research team meetings were held every other week to coordinate logistics and assess and recalibrate inter-rater reliability using videotaped visits and medical records that were not part of the larger study. The high degree of inter-rater reliability achieved with this approach has been reported previously.¹

After data were collected from each physician, the board of directors met to review study progress and reassess the study protocol. The academic research team, including all consultants, also met to refine the protocol and plan the second round of physician visits. Initial plans called for ongoing analysis of the ethnographic field notes, but this proved to be infeasible because of their large volume and the study demands. However, at the study midpoint, Drs Crabtree and Miller independently analyzed the field notes using an immersion crystallization technique.⁵¹ Based on the richness of the information, they developed a template⁵² for gathering field notes during the second round of physician visits.

Data collection procedures were repeated, and each physician was visited a second time. The 4 months (on average) between visits helped assure that seasonal variations in health problems did not unduly affect the characterization of patient care. After the second data collection visit, physicians completed a detailed questionnaire.

Data Analysis and Production of Scholarly Output

The data were entered by optical scanner and manually verified. Quantitative data analyses were performed by Cleveland research team members in response to the initial research aims and additional questions raised by the research team and research network board. Qualitative data analyses were subcontracted to the University of Nebraska, with additional grant support from the American Academy of Family Physicians for more in-depth analyses.

Multiple papers were begun with diverse lead authorship. Preference in determining paper topics was given to methodologic manuscripts, topics with timely policy implications, and papers for which individual team members had a particular passion. In response to a call from the editorial office of JFP, a proposal for a theme issue on the DOPC Study was made and accepted. The opportunity to publish early scholarly output in one place greatly increased the potential for papers on diverse topics that would help cohesively describe several aspects of the value of family practice. The deadline for the theme issue also made the paper writing a high priority. Of 14 manuscripts accepted after going through peer review, 10 were included in the May 1998 issue of JFP,^{2,8-11,14,15,17,18,25,40} with one paper published in each of 4 subsequent issues.^13,16,26,27 Other analyses and papers have focused on the original research themes, new topics, more complex analyses, and expansion into the non–family practice literature.

Opportunities to propose paper topics have been extended to all study participants, including the academic research team, consultants, and RAPP members. Proposed topics are reviewed for feasibility and potential conflicts with other papers. The data set has spawned 2 masters theses^16,19 and one doctoral dissertation^3,4,12,24 and has led to new collaborations with complementary content experts.

Related Research Initiatives

Concurrent with the DOPC study, Dr Crabtree and his colleagues in Nebraska conducted a series of related inquiries.^{3-5,10,12,24,39,53-56} These studies have provided complementary information and advanced multimethod approaches for studying primary care practice. Close collaboration and open information sharing among the research teams and collaborators have greatly facilitated the discovery of new methods and insights into family practice and have furthered the research trajectory of the collaborating groups. These collaborations spawned the Center for Research in Family Practice and Primary Care, a multisite consortium funded by the American Academy of Family Physicians.

DOPC Study collaborations have led to other research initiatives as well. For example, a desire for more in-depth qualitative data led to a comparative case study of a smaller number of purposively selected practices in Nebraska, funded by the AHCPR with Dr Crabtree as principal investigator. In addition, after reviewing the initial findings of the first round of DOPC data, the RAPP board of directors developed a study of competing demands outside the examination room, which has led to related inquires.

Based on emerging insights from the DOPC Study on the competing demands of family practice, a competing continuation application was funded by the NCI for a trial to improve clinical preventive service delivery. The Study to Enhance Prevention by Understanding Practice (STEP-UP) was developed with input from the research team and the RAPP board of directors, with collaboration from family practice researchers at Dartmouth, led by Allen Dietrich.^57,58 Building on complexity theory–based insights from the DOPC Study,⁸ STEP-UP uses a multimethod practice assessment to understand the unique attributes of family practices and tailor intervention strategies. This approach increased preventive service delivery rates⁵⁹ and led to a more comprehensive assessment and improvement strategy that is being evaluated in the delayed intervention group. The participants include DOPC Study practices and new RAPP members.

Continuing efforts to develop the RAPP network have included free continuing medical education conferences for participants in practice-based research and quality improvement projects. An ongoing research network newsletter periodically publishes a 1-page Research Prospectus Worksheet* to encourage research ideas from RAPP members. The Cleveland research team provides rapid turnaround methodological consultation for study proposals, and those involving multiple practices are reviewed by the RAPP board of directors. In addition, RAPP members are encouraged to serve as authors on DOPC papers, and approximately half have provided internal peer review before submission of papers.

Several RAPP members have received external funding for their own research projects. These include studies of causes of bilateral leg edema in family practice,^60,61 an evaluation of a family-centered approach to diagnosis and treatment of respiratory infection, a clinical trial of therapeutic touch for carpal tunnel syndrome, and development of practical new methods for community-oriented primary care.⁶² A recent RAPP study, in collaboration with the NEON network, used a card study methodology to describe the “oh, by the way” phenomenon in which patients raise issues after the clinician thinks the outpatient visit is finished. In addition, the discovery of high rates of care of a secondary patient11 in the DOPC study led to an ASPN card study to elucidate the content of care provided to family members other than the identified patient for an outpatient visit.⁶³ An additional ASPN collaboration, using the Components of Primary Care Instrument^3,4 that was developed as part of the DOPC Study, examined the effect of different aspects of managed care on the delivery of 10 elements of quality primary care.⁶⁴

Lessons learned from the dopc process

Some of the lessons learned from the process of conducting the DOPC study are summarized in the Table 1. These lessons can be grouped into 6 categories, as follows.

A generalist perspective. A generalist perspective that places research questions in the context of the competing opportunities and complexity of family practice is needed for true family practice and primary care research.⁶⁵ Although this perspective is essential if we are to diminish the current chasm between discovery and practice, it has not been supported by those who fund research. One strategy for addressing this funding issue is to identify topics and multimethod approaches that allow simultaneous pursuit of both categorical and generalist perspectives.

Involvement. The involvement of community practices and practicing clinicians as partners is essential for research about primary care practice.^{66, 67} New knowledge from discoveries in the settings in which most people get most of their medical care will help end the dichotomy between research and dissemination. Practice-based research networks can help bridge this gap by asking and answering questions from the perspective and setting in which the findings will be applied.^68,69 (It is worth noting, however, that most successful research networks are built around a group of clinicians who are committed to conducting research in their practices. Developing a network around a particular study, as with the RAPP network, requires attention to fostering clinician ideas and nurturing relationships that extend beyond the initial study.) Greater involvement of nonclinician health care professionals, patients, and communities can also increase the relevance of research to meet the population’s health care needs.^67,70

Transdisciplinary team process. A transdisciplinary team process in which diverse specialized expertise is integrated toward a common goal can be a tremendous resource for innovation and productivity. Development of a transdisciplinary team is a long-term process that requires trust, shared vision, open leadership, idea sharing, and group meetings. In addition, team members with particular expertise must be willing to commit to creating new knowledge that transcends their disciplinary perspectives.⁷¹ Such collaboration creates the mentality of a bigger pie in which the size of each participants’ piece is increased, rather than a mentality of finite resources in which a bigger piece for one member creates a shortage for another.⁷²

Multimethodology. A multimethod approach in which quantitative and qualitative methods are integrated creates the opportunity to generate new methods, assure rigor, and maximize the efficiency of new discovery.^6,32,33,47 Multimethod approaches allow testing of a priori hypotheses while creating new understanding.

Openness. Openness to emerging insights is fostered by the generalist perspective, by participatory multimethod research approaches, and by building the project from pilot data and knowledge of previous work. In the DOPC study, openness to new methods led to the “Eureka!” moment of deciding to do direct observation. The involvement of clinician and nurse perspectives in study design and conduct and the inductive use of qualitative data to discover the relevance of complexity science to understanding and enhancing primary care practice also reflected the study’s openness to new approaches.

Thinking big, but starting small. This creates a larger vision that can guide and inspire individual decisions and creates an overall research trajectory built on incremental steps. The DOPC Study began with a large idea of improving practice. Grounding in real world practice led to development of innovative new methods to try to understand primary care practice and ongoing efforts to improve practice. These major undertakings, however, were built on a foundation of small pilot studies and multiple interactions among researchers and practicing family physicians.

Applying these insights to other studies may help to advance the generation of new knowledge about family practice and primary care.⁷³

Acknowledgments

This research was supported by grants from the National Cancer Institute (1R01 CA60962, 2R01 CA60962 and K24 CA81931), the Agency for Health Care Policy and Research (1R01 HS08776), the Ohio Academy of Family Physicians, the American Academy of Family Physicians, Generalist Physician Faculty Scholar Awards to Drs Stange and Jaén from the Robert Wood Johnson Foundation, and a Family Practice Research Center Grant from the American Academy of Family Physicians. The authors are grateful to the RAPP physicians, other clinicians, office staffs, and patients, without whose participation our study would not have been possible. We are also indebted to the many people who have participated and continue to participate in the genesis of related ideas and scholarly output that continues to emerge from the original study. Members of the DOPC Writing group also include: Authors from the Academic Research Team: Stephen J. Zyzanski, PhD, Department of Family Medicine, Case Western Reserve University, Cleveland, Ohio; Benjamin F. Crabtree, PhD, Department of Family Medicine, UMDNJ-RWJ Medical School, New Brunswick, NJ; William L. Miller, MD, MA, Department of Family Practice, Lehigh Valley Hospital, Allentown, Pa; Carlos Roberto Jaén, MD, PhD, Center for Urban Research in Primary Care, SUNY, Buffalo, NY; Susan A. Flocke, PhD, Department of Family Medicine, Case Western Reserve University, Cleveland, Ohio; Robert B. Kelly, MD, MS, Department of Family Practice, MetroHealth Medical Center, Cleveland, Ohio; William R. Gillanders, MD, Family Practice Residency Program, Sutter Health, Sacramento, Calif; Valerie Gilchrist, MD, Department of Family Practice, NorthEast Ohio Universities College of Medicine, Rootstown, Ohio; Jason Chao, MD, MS, Department of Family Medicine, Case Western Reserve University; J. Christopher Shank, MD, Methodist/Indiana University Family Practice Residency, Indianapolis, Ind; Daniel L. Dunn, PhD, Integrated Health Care Information Service, Cambridge, Mass; Jack H. Medalie, MD, MPH, Department of Family Medicine, Case Western Reserve University, Cleveland, Ohio; Doreen Langa, BA, American University School of Law, Washington, DC; Virginia Aita, PhD, Department of Family Practice, University of Nebraska Medical Center, Omaha; Meredith A. Goodwin, MS, Department of Family Medicine, Case Western Reserve University, Cleveland, Ohio; and Robin S. Gotler, MA, Department of Family Medicine, Case Western Reserve University, Cleveland, Ohio. Research Nurse Team Authors: Lisa B. Ballou, RN, FNP; Catherine M. Corrigan, RN; Luzmaria Jaén, RN; Sherry Patzke, RN; Frances F. Powers, RN; Kathleen L. Schneeberger, RN; Kelly Warner, RN; and Susan Zronek, RN. Authors from the RAPP Board of Directors: Robert Blankfield, MD; Henry Bloom, MD; Valerie Gilchrist, MD; Gwen Haas, MD; Patricia Kellner, MD; Sa Koo Lee, MD; Conrad Lindes, MD; Dennis McCluskey, MD; Thomas Mettee, MD; Albert Miller, MD; Michael Rabovsky, MD; and Archie Wilkinson, MD.

The Direct Observation of Primary Care (DOPC) Study has contributed to the understanding of family practice and has fostered the development of new primary care research methods^1-6 and theoretical perspectives.^7-10 The study’s findings have important implications for improving patient care^11-23 and developing policies^9,24-31 that maximize the impact of a generalist patient-centered approach toward the health of individuals, families, and communities.⁹ This study has spawned a large portfolio of related inquiry, including an in-depth qualitative study of family practices, a multimethod community practice intervention trial, and a new family practice research center.

The DOPC Study story represents a unique confluence of ideas, people, and opportunities. However, we hope that readers might glean insights relevant to their lines of inquiry and that this article will stimulate the continued development of a unique primary care and family practice research agenda.*

The Dopc Story

Concept Development

In 1988, family practice researchers Kurt Stange and Stephen Zyzanski collaborated on a paper about the benefits of integrating quantitative and qualitative research methods.³² While writing a second manuscript on the topic, they invited Benjamin Crabtree and William Miller (emerging experts on the application of qualitative research methods to primary care) to collaborate.³³

At the same time, a group of family physicians and researchers affiliated with Case Western Reserve University in Cleveland, Ohio, was attempting to develop an innovative approach to improving clinical preventive service delivery in practice. With a grant from the Ohio Academy of Family Physicians Foundation, they conducted a survey on family physician agreement with United States Preventive Services Task Force recommendations.^34-36 The survey findings, conversations with respondents, and a review of the literature led to the conclusion that current approaches to improving clinical preventive service delivery were limited by a lack of understanding of the true nature of family practice³⁷ and that efforts to improve practice should be preceded by efforts to understand practice.^38,39 Before designing an intervention study, further insight into the “black box” of real world family practice was needed.^2,40

The research group, which now included family practice academicians, clinicians, and methodologists, began exploring the development of a research network backward from the traditionally successful models used by the Ambulatory Sentinel Practice Network (ASPN) and other networks.^41-44 Rather than developing the infrastructure around a network of clinicians that performs research by gathering data, this new network was developed around a large descriptive study of the content and context of family practice. Funding for a specific study would be easier to obtain than start-up costs for a practice-based research infrastructure. Also, given the demands of clinical practice,⁴⁵ a well-funded study in a regional network could collect more extensive data than a study conducted by individual practices, providing opportunities for spinoff studies and other clinician-initiated inquiries. The research team decided to explore research opportunities with national funding agencies, with close communications and extensive input from local practicing family physicians.

During this time, a series of primary care research conferences sponsored by the Agency for Health Care Policy and Research (AHCPR) provided fertile ground for exploring research ideas and methods from a multidisciplinary perspective. At one conference, research team member Carlos Jaén (an epidemiologist and family physician), team leader Kurt Stange, and Paul Nutting (director of primary care research at AHCPR at the time) discussed the research network and project. The recognition emerged that many worthwhile primary care activities, including preventive service delivery, are not carried out during patient visits because of the competing demands imposed by other activities. This competing demands mode¹⁷ of preventive service delivery and primary care provided an important initial theoretical framework for what would become the DOPC Study.

Research Design

The research team began refining study questions and developing methods. A critical event occurred during a discussion of methods for measuring the content of outpatient family practice visits. Jason Chao, a family practice academician, enumerated these methods:“…chart review, patient questionnaire, billing data. One could do direct observation, but you can’t do that.” As everyone nodded agreement, his colleague Robert Kelly interrupted, “Why not? Why can’t you do direct observation?” The group listed many good reasons: intrusiveness, unacceptability to patients and clinicians, expense, and the potential to bias behavior. However, the question “Why not?” remained and created a shared sense that direct observation of real world family practices represented an opportunity to make a unique contribution. The group decided to include direct observation as a major measurement technique and to add a methodologic goal of establishing the validity and reliability of nonobservational techniques for assessing the content of outpatient medical practice. An additional advance occurred with the publication of the Davis Observation Code (DOC)⁴⁶ that classified patient visits into 20 different behavioral codes measured in 15-second intervals. Lead author Edward Callahan agreed to become a collaborator.

Limited existing research on the content of community primary care practice meant that the group would have difficulty in anticipating all content areas worth measuring and questions worth asking before immersing themselves in community family practice settings. Therefore, Drs Crabtree and Miller were asked to join the team to design a multimethod approach that integrated quantitative and qualitative methods.⁴⁷ Project design was pursued further in research team meetings, telephone conversations, and interactions with out-of-town collaborators during national professional meetings. These face-to-face meetings were essential to developing the trust, communication, and shared vision necessary for a transdisciplinary multimethod study.

Conversations with local family physicians soon revealed that preventive service delivery, although an important aspect of family practice, was not a sufficiently compelling research question to engage a new practice-based research network. A broader focus, such as the content of family practice, would engage the largest number of clinicians and be less likely to bias clinician behavior during direct observation. At the suggestion of practicing family physician Michael Rabovsky, in whose office the protocol was being pilot tested, the study was expanded to address the Medicare Resource-Based Relative Value System (RBRVS)-based billing system. Health economist Daniel Dunn, who helped develop the RBRVS,⁴⁸ was invited to participate.

Based on discussions with practicing family physicians, a strategy was developed for recruiting practice-based research network members. Members of the Ohio Academy of Family Physicians (OAFP) in Northeast Ohio were targeted to facilitate easy meeting of practices and travel of study teams to practice sites.⁴⁵ A letter describing the study and proposed network was sent to all 531 OAFP active members in the area. A total of 138 physicians responded and formed the fledgling Research Association of Practicing Physicians (RAPP) network. A working relationship was established with the NorthEast Ohio Network (NEON), a practice-based research network of 6 community residency training sites affiliated with the NorthEast Ohio Universities Colleges of Medicine, directed by William Gillanders (and later Valerie Gilchrist). NEON physicians were trained in National Ambulatory Medical Care Survey (NAMCS)⁴⁹ data collection techniques and provided the opportunity to evaluate the validity of the NAMCS methods compared with direct observation. These development activities were supported by considerable in-kind contributions of investigator time from the participants’ institutions.

Pursuit of Funding

A research concept paper was sent to the AHCPR for feedback. The response indicated that intervention studies were more compatible with funding priorities than the proposed observational study. The critique also pointed out “fatal flaws” engendered by direct observation methods and expressed skepticism that community physicians would allow such observation of their practices. These concerns were addressed with pilot data and a strengthened argument about the need for efforts to understand practice before attempting to change it. An investigator-initiated (R01) grant application was submitted to the AHCPR. A secondary assignment to the National Cancer Institute (NCI) was requested because of the clinical preventive service delivery focus and the important potential of understanding family practice and competing demands for the subsequent design of interventions to enhance cancer prevention and control.

The initial application was favorably reviewed and received a priority score near the funding line. In response to advice from NCI and AHCPR program officers, the research team allowed the application to be considered for funding during 3 upcoming NCI council meetings. Regular letters to research network members kept them informed of the funding status. After 1 year of narrowly missing the funding line, the grant application was revised and resubmitted in response to the scientific review committee’s critique, with increased emphasis on the implications of the study for cancer prevention and control. It was funded by the NCI, with an additional grant from the Robert Wood Johnson Foundation Generalist Physician Faculty Scholars Program, to develop communication and clinician-initiated research in RAPP and additional methodologic and descriptive aims.

Planning and Conduct of Fieldwork

In 1994, more than 3 years after the idea was conceived, the board of directors of the 138-member research network (RAPP) was formally activated. The board’s 14 network volunteer members, several of whom helped develop the study and the network, were active in planning the practical implementation and refinement of the study protocol. When a board member suggested that they review the details of the direct observation measures before the study, the board concluded that as study participants they should not be involved in planning study measures, to avoid biasing their behavior during direct observation.

With funding, 2 logistic aspects of the study gained importance. First, it had been 2 years since the research network was formed. Whether physicians had retained their commitment to participate was unknown. That concern was laid to rest, however, when the vast majority of physicians expressed continued interest in the study. In retrospect, the 2-year struggle to obtain funding helped bond the network and create a sense of ownership and allegiance to the project.

The second major logistic issue was the need to recruit 8 research nurses. Job requirements included excellent interpersonal skills, sensitivity to the demands of real world community family practice, attention to detail in collecting reliable and valid quantitative data from multiple measures, an open-minded observational ability to simultaneously collect qualitative data, willingness to drive to multiple and sometimes distant sites, and interest in a 1-year job at a university salary. Hiring 8 nurses who could meet these requirements and start the 2-month training process on the same date seemed unrealistic at best. Yet, because of word of mouth advertising, the excitement generated by the study, the recent termination of another research project at the university, and the excellent reputation of the department, 8 highly qualified research nurses were found.

During their 8-week training, the research nurses were enlisted as true partners. They helped refine the research protocol and instruments, and items were added to the measures to reflect their interests. Using videotaped encounters, Dr Callahan instructed the nurses in applying the DOC, and they took the lead in adapting it for the study. As the immensity of the quantitative data collection requirements grew, Drs Miller and Crabtree scaled back the qualitative data collection protocol. They trained the research nurses in observational techniques and in how to dictate ethnographic field notes to record unanticipated findings, provide rich descriptions of quantitatively measured variables, and critique the study methods’ accuracy in capturing the phenomena under study.⁵⁰

Details of the data collection procedure have been reported elsewhere.^1,2 Briefly, teams of 2 research nurses spent 1 day observing patient care by the 138 participating RAPP members. One nurse obtained verbal informed consent from patients in the waiting room and distributed patient exit questionnaires. The other nurse accompanied the physician, directly observing consecutive visits by consenting patients and recording observations using the DOC and a direct observation checklist. Typically, the nurses exchanged roles after lunch. They returned on a subsequent day to perform medical record reviews for each other’s observed visits and to collect billing data. On the basis of observation and brief interviews with key informants, the research nurses completed a practice environment checklist. They dictated ethnographic field notes immediately after leaving the practice.¹⁰

During the course of the fieldwork, research team meetings were held every other week to coordinate logistics and assess and recalibrate inter-rater reliability using videotaped visits and medical records that were not part of the larger study. The high degree of inter-rater reliability achieved with this approach has been reported previously.¹

After data were collected from each physician, the board of directors met to review study progress and reassess the study protocol. The academic research team, including all consultants, also met to refine the protocol and plan the second round of physician visits. Initial plans called for ongoing analysis of the ethnographic field notes, but this proved to be infeasible because of their large volume and the study demands. However, at the study midpoint, Drs Crabtree and Miller independently analyzed the field notes using an immersion crystallization technique.⁵¹ Based on the richness of the information, they developed a template⁵² for gathering field notes during the second round of physician visits.

Data collection procedures were repeated, and each physician was visited a second time. The 4 months (on average) between visits helped assure that seasonal variations in health problems did not unduly affect the characterization of patient care. After the second data collection visit, physicians completed a detailed questionnaire.

Data Analysis and Production of Scholarly Output

The data were entered by optical scanner and manually verified. Quantitative data analyses were performed by Cleveland research team members in response to the initial research aims and additional questions raised by the research team and research network board. Qualitative data analyses were subcontracted to the University of Nebraska, with additional grant support from the American Academy of Family Physicians for more in-depth analyses.

Multiple papers were begun with diverse lead authorship. Preference in determining paper topics was given to methodologic manuscripts, topics with timely policy implications, and papers for which individual team members had a particular passion. In response to a call from the editorial office of JFP, a proposal for a theme issue on the DOPC Study was made and accepted. The opportunity to publish early scholarly output in one place greatly increased the potential for papers on diverse topics that would help cohesively describe several aspects of the value of family practice. The deadline for the theme issue also made the paper writing a high priority. Of 14 manuscripts accepted after going through peer review, 10 were included in the May 1998 issue of JFP,^{2,8-11,14,15,17,18,25,40} with one paper published in each of 4 subsequent issues.^13,16,26,27 Other analyses and papers have focused on the original research themes, new topics, more complex analyses, and expansion into the non–family practice literature.

Opportunities to propose paper topics have been extended to all study participants, including the academic research team, consultants, and RAPP members. Proposed topics are reviewed for feasibility and potential conflicts with other papers. The data set has spawned 2 masters theses^16,19 and one doctoral dissertation^3,4,12,24 and has led to new collaborations with complementary content experts.

Related Research Initiatives

Concurrent with the DOPC study, Dr Crabtree and his colleagues in Nebraska conducted a series of related inquiries.^{3-5,10,12,24,39,53-56} These studies have provided complementary information and advanced multimethod approaches for studying primary care practice. Close collaboration and open information sharing among the research teams and collaborators have greatly facilitated the discovery of new methods and insights into family practice and have furthered the research trajectory of the collaborating groups. These collaborations spawned the Center for Research in Family Practice and Primary Care, a multisite consortium funded by the American Academy of Family Physicians.

DOPC Study collaborations have led to other research initiatives as well. For example, a desire for more in-depth qualitative data led to a comparative case study of a smaller number of purposively selected practices in Nebraska, funded by the AHCPR with Dr Crabtree as principal investigator. In addition, after reviewing the initial findings of the first round of DOPC data, the RAPP board of directors developed a study of competing demands outside the examination room, which has led to related inquires.

Based on emerging insights from the DOPC Study on the competing demands of family practice, a competing continuation application was funded by the NCI for a trial to improve clinical preventive service delivery. The Study to Enhance Prevention by Understanding Practice (STEP-UP) was developed with input from the research team and the RAPP board of directors, with collaboration from family practice researchers at Dartmouth, led by Allen Dietrich.^57,58 Building on complexity theory–based insights from the DOPC Study,⁸ STEP-UP uses a multimethod practice assessment to understand the unique attributes of family practices and tailor intervention strategies. This approach increased preventive service delivery rates⁵⁹ and led to a more comprehensive assessment and improvement strategy that is being evaluated in the delayed intervention group. The participants include DOPC Study practices and new RAPP members.

Continuing efforts to develop the RAPP network have included free continuing medical education conferences for participants in practice-based research and quality improvement projects. An ongoing research network newsletter periodically publishes a 1-page Research Prospectus Worksheet* to encourage research ideas from RAPP members. The Cleveland research team provides rapid turnaround methodological consultation for study proposals, and those involving multiple practices are reviewed by the RAPP board of directors. In addition, RAPP members are encouraged to serve as authors on DOPC papers, and approximately half have provided internal peer review before submission of papers.

Several RAPP members have received external funding for their own research projects. These include studies of causes of bilateral leg edema in family practice,^60,61 an evaluation of a family-centered approach to diagnosis and treatment of respiratory infection, a clinical trial of therapeutic touch for carpal tunnel syndrome, and development of practical new methods for community-oriented primary care.⁶² A recent RAPP study, in collaboration with the NEON network, used a card study methodology to describe the “oh, by the way” phenomenon in which patients raise issues after the clinician thinks the outpatient visit is finished. In addition, the discovery of high rates of care of a secondary patient11 in the DOPC study led to an ASPN card study to elucidate the content of care provided to family members other than the identified patient for an outpatient visit.⁶³ An additional ASPN collaboration, using the Components of Primary Care Instrument^3,4 that was developed as part of the DOPC Study, examined the effect of different aspects of managed care on the delivery of 10 elements of quality primary care.⁶⁴

Lessons learned from the dopc process

Some of the lessons learned from the process of conducting the DOPC study are summarized in the Table 1. These lessons can be grouped into 6 categories, as follows.

A generalist perspective. A generalist perspective that places research questions in the context of the competing opportunities and complexity of family practice is needed for true family practice and primary care research.⁶⁵ Although this perspective is essential if we are to diminish the current chasm between discovery and practice, it has not been supported by those who fund research. One strategy for addressing this funding issue is to identify topics and multimethod approaches that allow simultaneous pursuit of both categorical and generalist perspectives.

Involvement. The involvement of community practices and practicing clinicians as partners is essential for research about primary care practice.^{66, 67} New knowledge from discoveries in the settings in which most people get most of their medical care will help end the dichotomy between research and dissemination. Practice-based research networks can help bridge this gap by asking and answering questions from the perspective and setting in which the findings will be applied.^68,69 (It is worth noting, however, that most successful research networks are built around a group of clinicians who are committed to conducting research in their practices. Developing a network around a particular study, as with the RAPP network, requires attention to fostering clinician ideas and nurturing relationships that extend beyond the initial study.) Greater involvement of nonclinician health care professionals, patients, and communities can also increase the relevance of research to meet the population’s health care needs.^67,70

Transdisciplinary team process. A transdisciplinary team process in which diverse specialized expertise is integrated toward a common goal can be a tremendous resource for innovation and productivity. Development of a transdisciplinary team is a long-term process that requires trust, shared vision, open leadership, idea sharing, and group meetings. In addition, team members with particular expertise must be willing to commit to creating new knowledge that transcends their disciplinary perspectives.⁷¹ Such collaboration creates the mentality of a bigger pie in which the size of each participants’ piece is increased, rather than a mentality of finite resources in which a bigger piece for one member creates a shortage for another.⁷²

Multimethodology. A multimethod approach in which quantitative and qualitative methods are integrated creates the opportunity to generate new methods, assure rigor, and maximize the efficiency of new discovery.^6,32,33,47 Multimethod approaches allow testing of a priori hypotheses while creating new understanding.

Openness. Openness to emerging insights is fostered by the generalist perspective, by participatory multimethod research approaches, and by building the project from pilot data and knowledge of previous work. In the DOPC study, openness to new methods led to the “Eureka!” moment of deciding to do direct observation. The involvement of clinician and nurse perspectives in study design and conduct and the inductive use of qualitative data to discover the relevance of complexity science to understanding and enhancing primary care practice also reflected the study’s openness to new approaches.

Thinking big, but starting small. This creates a larger vision that can guide and inspire individual decisions and creates an overall research trajectory built on incremental steps. The DOPC Study began with a large idea of improving practice. Grounding in real world practice led to development of innovative new methods to try to understand primary care practice and ongoing efforts to improve practice. These major undertakings, however, were built on a foundation of small pilot studies and multiple interactions among researchers and practicing family physicians.

Applying these insights to other studies may help to advance the generation of new knowledge about family practice and primary care.⁷³

Acknowledgments

This research was supported by grants from the National Cancer Institute (1R01 CA60962, 2R01 CA60962 and K24 CA81931), the Agency for Health Care Policy and Research (1R01 HS08776), the Ohio Academy of Family Physicians, the American Academy of Family Physicians, Generalist Physician Faculty Scholar Awards to Drs Stange and Jaén from the Robert Wood Johnson Foundation, and a Family Practice Research Center Grant from the American Academy of Family Physicians. The authors are grateful to the RAPP physicians, other clinicians, office staffs, and patients, without whose participation our study would not have been possible. We are also indebted to the many people who have participated and continue to participate in the genesis of related ideas and scholarly output that continues to emerge from the original study. Members of the DOPC Writing group also include: Authors from the Academic Research Team: Stephen J. Zyzanski, PhD, Department of Family Medicine, Case Western Reserve University, Cleveland, Ohio; Benjamin F. Crabtree, PhD, Department of Family Medicine, UMDNJ-RWJ Medical School, New Brunswick, NJ; William L. Miller, MD, MA, Department of Family Practice, Lehigh Valley Hospital, Allentown, Pa; Carlos Roberto Jaén, MD, PhD, Center for Urban Research in Primary Care, SUNY, Buffalo, NY; Susan A. Flocke, PhD, Department of Family Medicine, Case Western Reserve University, Cleveland, Ohio; Robert B. Kelly, MD, MS, Department of Family Practice, MetroHealth Medical Center, Cleveland, Ohio; William R. Gillanders, MD, Family Practice Residency Program, Sutter Health, Sacramento, Calif; Valerie Gilchrist, MD, Department of Family Practice, NorthEast Ohio Universities College of Medicine, Rootstown, Ohio; Jason Chao, MD, MS, Department of Family Medicine, Case Western Reserve University; J. Christopher Shank, MD, Methodist/Indiana University Family Practice Residency, Indianapolis, Ind; Daniel L. Dunn, PhD, Integrated Health Care Information Service, Cambridge, Mass; Jack H. Medalie, MD, MPH, Department of Family Medicine, Case Western Reserve University, Cleveland, Ohio; Doreen Langa, BA, American University School of Law, Washington, DC; Virginia Aita, PhD, Department of Family Practice, University of Nebraska Medical Center, Omaha; Meredith A. Goodwin, MS, Department of Family Medicine, Case Western Reserve University, Cleveland, Ohio; and Robin S. Gotler, MA, Department of Family Medicine, Case Western Reserve University, Cleveland, Ohio. Research Nurse Team Authors: Lisa B. Ballou, RN, FNP; Catherine M. Corrigan, RN; Luzmaria Jaén, RN; Sherry Patzke, RN; Frances F. Powers, RN; Kathleen L. Schneeberger, RN; Kelly Warner, RN; and Susan Zronek, RN. Authors from the RAPP Board of Directors: Robert Blankfield, MD; Henry Bloom, MD; Valerie Gilchrist, MD; Gwen Haas, MD; Patricia Kellner, MD; Sa Koo Lee, MD; Conrad Lindes, MD; Dennis McCluskey, MD; Thomas Mettee, MD; Albert Miller, MD; Michael Rabovsky, MD; and Archie Wilkinson, MD.

The Direct Observation of Primary Care (DOPC) Study has contributed to the understanding of family practice and has fostered the development of new primary care research methods^1-6 and theoretical perspectives.^7-10 The study’s findings have important implications for improving patient care^11-23 and developing policies^9,24-31 that maximize the impact of a generalist patient-centered approach toward the health of individuals, families, and communities.⁹ This study has spawned a large portfolio of related inquiry, including an in-depth qualitative study of family practices, a multimethod community practice intervention trial, and a new family practice research center.

The DOPC Study story represents a unique confluence of ideas, people, and opportunities. However, we hope that readers might glean insights relevant to their lines of inquiry and that this article will stimulate the continued development of a unique primary care and family practice research agenda.*

The Dopc Story

Concept Development

In 1988, family practice researchers Kurt Stange and Stephen Zyzanski collaborated on a paper about the benefits of integrating quantitative and qualitative research methods.³² While writing a second manuscript on the topic, they invited Benjamin Crabtree and William Miller (emerging experts on the application of qualitative research methods to primary care) to collaborate.³³

At the same time, a group of family physicians and researchers affiliated with Case Western Reserve University in Cleveland, Ohio, was attempting to develop an innovative approach to improving clinical preventive service delivery in practice. With a grant from the Ohio Academy of Family Physicians Foundation, they conducted a survey on family physician agreement with United States Preventive Services Task Force recommendations.^34-36 The survey findings, conversations with respondents, and a review of the literature led to the conclusion that current approaches to improving clinical preventive service delivery were limited by a lack of understanding of the true nature of family practice³⁷ and that efforts to improve practice should be preceded by efforts to understand practice.^38,39 Before designing an intervention study, further insight into the “black box” of real world family practice was needed.^2,40

The research group, which now included family practice academicians, clinicians, and methodologists, began exploring the development of a research network backward from the traditionally successful models used by the Ambulatory Sentinel Practice Network (ASPN) and other networks.^41-44 Rather than developing the infrastructure around a network of clinicians that performs research by gathering data, this new network was developed around a large descriptive study of the content and context of family practice. Funding for a specific study would be easier to obtain than start-up costs for a practice-based research infrastructure. Also, given the demands of clinical practice,⁴⁵ a well-funded study in a regional network could collect more extensive data than a study conducted by individual practices, providing opportunities for spinoff studies and other clinician-initiated inquiries. The research team decided to explore research opportunities with national funding agencies, with close communications and extensive input from local practicing family physicians.

During this time, a series of primary care research conferences sponsored by the Agency for Health Care Policy and Research (AHCPR) provided fertile ground for exploring research ideas and methods from a multidisciplinary perspective. At one conference, research team member Carlos Jaén (an epidemiologist and family physician), team leader Kurt Stange, and Paul Nutting (director of primary care research at AHCPR at the time) discussed the research network and project. The recognition emerged that many worthwhile primary care activities, including preventive service delivery, are not carried out during patient visits because of the competing demands imposed by other activities. This competing demands mode¹⁷ of preventive service delivery and primary care provided an important initial theoretical framework for what would become the DOPC Study.

Research Design

The research team began refining study questions and developing methods. A critical event occurred during a discussion of methods for measuring the content of outpatient family practice visits. Jason Chao, a family practice academician, enumerated these methods:“…chart review, patient questionnaire, billing data. One could do direct observation, but you can’t do that.” As everyone nodded agreement, his colleague Robert Kelly interrupted, “Why not? Why can’t you do direct observation?” The group listed many good reasons: intrusiveness, unacceptability to patients and clinicians, expense, and the potential to bias behavior. However, the question “Why not?” remained and created a shared sense that direct observation of real world family practices represented an opportunity to make a unique contribution. The group decided to include direct observation as a major measurement technique and to add a methodologic goal of establishing the validity and reliability of nonobservational techniques for assessing the content of outpatient medical practice. An additional advance occurred with the publication of the Davis Observation Code (DOC)⁴⁶ that classified patient visits into 20 different behavioral codes measured in 15-second intervals. Lead author Edward Callahan agreed to become a collaborator.

Limited existing research on the content of community primary care practice meant that the group would have difficulty in anticipating all content areas worth measuring and questions worth asking before immersing themselves in community family practice settings. Therefore, Drs Crabtree and Miller were asked to join the team to design a multimethod approach that integrated quantitative and qualitative methods.⁴⁷ Project design was pursued further in research team meetings, telephone conversations, and interactions with out-of-town collaborators during national professional meetings. These face-to-face meetings were essential to developing the trust, communication, and shared vision necessary for a transdisciplinary multimethod study.

Conversations with local family physicians soon revealed that preventive service delivery, although an important aspect of family practice, was not a sufficiently compelling research question to engage a new practice-based research network. A broader focus, such as the content of family practice, would engage the largest number of clinicians and be less likely to bias clinician behavior during direct observation. At the suggestion of practicing family physician Michael Rabovsky, in whose office the protocol was being pilot tested, the study was expanded to address the Medicare Resource-Based Relative Value System (RBRVS)-based billing system. Health economist Daniel Dunn, who helped develop the RBRVS,⁴⁸ was invited to participate.

Based on discussions with practicing family physicians, a strategy was developed for recruiting practice-based research network members. Members of the Ohio Academy of Family Physicians (OAFP) in Northeast Ohio were targeted to facilitate easy meeting of practices and travel of study teams to practice sites.⁴⁵ A letter describing the study and proposed network was sent to all 531 OAFP active members in the area. A total of 138 physicians responded and formed the fledgling Research Association of Practicing Physicians (RAPP) network. A working relationship was established with the NorthEast Ohio Network (NEON), a practice-based research network of 6 community residency training sites affiliated with the NorthEast Ohio Universities Colleges of Medicine, directed by William Gillanders (and later Valerie Gilchrist). NEON physicians were trained in National Ambulatory Medical Care Survey (NAMCS)⁴⁹ data collection techniques and provided the opportunity to evaluate the validity of the NAMCS methods compared with direct observation. These development activities were supported by considerable in-kind contributions of investigator time from the participants’ institutions.

Pursuit of Funding

A research concept paper was sent to the AHCPR for feedback. The response indicated that intervention studies were more compatible with funding priorities than the proposed observational study. The critique also pointed out “fatal flaws” engendered by direct observation methods and expressed skepticism that community physicians would allow such observation of their practices. These concerns were addressed with pilot data and a strengthened argument about the need for efforts to understand practice before attempting to change it. An investigator-initiated (R01) grant application was submitted to the AHCPR. A secondary assignment to the National Cancer Institute (NCI) was requested because of the clinical preventive service delivery focus and the important potential of understanding family practice and competing demands for the subsequent design of interventions to enhance cancer prevention and control.

The initial application was favorably reviewed and received a priority score near the funding line. In response to advice from NCI and AHCPR program officers, the research team allowed the application to be considered for funding during 3 upcoming NCI council meetings. Regular letters to research network members kept them informed of the funding status. After 1 year of narrowly missing the funding line, the grant application was revised and resubmitted in response to the scientific review committee’s critique, with increased emphasis on the implications of the study for cancer prevention and control. It was funded by the NCI, with an additional grant from the Robert Wood Johnson Foundation Generalist Physician Faculty Scholars Program, to develop communication and clinician-initiated research in RAPP and additional methodologic and descriptive aims.

Planning and Conduct of Fieldwork

In 1994, more than 3 years after the idea was conceived, the board of directors of the 138-member research network (RAPP) was formally activated. The board’s 14 network volunteer members, several of whom helped develop the study and the network, were active in planning the practical implementation and refinement of the study protocol. When a board member suggested that they review the details of the direct observation measures before the study, the board concluded that as study participants they should not be involved in planning study measures, to avoid biasing their behavior during direct observation.

With funding, 2 logistic aspects of the study gained importance. First, it had been 2 years since the research network was formed. Whether physicians had retained their commitment to participate was unknown. That concern was laid to rest, however, when the vast majority of physicians expressed continued interest in the study. In retrospect, the 2-year struggle to obtain funding helped bond the network and create a sense of ownership and allegiance to the project.

The second major logistic issue was the need to recruit 8 research nurses. Job requirements included excellent interpersonal skills, sensitivity to the demands of real world community family practice, attention to detail in collecting reliable and valid quantitative data from multiple measures, an open-minded observational ability to simultaneously collect qualitative data, willingness to drive to multiple and sometimes distant sites, and interest in a 1-year job at a university salary. Hiring 8 nurses who could meet these requirements and start the 2-month training process on the same date seemed unrealistic at best. Yet, because of word of mouth advertising, the excitement generated by the study, the recent termination of another research project at the university, and the excellent reputation of the department, 8 highly qualified research nurses were found.

During their 8-week training, the research nurses were enlisted as true partners. They helped refine the research protocol and instruments, and items were added to the measures to reflect their interests. Using videotaped encounters, Dr Callahan instructed the nurses in applying the DOC, and they took the lead in adapting it for the study. As the immensity of the quantitative data collection requirements grew, Drs Miller and Crabtree scaled back the qualitative data collection protocol. They trained the research nurses in observational techniques and in how to dictate ethnographic field notes to record unanticipated findings, provide rich descriptions of quantitatively measured variables, and critique the study methods’ accuracy in capturing the phenomena under study.⁵⁰

Details of the data collection procedure have been reported elsewhere.^1,2 Briefly, teams of 2 research nurses spent 1 day observing patient care by the 138 participating RAPP members. One nurse obtained verbal informed consent from patients in the waiting room and distributed patient exit questionnaires. The other nurse accompanied the physician, directly observing consecutive visits by consenting patients and recording observations using the DOC and a direct observation checklist. Typically, the nurses exchanged roles after lunch. They returned on a subsequent day to perform medical record reviews for each other’s observed visits and to collect billing data. On the basis of observation and brief interviews with key informants, the research nurses completed a practice environment checklist. They dictated ethnographic field notes immediately after leaving the practice.¹⁰

During the course of the fieldwork, research team meetings were held every other week to coordinate logistics and assess and recalibrate inter-rater reliability using videotaped visits and medical records that were not part of the larger study. The high degree of inter-rater reliability achieved with this approach has been reported previously.¹

After data were collected from each physician, the board of directors met to review study progress and reassess the study protocol. The academic research team, including all consultants, also met to refine the protocol and plan the second round of physician visits. Initial plans called for ongoing analysis of the ethnographic field notes, but this proved to be infeasible because of their large volume and the study demands. However, at the study midpoint, Drs Crabtree and Miller independently analyzed the field notes using an immersion crystallization technique.⁵¹ Based on the richness of the information, they developed a template⁵² for gathering field notes during the second round of physician visits.

Data collection procedures were repeated, and each physician was visited a second time. The 4 months (on average) between visits helped assure that seasonal variations in health problems did not unduly affect the characterization of patient care. After the second data collection visit, physicians completed a detailed questionnaire.

Data Analysis and Production of Scholarly Output

The data were entered by optical scanner and manually verified. Quantitative data analyses were performed by Cleveland research team members in response to the initial research aims and additional questions raised by the research team and research network board. Qualitative data analyses were subcontracted to the University of Nebraska, with additional grant support from the American Academy of Family Physicians for more in-depth analyses.

Multiple papers were begun with diverse lead authorship. Preference in determining paper topics was given to methodologic manuscripts, topics with timely policy implications, and papers for which individual team members had a particular passion. In response to a call from the editorial office of JFP, a proposal for a theme issue on the DOPC Study was made and accepted. The opportunity to publish early scholarly output in one place greatly increased the potential for papers on diverse topics that would help cohesively describe several aspects of the value of family practice. The deadline for the theme issue also made the paper writing a high priority. Of 14 manuscripts accepted after going through peer review, 10 were included in the May 1998 issue of JFP,^{2,8-11,14,15,17,18,25,40} with one paper published in each of 4 subsequent issues.^13,16,26,27 Other analyses and papers have focused on the original research themes, new topics, more complex analyses, and expansion into the non–family practice literature.

Opportunities to propose paper topics have been extended to all study participants, including the academic research team, consultants, and RAPP members. Proposed topics are reviewed for feasibility and potential conflicts with other papers. The data set has spawned 2 masters theses^16,19 and one doctoral dissertation^3,4,12,24 and has led to new collaborations with complementary content experts.

Related Research Initiatives

Concurrent with the DOPC study, Dr Crabtree and his colleagues in Nebraska conducted a series of related inquiries.^{3-5,10,12,24,39,53-56} These studies have provided complementary information and advanced multimethod approaches for studying primary care practice. Close collaboration and open information sharing among the research teams and collaborators have greatly facilitated the discovery of new methods and insights into family practice and have furthered the research trajectory of the collaborating groups. These collaborations spawned the Center for Research in Family Practice and Primary Care, a multisite consortium funded by the American Academy of Family Physicians.

DOPC Study collaborations have led to other research initiatives as well. For example, a desire for more in-depth qualitative data led to a comparative case study of a smaller number of purposively selected practices in Nebraska, funded by the AHCPR with Dr Crabtree as principal investigator. In addition, after reviewing the initial findings of the first round of DOPC data, the RAPP board of directors developed a study of competing demands outside the examination room, which has led to related inquires.

Based on emerging insights from the DOPC Study on the competing demands of family practice, a competing continuation application was funded by the NCI for a trial to improve clinical preventive service delivery. The Study to Enhance Prevention by Understanding Practice (STEP-UP) was developed with input from the research team and the RAPP board of directors, with collaboration from family practice researchers at Dartmouth, led by Allen Dietrich.^57,58 Building on complexity theory–based insights from the DOPC Study,⁸ STEP-UP uses a multimethod practice assessment to understand the unique attributes of family practices and tailor intervention strategies. This approach increased preventive service delivery rates⁵⁹ and led to a more comprehensive assessment and improvement strategy that is being evaluated in the delayed intervention group. The participants include DOPC Study practices and new RAPP members.

Continuing efforts to develop the RAPP network have included free continuing medical education conferences for participants in practice-based research and quality improvement projects. An ongoing research network newsletter periodically publishes a 1-page Research Prospectus Worksheet* to encourage research ideas from RAPP members. The Cleveland research team provides rapid turnaround methodological consultation for study proposals, and those involving multiple practices are reviewed by the RAPP board of directors. In addition, RAPP members are encouraged to serve as authors on DOPC papers, and approximately half have provided internal peer review before submission of papers.

Several RAPP members have received external funding for their own research projects. These include studies of causes of bilateral leg edema in family practice,^60,61 an evaluation of a family-centered approach to diagnosis and treatment of respiratory infection, a clinical trial of therapeutic touch for carpal tunnel syndrome, and development of practical new methods for community-oriented primary care.⁶² A recent RAPP study, in collaboration with the NEON network, used a card study methodology to describe the “oh, by the way” phenomenon in which patients raise issues after the clinician thinks the outpatient visit is finished. In addition, the discovery of high rates of care of a secondary patient11 in the DOPC study led to an ASPN card study to elucidate the content of care provided to family members other than the identified patient for an outpatient visit.⁶³ An additional ASPN collaboration, using the Components of Primary Care Instrument^3,4 that was developed as part of the DOPC Study, examined the effect of different aspects of managed care on the delivery of 10 elements of quality primary care.⁶⁴

Lessons learned from the dopc process

Some of the lessons learned from the process of conducting the DOPC study are summarized in the Table 1. These lessons can be grouped into 6 categories, as follows.

A generalist perspective. A generalist perspective that places research questions in the context of the competing opportunities and complexity of family practice is needed for true family practice and primary care research.⁶⁵ Although this perspective is essential if we are to diminish the current chasm between discovery and practice, it has not been supported by those who fund research. One strategy for addressing this funding issue is to identify topics and multimethod approaches that allow simultaneous pursuit of both categorical and generalist perspectives.

Involvement. The involvement of community practices and practicing clinicians as partners is essential for research about primary care practice.^{66, 67} New knowledge from discoveries in the settings in which most people get most of their medical care will help end the dichotomy between research and dissemination. Practice-based research networks can help bridge this gap by asking and answering questions from the perspective and setting in which the findings will be applied.^68,69 (It is worth noting, however, that most successful research networks are built around a group of clinicians who are committed to conducting research in their practices. Developing a network around a particular study, as with the RAPP network, requires attention to fostering clinician ideas and nurturing relationships that extend beyond the initial study.) Greater involvement of nonclinician health care professionals, patients, and communities can also increase the relevance of research to meet the population’s health care needs.^67,70

Transdisciplinary team process. A transdisciplinary team process in which diverse specialized expertise is integrated toward a common goal can be a tremendous resource for innovation and productivity. Development of a transdisciplinary team is a long-term process that requires trust, shared vision, open leadership, idea sharing, and group meetings. In addition, team members with particular expertise must be willing to commit to creating new knowledge that transcends their disciplinary perspectives.⁷¹ Such collaboration creates the mentality of a bigger pie in which the size of each participants’ piece is increased, rather than a mentality of finite resources in which a bigger piece for one member creates a shortage for another.⁷²

Multimethodology. A multimethod approach in which quantitative and qualitative methods are integrated creates the opportunity to generate new methods, assure rigor, and maximize the efficiency of new discovery.^6,32,33,47 Multimethod approaches allow testing of a priori hypotheses while creating new understanding.

Openness. Openness to emerging insights is fostered by the generalist perspective, by participatory multimethod research approaches, and by building the project from pilot data and knowledge of previous work. In the DOPC study, openness to new methods led to the “Eureka!” moment of deciding to do direct observation. The involvement of clinician and nurse perspectives in study design and conduct and the inductive use of qualitative data to discover the relevance of complexity science to understanding and enhancing primary care practice also reflected the study’s openness to new approaches.

Thinking big, but starting small. This creates a larger vision that can guide and inspire individual decisions and creates an overall research trajectory built on incremental steps. The DOPC Study began with a large idea of improving practice. Grounding in real world practice led to development of innovative new methods to try to understand primary care practice and ongoing efforts to improve practice. These major undertakings, however, were built on a foundation of small pilot studies and multiple interactions among researchers and practicing family physicians.

Applying these insights to other studies may help to advance the generation of new knowledge about family practice and primary care.⁷³

Acknowledgments

This research was supported by grants from the National Cancer Institute (1R01 CA60962, 2R01 CA60962 and K24 CA81931), the Agency for Health Care Policy and Research (1R01 HS08776), the Ohio Academy of Family Physicians, the American Academy of Family Physicians, Generalist Physician Faculty Scholar Awards to Drs Stange and Jaén from the Robert Wood Johnson Foundation, and a Family Practice Research Center Grant from the American Academy of Family Physicians. The authors are grateful to the RAPP physicians, other clinicians, office staffs, and patients, without whose participation our study would not have been possible. We are also indebted to the many people who have participated and continue to participate in the genesis of related ideas and scholarly output that continues to emerge from the original study. Members of the DOPC Writing group also include: Authors from the Academic Research Team: Stephen J. Zyzanski, PhD, Department of Family Medicine, Case Western Reserve University, Cleveland, Ohio; Benjamin F. Crabtree, PhD, Department of Family Medicine, UMDNJ-RWJ Medical School, New Brunswick, NJ; William L. Miller, MD, MA, Department of Family Practice, Lehigh Valley Hospital, Allentown, Pa; Carlos Roberto Jaén, MD, PhD, Center for Urban Research in Primary Care, SUNY, Buffalo, NY; Susan A. Flocke, PhD, Department of Family Medicine, Case Western Reserve University, Cleveland, Ohio; Robert B. Kelly, MD, MS, Department of Family Practice, MetroHealth Medical Center, Cleveland, Ohio; William R. Gillanders, MD, Family Practice Residency Program, Sutter Health, Sacramento, Calif; Valerie Gilchrist, MD, Department of Family Practice, NorthEast Ohio Universities College of Medicine, Rootstown, Ohio; Jason Chao, MD, MS, Department of Family Medicine, Case Western Reserve University; J. Christopher Shank, MD, Methodist/Indiana University Family Practice Residency, Indianapolis, Ind; Daniel L. Dunn, PhD, Integrated Health Care Information Service, Cambridge, Mass; Jack H. Medalie, MD, MPH, Department of Family Medicine, Case Western Reserve University, Cleveland, Ohio; Doreen Langa, BA, American University School of Law, Washington, DC; Virginia Aita, PhD, Department of Family Practice, University of Nebraska Medical Center, Omaha; Meredith A. Goodwin, MS, Department of Family Medicine, Case Western Reserve University, Cleveland, Ohio; and Robin S. Gotler, MA, Department of Family Medicine, Case Western Reserve University, Cleveland, Ohio. Research Nurse Team Authors: Lisa B. Ballou, RN, FNP; Catherine M. Corrigan, RN; Luzmaria Jaén, RN; Sherry Patzke, RN; Frances F. Powers, RN; Kathleen L. Schneeberger, RN; Kelly Warner, RN; and Susan Zronek, RN. Authors from the RAPP Board of Directors: Robert Blankfield, MD; Henry Bloom, MD; Valerie Gilchrist, MD; Gwen Haas, MD; Patricia Kellner, MD; Sa Koo Lee, MD; Conrad Lindes, MD; Dennis McCluskey, MD; Thomas Mettee, MD; Albert Miller, MD; Michael Rabovsky, MD; and Archie Wilkinson, MD.