Paul A. James, MD

Between competing feelings of fatigue and satisfaction from a busy day of helping others, I am occasionally unsettled by a gnawing distraction: Did I miss anything today?

Most of us accept that some inherent uncertainty exists in the practice of medicine. Family physicians depend on minor obsessive-compulsive rituals to prevent subtle mistakes that may result in catastrophe. The Institute of Medicine released a report last year highlighting the large number of medical errors that shorten lives in the United States. However, family physicians are less likely to commit overt errors of commission, mistakenly taking a wrong action. Our errors are more subtle acts of omission: We forget, lose, misplace, or simply do not prioritize some piece of information that, in retrospect, should have changed our approach to a patient problem.

Optimal patient care requires the right action at the right time. Researchers in the United States spend significant resources attempting to decipher the right action for a multitude of health problems but fail to examine patient management issues in the context of longitudinal care for individuals. Errors occasionally occur because we do not take the right action, but more often we do not take the action at the right time, especially for chronic diseases.

Two recent studies provide evidence of potential primary care errors where the sense of immediacy may have been blunted by the steady march of time. Schootman and colleagues³ found that 14% of abnormal findings on breast cancer screening had inadequate follow-up. McBride and coworkers⁴ showed that documentation of appropriate management for cardiovascular disease risks (such as management of cholesterol >200mg/dL) was found in approximately 65% of charts. Although these studies do not specifically address medical errors in primary care practices, they do suggest that we should strengthen our office management procedures to reduce potential errors, especially those relating to follow-up of potential health problems.

In this issue of the Journal, Mold and colleagues⁵ report their examination of methods for managing laboratory information and offer an interesting glimpse into how practice-based research can be used to investigate common primary care problems where errors may occur. Ordering a laboratory test is the inciting event for cascading actions and reactions intended to provide clinical clarity, better patient health, and a satisfied clinician. However, too often we must contend with inadequate specimens, reports that do not return in their usual time frame, or the wrong laboratory test being done. In spite of these potential traps, we remain trusting of our abilities to avoid catastrophic medical outcomes.

Mold and coworkers explored the management and reporting of laboratory tests with the goal of finding best practices. The greatest asset of this study is its description of real world practice. The authors did not test for best practices. To test, one must rigidly control extraneous variables and compare a practice with other known strategies. The importance of this study lies in the direction it points and its intended destination. Mold and colleagues asked an important question, examined practices, and sought improvement through practice-based research.

Laboratory management

Mold and coworkers examined 4 steps in the management of laboratory testing. The first step is tracking tests until results are received. Developing a system that tracks physician orders, specimen collection, specimen transport and receipt, and return of results over time is vital to effective quality monitoring. The complex portion of this task appears to be what happens between the day of collection and receipt of results. The optimal practice in this study used a log in the laboratory with a second registry using billing data.

However, logbook information is isolated from the medical record and devoid of the pertinent patient information found there. Laboratory information management is really patient information management. The process of tracking information in the form of a question until it becomes an answer is a complex task. Seamless information systems that can link our office notes and plans to laboratory and radiological assessments are eagerly sought improvements to clinical practice.

Tracking information over time is difficult because no standard formats exist for how information should be organized. Laboratory management is similar to other pieces of data that we attempt to track. Examples include disease-specific measures, such as diabetic flow sheets and preventative strategies such as Papanicolaou tests, lipid testing, or documentation of smoking status. The problem is that tracking tools can distance us from patient-specific data that provide useful cues to improve effective care. Practicing primary care clinicians must eventually develop patient-specific information that is accessed at the right time.

Notifying patients of results is the second step in laboratory management. Mold and coworkers found that laboratory results can be most simply explained with a note on the laboratory report mailed to the patient. Patients expect information to be in the form of clear answers. Unfortunately, information obtained from laboratory testing may not always clarify a condition, and ambiguous laboratory results may confuse patients. Further research in primary care should seek to elucidate the determinants of high-quality communication. We should test other strategies for communicating results to patients. Allowing them an opportunity to participate in their own medical decisions has been shown beneficial.⁶ Despite these complexities, this study encourages us to simplify our processes to ensure that patients are notified.

The third step is to document the notification of the patient by placing the original laboratory report in the medical record. Perhaps more important than simple notification is placing the information in the context of a patient’s problem or life goals. Then we should document how patients understand and interpret the information we provide within the context of their goals.⁷

The fourth step, assuring that recommended follow-up for an abnormal test result occurs, had no best method. Physicians may assume that this step goes beyond their legal responsibility and that it is the patient’s responsibility to use the information provided. However, follow-up may be the shared responsibility of both the patient and the family physician. We must provide the information patients need to make good decisions. Then we need to document the decisions that patients make and how these decisions may change over time.

It’s about time

In family practice, we watch and wait. While we are waiting, other competing demands intervene. We may be distracted by a new problem, a patient’s reluctance to prioritize the problem, or we may simply forget to follow-up. We attempt to keep our clinical antennae tuned for potential hazards along the traditional diagnostic and therapeutic paths. Our habits remind us to always check twice or to call the laboratory if we remember that a result is tardy. Unfortunately, we depend on our own memory, because we simply do not know a better way.

We often treat time as our ally. It allows us the opportunity to study, revisit, and recheck. However, events can quickly turn, and time may become a formidable enemy. Management of information over time is central to quality systems in primary care. Mold and colleagues have begun the process of assessing our management of laboratory testing and finding opportunities for improvement. In their study, many times no system was in place. Even when a good system was present, 15% of laboratory tests ordered had no results found in the medical record.

It is time we develop systems to aid us in attaining high-quality patient care. We should realize that time is neither our friend nor our foe, but one more resource that we need to manage effectively to help our patients. Developing systems of information management that can retrieve information and remind us to perform certain tasks should be an important priority for future practice-based research.

Between competing feelings of fatigue and satisfaction from a busy day of helping others, I am occasionally unsettled by a gnawing distraction: Did I miss anything today?

Most of us accept that some inherent uncertainty exists in the practice of medicine. Family physicians depend on minor obsessive-compulsive rituals to prevent subtle mistakes that may result in catastrophe. The Institute of Medicine released a report last year highlighting the large number of medical errors that shorten lives in the United States. However, family physicians are less likely to commit overt errors of commission, mistakenly taking a wrong action. Our errors are more subtle acts of omission: We forget, lose, misplace, or simply do not prioritize some piece of information that, in retrospect, should have changed our approach to a patient problem.

Optimal patient care requires the right action at the right time. Researchers in the United States spend significant resources attempting to decipher the right action for a multitude of health problems but fail to examine patient management issues in the context of longitudinal care for individuals. Errors occasionally occur because we do not take the right action, but more often we do not take the action at the right time, especially for chronic diseases.

Two recent studies provide evidence of potential primary care errors where the sense of immediacy may have been blunted by the steady march of time. Schootman and colleagues³ found that 14% of abnormal findings on breast cancer screening had inadequate follow-up. McBride and coworkers⁴ showed that documentation of appropriate management for cardiovascular disease risks (such as management of cholesterol >200mg/dL) was found in approximately 65% of charts. Although these studies do not specifically address medical errors in primary care practices, they do suggest that we should strengthen our office management procedures to reduce potential errors, especially those relating to follow-up of potential health problems.

In this issue of the Journal, Mold and colleagues⁵ report their examination of methods for managing laboratory information and offer an interesting glimpse into how practice-based research can be used to investigate common primary care problems where errors may occur. Ordering a laboratory test is the inciting event for cascading actions and reactions intended to provide clinical clarity, better patient health, and a satisfied clinician. However, too often we must contend with inadequate specimens, reports that do not return in their usual time frame, or the wrong laboratory test being done. In spite of these potential traps, we remain trusting of our abilities to avoid catastrophic medical outcomes.

Mold and coworkers explored the management and reporting of laboratory tests with the goal of finding best practices. The greatest asset of this study is its description of real world practice. The authors did not test for best practices. To test, one must rigidly control extraneous variables and compare a practice with other known strategies. The importance of this study lies in the direction it points and its intended destination. Mold and colleagues asked an important question, examined practices, and sought improvement through practice-based research.

Laboratory management

Mold and coworkers examined 4 steps in the management of laboratory testing. The first step is tracking tests until results are received. Developing a system that tracks physician orders, specimen collection, specimen transport and receipt, and return of results over time is vital to effective quality monitoring. The complex portion of this task appears to be what happens between the day of collection and receipt of results. The optimal practice in this study used a log in the laboratory with a second registry using billing data.

However, logbook information is isolated from the medical record and devoid of the pertinent patient information found there. Laboratory information management is really patient information management. The process of tracking information in the form of a question until it becomes an answer is a complex task. Seamless information systems that can link our office notes and plans to laboratory and radiological assessments are eagerly sought improvements to clinical practice.

Tracking information over time is difficult because no standard formats exist for how information should be organized. Laboratory management is similar to other pieces of data that we attempt to track. Examples include disease-specific measures, such as diabetic flow sheets and preventative strategies such as Papanicolaou tests, lipid testing, or documentation of smoking status. The problem is that tracking tools can distance us from patient-specific data that provide useful cues to improve effective care. Practicing primary care clinicians must eventually develop patient-specific information that is accessed at the right time.

Notifying patients of results is the second step in laboratory management. Mold and coworkers found that laboratory results can be most simply explained with a note on the laboratory report mailed to the patient. Patients expect information to be in the form of clear answers. Unfortunately, information obtained from laboratory testing may not always clarify a condition, and ambiguous laboratory results may confuse patients. Further research in primary care should seek to elucidate the determinants of high-quality communication. We should test other strategies for communicating results to patients. Allowing them an opportunity to participate in their own medical decisions has been shown beneficial.⁶ Despite these complexities, this study encourages us to simplify our processes to ensure that patients are notified.

The third step is to document the notification of the patient by placing the original laboratory report in the medical record. Perhaps more important than simple notification is placing the information in the context of a patient’s problem or life goals. Then we should document how patients understand and interpret the information we provide within the context of their goals.⁷

The fourth step, assuring that recommended follow-up for an abnormal test result occurs, had no best method. Physicians may assume that this step goes beyond their legal responsibility and that it is the patient’s responsibility to use the information provided. However, follow-up may be the shared responsibility of both the patient and the family physician. We must provide the information patients need to make good decisions. Then we need to document the decisions that patients make and how these decisions may change over time.

It’s about time

In family practice, we watch and wait. While we are waiting, other competing demands intervene. We may be distracted by a new problem, a patient’s reluctance to prioritize the problem, or we may simply forget to follow-up. We attempt to keep our clinical antennae tuned for potential hazards along the traditional diagnostic and therapeutic paths. Our habits remind us to always check twice or to call the laboratory if we remember that a result is tardy. Unfortunately, we depend on our own memory, because we simply do not know a better way.

We often treat time as our ally. It allows us the opportunity to study, revisit, and recheck. However, events can quickly turn, and time may become a formidable enemy. Management of information over time is central to quality systems in primary care. Mold and colleagues have begun the process of assessing our management of laboratory testing and finding opportunities for improvement. In their study, many times no system was in place. Even when a good system was present, 15% of laboratory tests ordered had no results found in the medical record.

It is time we develop systems to aid us in attaining high-quality patient care. We should realize that time is neither our friend nor our foe, but one more resource that we need to manage effectively to help our patients. Developing systems of information management that can retrieve information and remind us to perform certain tasks should be an important priority for future practice-based research.

Between competing feelings of fatigue and satisfaction from a busy day of helping others, I am occasionally unsettled by a gnawing distraction: Did I miss anything today?

Most of us accept that some inherent uncertainty exists in the practice of medicine. Family physicians depend on minor obsessive-compulsive rituals to prevent subtle mistakes that may result in catastrophe. The Institute of Medicine released a report last year highlighting the large number of medical errors that shorten lives in the United States. However, family physicians are less likely to commit overt errors of commission, mistakenly taking a wrong action. Our errors are more subtle acts of omission: We forget, lose, misplace, or simply do not prioritize some piece of information that, in retrospect, should have changed our approach to a patient problem.

Optimal patient care requires the right action at the right time. Researchers in the United States spend significant resources attempting to decipher the right action for a multitude of health problems but fail to examine patient management issues in the context of longitudinal care for individuals. Errors occasionally occur because we do not take the right action, but more often we do not take the action at the right time, especially for chronic diseases.

Two recent studies provide evidence of potential primary care errors where the sense of immediacy may have been blunted by the steady march of time. Schootman and colleagues³ found that 14% of abnormal findings on breast cancer screening had inadequate follow-up. McBride and coworkers⁴ showed that documentation of appropriate management for cardiovascular disease risks (such as management of cholesterol >200mg/dL) was found in approximately 65% of charts. Although these studies do not specifically address medical errors in primary care practices, they do suggest that we should strengthen our office management procedures to reduce potential errors, especially those relating to follow-up of potential health problems.

In this issue of the Journal, Mold and colleagues⁵ report their examination of methods for managing laboratory information and offer an interesting glimpse into how practice-based research can be used to investigate common primary care problems where errors may occur. Ordering a laboratory test is the inciting event for cascading actions and reactions intended to provide clinical clarity, better patient health, and a satisfied clinician. However, too often we must contend with inadequate specimens, reports that do not return in their usual time frame, or the wrong laboratory test being done. In spite of these potential traps, we remain trusting of our abilities to avoid catastrophic medical outcomes.

Mold and coworkers explored the management and reporting of laboratory tests with the goal of finding best practices. The greatest asset of this study is its description of real world practice. The authors did not test for best practices. To test, one must rigidly control extraneous variables and compare a practice with other known strategies. The importance of this study lies in the direction it points and its intended destination. Mold and colleagues asked an important question, examined practices, and sought improvement through practice-based research.

Laboratory management

Mold and coworkers examined 4 steps in the management of laboratory testing. The first step is tracking tests until results are received. Developing a system that tracks physician orders, specimen collection, specimen transport and receipt, and return of results over time is vital to effective quality monitoring. The complex portion of this task appears to be what happens between the day of collection and receipt of results. The optimal practice in this study used a log in the laboratory with a second registry using billing data.

However, logbook information is isolated from the medical record and devoid of the pertinent patient information found there. Laboratory information management is really patient information management. The process of tracking information in the form of a question until it becomes an answer is a complex task. Seamless information systems that can link our office notes and plans to laboratory and radiological assessments are eagerly sought improvements to clinical practice.

Tracking information over time is difficult because no standard formats exist for how information should be organized. Laboratory management is similar to other pieces of data that we attempt to track. Examples include disease-specific measures, such as diabetic flow sheets and preventative strategies such as Papanicolaou tests, lipid testing, or documentation of smoking status. The problem is that tracking tools can distance us from patient-specific data that provide useful cues to improve effective care. Practicing primary care clinicians must eventually develop patient-specific information that is accessed at the right time.

Notifying patients of results is the second step in laboratory management. Mold and coworkers found that laboratory results can be most simply explained with a note on the laboratory report mailed to the patient. Patients expect information to be in the form of clear answers. Unfortunately, information obtained from laboratory testing may not always clarify a condition, and ambiguous laboratory results may confuse patients. Further research in primary care should seek to elucidate the determinants of high-quality communication. We should test other strategies for communicating results to patients. Allowing them an opportunity to participate in their own medical decisions has been shown beneficial.⁶ Despite these complexities, this study encourages us to simplify our processes to ensure that patients are notified.

The third step is to document the notification of the patient by placing the original laboratory report in the medical record. Perhaps more important than simple notification is placing the information in the context of a patient’s problem or life goals. Then we should document how patients understand and interpret the information we provide within the context of their goals.⁷

The fourth step, assuring that recommended follow-up for an abnormal test result occurs, had no best method. Physicians may assume that this step goes beyond their legal responsibility and that it is the patient’s responsibility to use the information provided. However, follow-up may be the shared responsibility of both the patient and the family physician. We must provide the information patients need to make good decisions. Then we need to document the decisions that patients make and how these decisions may change over time.

It’s about time

In family practice, we watch and wait. While we are waiting, other competing demands intervene. We may be distracted by a new problem, a patient’s reluctance to prioritize the problem, or we may simply forget to follow-up. We attempt to keep our clinical antennae tuned for potential hazards along the traditional diagnostic and therapeutic paths. Our habits remind us to always check twice or to call the laboratory if we remember that a result is tardy. Unfortunately, we depend on our own memory, because we simply do not know a better way.

We often treat time as our ally. It allows us the opportunity to study, revisit, and recheck. However, events can quickly turn, and time may become a formidable enemy. Management of information over time is central to quality systems in primary care. Mold and colleagues have begun the process of assessing our management of laboratory testing and finding opportunities for improvement. In their study, many times no system was in place. Even when a good system was present, 15% of laboratory tests ordered had no results found in the medical record.

It is time we develop systems to aid us in attaining high-quality patient care. We should realize that time is neither our friend nor our foe, but one more resource that we need to manage effectively to help our patients. Developing systems of information management that can retrieve information and remind us to perform certain tasks should be an important priority for future practice-based research.

Heart failure is a significant health problem in the United States for which primary care and specialist physicians provide care. Gross estimates suggest that more than 1 million hospitalizations and 400,000 new cases occur annually, at a cost of $10 billion.¹ Heart failure is a lethal condition with a mortality rate approaching 50% in 5 years.² Given the seriousness and prevalence of this condition and scientific evidence demonstrating reduced mortality with specific medical interventions, researchers have raised concerns about the care heart failure patients receive in primary care settings.^3-6

Evidence-based clinical practice guidelines were developed to educate physicians about appropriate processes of care.^1,7 Specifically, one guideline published by the Agency for Health Care Policy and Research (AHCPR) has been disseminated through pamphlets and published in the literature for primary care physicians.⁸ Yet, the extent of the dissemination and the effectiveness of applying the guideline in actual practice are unknown.⁹

The evaluation of clinical practice and measures of physician performance require appropriate translation of evidence-based clinical practice guidelines into explicit review criteria.¹⁰ Recommendations for review criteria for this clinical guideline have been published Table 1, and specific adherence rates have been recommended.^9,11 Few studies to date have examined the quality of care delivered in primary care settings using this rigorous methodology.¹²

We examined the quality of care provided to heart failure patients in upstate New York primary care offices. We measured quality using performance rates representing adherence to specific review criteria translated from the AHCPR heart failure clinical practice guideline. We studied 2 research questions: (1) How many heart failure patients in primary care settings found through claims data are actually eligible for measuring quality of care in accordance with an evidence-based guideline for systolic dysfunction? and (2) What are the adherence rates to specific measurable review criteria among this sample of primary care physicians?

Methods

Design and Sampling

We used a retrospective case review study design. Twenty-five practices from a practice-based research network in upstate New York voluntarily participated in a larger quality improvement project for heart failure. We solicited all physicians (n = 226) who had expressed interest in participating in a quality improvement program on a 1996 mail survey,¹³ as well as all physician members of the Western New York Practice-Based Research Network. Practices were selected according to practice location (urban, suburban, or rural), type of practice, order of receipt of physician-signed informed consent, and our goal of enrolling at least 400 patients with heart failure. Twenty-five physicians were selected from 35 respondents who signed informed consent forms. Their practices represented 47 physicians and 12 mid-level providers. We included solo (n = 9), group (n = 16), and hospital-affiliated (n = 5) practices with patient populations representing a broad case mix. They were located in rural (n = 13), urban (n = 4), and suburban (n = 8) sites.

Each practice was asked to provide a list of patients with the International Classification of Diseases (ICD-9-CM) code for congestive heart failure (428.00) from their billing database. Patient lists were generated, and primary care physicians were asked to review the lists to delete any names of deceased patients or those given a misdiagnosis. From 20 practices, every medical record of patients listed with heart failure was examined. In the 4 largest group practices and 1 solo practice, patients were systematically selected by ordering the patient list alphabetically and selecting every nth patient. To meet our goal of enrolling approximately 400 patients, 25 to 40 patients were selected from each of these 5 practices. These samples represented 19% to 55% of all patients on the lists provided by the practices. A total of 420 patients were selected.

Selection Criteria for Heart Failure Patients

Of those with a documented ICD-9-CM code 428.00, patients included were those with 3 or more office visits with heart failure documented in the assessment (suspected heart failure) and for whom another competing diagnosis for volume overload was not later determined (verified heart failure). Of the 420 patients with a billing diagnosis of heart failure, 25 had insufficient documentation in the medical record. Thus, data from 395 records were studied for compliance with the diagnostic criteria for heart failure, as these records reflected physician suspicion of the condition during at least 3 visits. Of these, an additional 57 patient charts were excluded, because an etiology for volume overload was found other than simple left ventricular failure. These records included those patients with valvular heart disease and those with volume overload due to noncardiac etiologies. Thus, we assessed 338 medical records documenting sustained management of heart failure by the primary care physician for their compliance with the treatment review criteria. We report this data for 2 subgroups according to left ventricular ejection fraction (LVEF) test result, to differentiate systolic heart failure from diastolic and unclassified heart failure.

Measures Used

Initially we developed 4 review criteria⁹ using a measurement validity method described by Palmer and colleagues.¹⁰ This method is a systematic and rigorous approach for translating guideline recommendations into measurable review criteria. We expanded our data collection to test 5 additional criteria recommended by researchers at RAND¹¹Table 1. Three of the 9 criteria focused on the diagnosis of heart failure and education; 6 measured pharmacologic management and monitoring of patients known to have systolic heart failure defined as an LVEF 40%. Two of these criteria (laboratory tests and patient counseling) had multiple measures, making a total of 17 adherence measures. Patients’ cardiac functional status was assessed by asking each primary physician to rate their patients using the New York Heart Association (NYHA) classification system.¹⁴ Responses were returned no later than October 1997.

Medical Record Review

In 1996, a separate study enabled the development and testing of the chart extraction form using 99 patients in 4 practices. Minor revisions were made on the basis of recommendations from the nurse chart extractors and the participating physicians. The revised medical record reviews took an average of 1 hour and 15 minutes to complete. They included any data found within the office medical record, such as medication lists, problem lists, progress notes, consultation letters, hospital discharge data, emergency department visits, laboratory results, radiographic data, and old records from other physicians.

The chart extractors first collected data related to the initial date of diagnosis. Next, they recorded all LVEF tests, emergency department visits, and hospitalizations that occurred between the date of initial diagnosis and the date of the medical record review. Finally, between either January 1, 1994, or the date of diagnosis (whichever date was more recent) and the date of the medical record review, all progress notes for office visits were examined for documentation of heart failure evaluation, medications prescribed, and test use. This period provided a potential for 3 years of follow-up. Chart review occurred from December 1996 through March 1997.

To evaluate consistency across reviewers, a second blinded record review was completed using 45 patient records selected randomly. The k statistic was estimated to assess interrater reliability for each review criterion. For these analyses, only the 8 measures (representing 5 criteria) with a k Ž0.4 were used for analyses of adherence. The highest ratios were for the measurement of LVEF (0.57), measurement of renal function (0.63), prescribed trial of angiotensin-converting enzyme (ACE) inhibitors (0.80), and ACE inhibitor at target dose (0.72). In addition to this assessment of data quality, the chart extraction manager evaluated each extraction form; any questionable or missing information was verified during a follow-up chart review.

Analyses

The unit of analysis for both research questions was at the patient level, and weighted performance rates are reported for the total population of heart failure patients in the 25 practices. Initial plans to analyze performance at the physician and practice levels were hindered by the small number of patients in each practice. Seven of the 25 practices had · 10 patients with evidence of heart failure; 13 practices had 11 to 25 patients. Recent evidence suggests that performance rates lack stability with such small numbers.¹⁵ Weighted adherence rates were calculated to adjust for the systematic sampling in the 5 larger practices.

For the first research question (How many heart failure patients found through claims data are actually eligible for measuring quality of care in accordance with an evidence-based guideline for systolic dysfunction?), we calculated the adjusted percentages of patients listed in the administrative datasets who had evidence of heart failure in the chart (more than 3 visits with documentation), had verified heart failure, and had confirmed systolic heart failure (LVEF · 40%). For the second research question (What are the adherence rates to specific measurable review criteria among this sample of primary care physicians?), we used several denominators for the various review criteria. For the 3 diagnostic criteria, we used the estimated total number of patients who had evidence of heart failure documented in their charts. For the first ACE inhibitor review criterion, we evaluated performance separately among the systolic heart failure patients and all other heart failure patients (those with a normal LVEF or no LVEF documented). For the second ACE inhibitor criterion (evaluating dosages), the denominator was the estimated number of patients taking the drug at the time of the chart review.

Results

Tests of 2 proportions were run for the 2 pharmacologic review criteria to determine if the performance rates were significantly higher in the systolic heart failure group than with all other heart failure patients. Also, chi-square tests and comparison-of-mean t tests were conducted to compare descriptors of heart failure presentation and specific comorbidities between these 2 groups (using the original unadjusted sample).

Patient Characteristics in Cohort

The average age of the patients was 76 years (± 11 years), with nearly one fourth of the sample aged 85 years or older. Although nearly half (48%) of the patients had been given their diagnosis less than 2.5 years before the chart-review period, almost one fourth (24%) had received the diagnosis more than 5 years ago. The prevalence of comorbidities was high among these patients. Eighty-six percent of the sample had one or more diseases associated with heart failure. Chronic obstructive pulmonary disease, diabetes, and arthritis were documented in approximately one third of the patient medical records (29%, 35%, and 32%, respectively).

The comparison of heart failure presentation descriptors and comorbidities is presented in Table 2. Systolic heart failure patients were younger at the time of diagnosis and were less likely to have arthritis or osteoporosis listed as a comorbidity. Although the prevalence of coronary artery disease was statistically similar in the 2 groups, significantly more (P = .01) of the systolic heart failure patients had a history of myocardial infarction (55% of patients with low LVEF vs 45% among the others). In all other comparisons, including NYHA functional classification, no difference was found between the groups.

Accuracy of Administrative Databases

Of 740 patients in the billing database with the ICD-9-CM code for heart failure, the adjusted number with suspected heart failure was 661 (89%). Only 572 (77%) had verified heart failure by clinical criteria. In the study sample, a low LVEF consistent with systolic heart failure was found for only 142 patients (37% of those with documented evidence of heart failure), though a normal LVEF was found for an equal number of patients (n = 145, 37%). Thus, we estimate only 31% of all patients labeled with heart failure in administrative databases in these 25 practices had documented systolic dysfunction.

Diagnostic Criteria

The adjusted performance rates based on our weighted sampling for the diagnostic review criteria are presented in Table 3. Within 3 months before or following diagnosis, 60% of all suspected heart failure patients (n = 661 estimated) had a measure of LVEF documented. This rate increased to 67% for the time interval of 6 months before or after diagnosis. When any time frame was disregarded, we found that 82% had an LVEF test documented in the medical record.

Higher adherence rates would have been found for use of LVEF if we had used more specific criteria. Of 72 charts reviewed that did not have any documentation of an LVEF measure,¹³ (18%) showed the test had been ordered but no documentation of the result. Physicians were queried about LVEF testing for the remaining patients. They reported another 10 (14%) patients had LVEF measures taken while in the hospital and 8 (11%) patients refused the test. For our study these 31 patients were all grouped in the nonadherence category. Those patients without any LVEF measurement were significantly older (82 vs 75 years; t test P< .001) and had fewer comorbidities (average = 2.6 vs 3.5; t test P <.001) than patients who had the test.

Measuring initial laboratory evaluation was complicated by uncertainty about the time of diagnosis, place of diagnosis, and the time frame chosen for compliance. Thus, several review criteria had low ks (< 40%) and were not reported. For laboratory evaluation, performance rates ranged from a low of 30% for thyroid function to a high of 72% for renal function tests. Documentation of patient education about diet changes was also low (21% compliance). However, low compliance should be interpreted with caution, as medical record review has been found to be unreliable in assessing patient education.¹⁶

Treatment Review Criteria

The performance rates for treatment review criteria are shown in Table 3. The adjusted rate of ACE inhibitor use in all patients with a diagnosis of heart failure was 74% (n = 421). We report data for compliance with each class of heart failure to illustrate the importance of documenting a low LVEF. The adjusted adherence rate was significantly higher for systolic heart failure patients (91%) than for patients with normal LVEFs or no LVEF measured (62%) (for the test of the difference between 2 population proportions z = 7.88, P <.001). The performance rates for achieving the target dosages were also significantly different in the 2 groups (z = -2.38; P <.01). Eighty-seven percent of the estimated number of systolic heart failure patients taking an ACE inhibitor at the time of the chart audit (n = 421 adjusted and estimated) were achieving the target dose, compared with 94% of the other patients. The 95% confidence interval for this performance rate in the systolic heart failure group did overlap the proposed standard of quality. Of the 267 patients in the initial sample who had been prescribed a trial of ACE inhibitors,³⁷ (11%) met exclusion criteria for not taking one at the time of the chart review.

Discussion

Three important observations come from our study. Heart failure patients in primary care are heterogeneous, with half of the patients having a normal LVEF. The demographic variables in this study and others¹⁷ suggest that primary care heart failure patients are older and have more comorbidities than participants in randomized clinical trials of ACE inhibitors.¹⁸ These findings make assessments of quality of care difficult at best and impossible with certain methodologies. Primary care physicians’ use of ACE inhibitors in systolic heart failure patients is higher here than reported in other published studies Table 4. Finally, the accurate assessment of quality for chronic disease management in primary care is dependent on the use of appropriate methods and measures sensitive to the longitudinal processes of care. Study time frames influence the accuracy of quality assessments.

Previous studies have suggested that the diagnosis of heart failure in primary care is substandard because of overdiagnosis.^6,19 Yet, research is hampered by a lack of specificity in classifying heart failure. ICD-9-CM codes do not currently account for the different classes of heart failure. Although misdiagnosis occurs in practice, our data suggest that the syndrome of heart failure is heterogeneous, with systolic and diastolic heart failure being equally prevalent. This finding is perhaps not surprising given the age of our cohort, and is consistent with other studies suggesting a high incidence of heart failure with normal LVEF.^17,20 Determining the veracity of the diagnosis is vital for measuring quality.

Current views of heart failure may be no more accurate than musings on presbycardia were 50 years ago.²¹ Since primary care heart failure patients are older and have more comorbidities than participants in randomized clinical trials, important questions are raised about the generalizability and effectiveness of interventions on the basis of our best scientific evidence.

In Table 4 we summarize other studies that have assessed physician performance with heart failure patients. The performance standard for LVEF measurement in our study is similar to that found in a recent outpatient study²² that used the same ICD-9-CM selection criteria. Three other outpatient studies^19,23,24 were completed in the United Kingdom, where access to LVEF tests may be more limited than in the United States. Our findings reiterate the importance of an early measure of LVEF to classify heart failure, because a physician’s use of ACE inhibitors was strongly associated with documentation of a low LVEF. Among the 18% of patients in the initial study sample who had no documented measure of LVEF, the performance rate for ACE inhibitor use (78%) was significantly lower than the threshold rate recommended for systolic heart failure patients. It is unlikely that clinical characteristics could distinguish between heart failure classes.³¹

Misclassification of patients with heart failure is an important concern, both clinically and within administrative databases, and is complicated by comorbidities and uncertainties about the disease process over time. Many studies that suggest low physician compliance with ACE inhibitor prescription did not classify heart failure patients according to LVEF status.^3,4,5,24 In our study and in most others that classified heart failure by LVEF, however, there were substantially higher rates of ACE inhibitor use for patients with systolic heart failure.^{25-28, 22} Also, the studies reporting the lowest compliance rates had less specific sample selection criteria than our study, using more ICD-9-CM codes.^4,5,29 Two studies suggesting low compliance were based on physician self-report.^3,5 Although the National Ambulatory Medical Care Survey provides insightful snapshots of physician practices, it may underestimate pharmaceutical use over time for chronic diseases; ACE inhibitors were not linked to measures of LVEF for heart failure.⁵ The questionnaire used by Edep³ and others to assess physician performance presented standard descriptions of heart failure patients with low LVEF but then asked physicians to respond on the basis of their population of heart failure patients, not the specific patient presented. According to our data, those primary care physicians may have accurately reflected their use of ACE inhibitors with their population of heart failure patients, since a large proportion may have had normal LVEFs.

Compared with studies that did classify by LVEF, the compliance of New York family physicians with the AHCPR clinical guideline recommendation for ACE inhibitor use is higher than that found in an academic setting²² and in 2 studies of Medicare patients hospitalized for heart failure.^27,29 Other studies^25,26,30 have examined ACE inhibitor use for heart failure patients by internists and cardiologists and also found lower rates of use in the hospital setting. Our study could reflect actual change in clinician behavior since the dissemination of the guideline. However, it more likely reflects the degree to which our study controlled for the classification of heart failure according to LVEF and the time frame for our observation.³¹ The overall compliance rate for use of ACE inhibitors was 74% among patients with heart failure, verified through chart review. Yet, among patients with a confirmed LVEF · 40% and corrected for patients with contraindications, the performance rate was 91%.

This research raises important questions about the measures and methods for studying quality in primary care. Our attempt to apply a rigorous method of guideline adherence measurement to primary care settings resulted in measurable review criteria that revealed the complexities of care over time. Despite the emergence of evidence-based medicine, there remains significant uncertainty in the day-to-day care of patients with chronic disease. The diagnostic uncertainty of systolic heart failure has been emphasized, but we would also emphasize that uncertainty surrounds the complex care of the elderly with multiple comorbidities. Our study did not address these issues. Moreover, the application of treatment efficacy studies from younger patients to effectiveness in primary care senior populations raises concerns about the external validity of these randomized clinical trials.

Also, cross-sectional assessments of quality miss the purpose and process of longitudinal physician-patient relationships and underestimate the potential for diagnoses and therapeutic approaches to evolve over time. Our criteria for accepting physician performance as appropriate were likely more lenient because of the time frame used for compliance. For example, the dose of an ACE inhibitor had to remain constant for at least 6 months before we declared that the target dose had not been reached. Our experience in primary care settings suggests that the appropriateness for certain interventions, such as medication changes, is highly time sensitive, yet we know little of the contributors to timing of interventions in primary care.

Limitations

Though the 25 practices represent different types of offices in rural, suburban, and urban sites, the generalizability of our findings is limited, and larger studies should replicate this work, specifically with respect to the clarification of the syndrome of heart failure in primary care. The representativeness of the participating practices should be questioned. These practices are participants in a practice-based research network and were more likely to teach medical students; therefore, they may have been more up-to-date about heart failure, assessing LVEF, and using ACE inhibitors. However, we analyzed data from an earlier adherence survey of physicians in New York State¹³ and found no difference in the use of guidelines or in physician knowledge of the heart failure clinical practice guideline by participants and nonparticipants.

For studies of this magnitude, errors in data collection and data entry are possible. Our quality checks reduced this bias, as did a re-examination of the medical records at a later date for changes in care. Questions also arise about the validity and reliability of the quality measures. The reliability of some of the review criteria is hindered by the complexities of physician decision making and the inadequacies in documentation. Although we measured multiple indicators of quality, only 2 were solid with good interrater reliability and scientific evidence supporting them.

Although assessing LVEF and ACE inhibitor prescriptions does approximate a standard of technical quality that evidence increasingly asserts improves patient outcomes, most physicians might argue that even these do not measure quality. Other activities, such as patient education about low-salt diets, exercise, and medication compliance, are important, but concerns about the quality of the data for these measures limit their utility for judging quality of care.¹⁶ Also, examination of patient care should better evaluate the causes of variability, especially patient and other nonclinical factors that might supersede the technical standards established. For example, other work has suggested that physicians are less likely to order an echocardiogram if patient-centered nondisease factors become a priority.³²

We believe the most reliable, accurate, and valid performance measures for systolic heart failure are those for pharmaceutic use and measures of LVEF, but the optimum time frame for observation requires further study. All other measures are suspect because of the variability of chart documentation, complexity of decision making, and timing of actions. Despite these difficulties, our study does establish benchmarks for comparison, and thus may serve as a foundation for others to attempt quality studies in primary care. Finally, our study does not answer the most important question of whether adherence to the guideline translates into improved outcomes.

We used review criteria translated from an evidence-based clinical guideline to evaluate the quality of care for primary care patients with heart failure in upstate New York. Primary care physicians should critically examine their practices for testing LVEF in patients suspected to have heart failure, as this appears below standard. Performance rates for ACE inhibitor use were above those noted in other studies and were acceptable for patients with documented systolic dysfunction. For patients who did not have a measure of LVEF documented, however, we noted lower quality of care as measured by this disease-specific guideline. Improved dosing of ACE inhibitors is needed to achieve target dosages in heart failure patients, while further study is needed to clarify the syndrome of heart failure in primary care settings.

Acknowledgments

This research was supported by the New York State Department of Health, Health Quality and Information Unit. It also was supported in part by a Family Practice Research Center grant from the American Academy of Family Physicians. We thank Erin Cleary for her assistance with manuscript preparation and Kevin Grumbach, MD, for reviewing an earlier version of the manuscript. We also thank the physicians and staff in the participating practices of the Western New York Practice Based Research Network, including: John Brewer, MD, and Roberta Gebhard, MD, Upper West Side Family Health Center; Arunas A. Budnikas, MD; Daniel Cracium, MD; Farmington Family Practice; John C. Dickinson, MD, and Ronald M. Epstein, MD, Highland Family Medicine Center; Paul A. Frame, MD, Tri-County Family Medicine; Samuel K. Gooldy, MD; Cynthia E. Hadley, MD; Karl F. Hafner, MD; Jeffrey L. Hanson, MD; David P. Haswell, MD, Bassett Healthcare; Peter Kowalski, MD; William R. Kuehling, MD; Herbert A. Laughlin, MD; Thomas P. Lawrence, MD, Cuba Family Health Center; Emanuel Y. Li, MD; North Rochester Family Medicine; David M. Newman, MD; Arun Patel, MD; Raghu Ram, MD; Richard J. Ruh, MD, Orchard Park Family Practice; Timothy Siepel, MD; William Stephan, MD; Louis Lazar Family Medicine Center; and Tri-County Family Medicine Faculty Associates.

Heart failure is a significant health problem in the United States for which primary care and specialist physicians provide care. Gross estimates suggest that more than 1 million hospitalizations and 400,000 new cases occur annually, at a cost of $10 billion.¹ Heart failure is a lethal condition with a mortality rate approaching 50% in 5 years.² Given the seriousness and prevalence of this condition and scientific evidence demonstrating reduced mortality with specific medical interventions, researchers have raised concerns about the care heart failure patients receive in primary care settings.^3-6

Evidence-based clinical practice guidelines were developed to educate physicians about appropriate processes of care.^1,7 Specifically, one guideline published by the Agency for Health Care Policy and Research (AHCPR) has been disseminated through pamphlets and published in the literature for primary care physicians.⁸ Yet, the extent of the dissemination and the effectiveness of applying the guideline in actual practice are unknown.⁹

The evaluation of clinical practice and measures of physician performance require appropriate translation of evidence-based clinical practice guidelines into explicit review criteria.¹⁰ Recommendations for review criteria for this clinical guideline have been published Table 1, and specific adherence rates have been recommended.^9,11 Few studies to date have examined the quality of care delivered in primary care settings using this rigorous methodology.¹²

We examined the quality of care provided to heart failure patients in upstate New York primary care offices. We measured quality using performance rates representing adherence to specific review criteria translated from the AHCPR heart failure clinical practice guideline. We studied 2 research questions: (1) How many heart failure patients in primary care settings found through claims data are actually eligible for measuring quality of care in accordance with an evidence-based guideline for systolic dysfunction? and (2) What are the adherence rates to specific measurable review criteria among this sample of primary care physicians?

Methods

Design and Sampling

We used a retrospective case review study design. Twenty-five practices from a practice-based research network in upstate New York voluntarily participated in a larger quality improvement project for heart failure. We solicited all physicians (n = 226) who had expressed interest in participating in a quality improvement program on a 1996 mail survey,¹³ as well as all physician members of the Western New York Practice-Based Research Network. Practices were selected according to practice location (urban, suburban, or rural), type of practice, order of receipt of physician-signed informed consent, and our goal of enrolling at least 400 patients with heart failure. Twenty-five physicians were selected from 35 respondents who signed informed consent forms. Their practices represented 47 physicians and 12 mid-level providers. We included solo (n = 9), group (n = 16), and hospital-affiliated (n = 5) practices with patient populations representing a broad case mix. They were located in rural (n = 13), urban (n = 4), and suburban (n = 8) sites.

Each practice was asked to provide a list of patients with the International Classification of Diseases (ICD-9-CM) code for congestive heart failure (428.00) from their billing database. Patient lists were generated, and primary care physicians were asked to review the lists to delete any names of deceased patients or those given a misdiagnosis. From 20 practices, every medical record of patients listed with heart failure was examined. In the 4 largest group practices and 1 solo practice, patients were systematically selected by ordering the patient list alphabetically and selecting every nth patient. To meet our goal of enrolling approximately 400 patients, 25 to 40 patients were selected from each of these 5 practices. These samples represented 19% to 55% of all patients on the lists provided by the practices. A total of 420 patients were selected.

Selection Criteria for Heart Failure Patients

Of those with a documented ICD-9-CM code 428.00, patients included were those with 3 or more office visits with heart failure documented in the assessment (suspected heart failure) and for whom another competing diagnosis for volume overload was not later determined (verified heart failure). Of the 420 patients with a billing diagnosis of heart failure, 25 had insufficient documentation in the medical record. Thus, data from 395 records were studied for compliance with the diagnostic criteria for heart failure, as these records reflected physician suspicion of the condition during at least 3 visits. Of these, an additional 57 patient charts were excluded, because an etiology for volume overload was found other than simple left ventricular failure. These records included those patients with valvular heart disease and those with volume overload due to noncardiac etiologies. Thus, we assessed 338 medical records documenting sustained management of heart failure by the primary care physician for their compliance with the treatment review criteria. We report this data for 2 subgroups according to left ventricular ejection fraction (LVEF) test result, to differentiate systolic heart failure from diastolic and unclassified heart failure.

Measures Used

Initially we developed 4 review criteria⁹ using a measurement validity method described by Palmer and colleagues.¹⁰ This method is a systematic and rigorous approach for translating guideline recommendations into measurable review criteria. We expanded our data collection to test 5 additional criteria recommended by researchers at RAND¹¹Table 1. Three of the 9 criteria focused on the diagnosis of heart failure and education; 6 measured pharmacologic management and monitoring of patients known to have systolic heart failure defined as an LVEF 40%. Two of these criteria (laboratory tests and patient counseling) had multiple measures, making a total of 17 adherence measures. Patients’ cardiac functional status was assessed by asking each primary physician to rate their patients using the New York Heart Association (NYHA) classification system.¹⁴ Responses were returned no later than October 1997.

Medical Record Review

In 1996, a separate study enabled the development and testing of the chart extraction form using 99 patients in 4 practices. Minor revisions were made on the basis of recommendations from the nurse chart extractors and the participating physicians. The revised medical record reviews took an average of 1 hour and 15 minutes to complete. They included any data found within the office medical record, such as medication lists, problem lists, progress notes, consultation letters, hospital discharge data, emergency department visits, laboratory results, radiographic data, and old records from other physicians.

The chart extractors first collected data related to the initial date of diagnosis. Next, they recorded all LVEF tests, emergency department visits, and hospitalizations that occurred between the date of initial diagnosis and the date of the medical record review. Finally, between either January 1, 1994, or the date of diagnosis (whichever date was more recent) and the date of the medical record review, all progress notes for office visits were examined for documentation of heart failure evaluation, medications prescribed, and test use. This period provided a potential for 3 years of follow-up. Chart review occurred from December 1996 through March 1997.

To evaluate consistency across reviewers, a second blinded record review was completed using 45 patient records selected randomly. The k statistic was estimated to assess interrater reliability for each review criterion. For these analyses, only the 8 measures (representing 5 criteria) with a k Ž0.4 were used for analyses of adherence. The highest ratios were for the measurement of LVEF (0.57), measurement of renal function (0.63), prescribed trial of angiotensin-converting enzyme (ACE) inhibitors (0.80), and ACE inhibitor at target dose (0.72). In addition to this assessment of data quality, the chart extraction manager evaluated each extraction form; any questionable or missing information was verified during a follow-up chart review.

Analyses

The unit of analysis for both research questions was at the patient level, and weighted performance rates are reported for the total population of heart failure patients in the 25 practices. Initial plans to analyze performance at the physician and practice levels were hindered by the small number of patients in each practice. Seven of the 25 practices had · 10 patients with evidence of heart failure; 13 practices had 11 to 25 patients. Recent evidence suggests that performance rates lack stability with such small numbers.¹⁵ Weighted adherence rates were calculated to adjust for the systematic sampling in the 5 larger practices.

For the first research question (How many heart failure patients found through claims data are actually eligible for measuring quality of care in accordance with an evidence-based guideline for systolic dysfunction?), we calculated the adjusted percentages of patients listed in the administrative datasets who had evidence of heart failure in the chart (more than 3 visits with documentation), had verified heart failure, and had confirmed systolic heart failure (LVEF · 40%). For the second research question (What are the adherence rates to specific measurable review criteria among this sample of primary care physicians?), we used several denominators for the various review criteria. For the 3 diagnostic criteria, we used the estimated total number of patients who had evidence of heart failure documented in their charts. For the first ACE inhibitor review criterion, we evaluated performance separately among the systolic heart failure patients and all other heart failure patients (those with a normal LVEF or no LVEF documented). For the second ACE inhibitor criterion (evaluating dosages), the denominator was the estimated number of patients taking the drug at the time of the chart review.

Results

Tests of 2 proportions were run for the 2 pharmacologic review criteria to determine if the performance rates were significantly higher in the systolic heart failure group than with all other heart failure patients. Also, chi-square tests and comparison-of-mean t tests were conducted to compare descriptors of heart failure presentation and specific comorbidities between these 2 groups (using the original unadjusted sample).

Patient Characteristics in Cohort

The average age of the patients was 76 years (± 11 years), with nearly one fourth of the sample aged 85 years or older. Although nearly half (48%) of the patients had been given their diagnosis less than 2.5 years before the chart-review period, almost one fourth (24%) had received the diagnosis more than 5 years ago. The prevalence of comorbidities was high among these patients. Eighty-six percent of the sample had one or more diseases associated with heart failure. Chronic obstructive pulmonary disease, diabetes, and arthritis were documented in approximately one third of the patient medical records (29%, 35%, and 32%, respectively).

The comparison of heart failure presentation descriptors and comorbidities is presented in Table 2. Systolic heart failure patients were younger at the time of diagnosis and were less likely to have arthritis or osteoporosis listed as a comorbidity. Although the prevalence of coronary artery disease was statistically similar in the 2 groups, significantly more (P = .01) of the systolic heart failure patients had a history of myocardial infarction (55% of patients with low LVEF vs 45% among the others). In all other comparisons, including NYHA functional classification, no difference was found between the groups.

Accuracy of Administrative Databases

Of 740 patients in the billing database with the ICD-9-CM code for heart failure, the adjusted number with suspected heart failure was 661 (89%). Only 572 (77%) had verified heart failure by clinical criteria. In the study sample, a low LVEF consistent with systolic heart failure was found for only 142 patients (37% of those with documented evidence of heart failure), though a normal LVEF was found for an equal number of patients (n = 145, 37%). Thus, we estimate only 31% of all patients labeled with heart failure in administrative databases in these 25 practices had documented systolic dysfunction.

Diagnostic Criteria

The adjusted performance rates based on our weighted sampling for the diagnostic review criteria are presented in Table 3. Within 3 months before or following diagnosis, 60% of all suspected heart failure patients (n = 661 estimated) had a measure of LVEF documented. This rate increased to 67% for the time interval of 6 months before or after diagnosis. When any time frame was disregarded, we found that 82% had an LVEF test documented in the medical record.

Higher adherence rates would have been found for use of LVEF if we had used more specific criteria. Of 72 charts reviewed that did not have any documentation of an LVEF measure,¹³ (18%) showed the test had been ordered but no documentation of the result. Physicians were queried about LVEF testing for the remaining patients. They reported another 10 (14%) patients had LVEF measures taken while in the hospital and 8 (11%) patients refused the test. For our study these 31 patients were all grouped in the nonadherence category. Those patients without any LVEF measurement were significantly older (82 vs 75 years; t test P< .001) and had fewer comorbidities (average = 2.6 vs 3.5; t test P <.001) than patients who had the test.

Measuring initial laboratory evaluation was complicated by uncertainty about the time of diagnosis, place of diagnosis, and the time frame chosen for compliance. Thus, several review criteria had low ks (< 40%) and were not reported. For laboratory evaluation, performance rates ranged from a low of 30% for thyroid function to a high of 72% for renal function tests. Documentation of patient education about diet changes was also low (21% compliance). However, low compliance should be interpreted with caution, as medical record review has been found to be unreliable in assessing patient education.¹⁶

Treatment Review Criteria

The performance rates for treatment review criteria are shown in Table 3. The adjusted rate of ACE inhibitor use in all patients with a diagnosis of heart failure was 74% (n = 421). We report data for compliance with each class of heart failure to illustrate the importance of documenting a low LVEF. The adjusted adherence rate was significantly higher for systolic heart failure patients (91%) than for patients with normal LVEFs or no LVEF measured (62%) (for the test of the difference between 2 population proportions z = 7.88, P <.001). The performance rates for achieving the target dosages were also significantly different in the 2 groups (z = -2.38; P <.01). Eighty-seven percent of the estimated number of systolic heart failure patients taking an ACE inhibitor at the time of the chart audit (n = 421 adjusted and estimated) were achieving the target dose, compared with 94% of the other patients. The 95% confidence interval for this performance rate in the systolic heart failure group did overlap the proposed standard of quality. Of the 267 patients in the initial sample who had been prescribed a trial of ACE inhibitors,³⁷ (11%) met exclusion criteria for not taking one at the time of the chart review.

Discussion

Three important observations come from our study. Heart failure patients in primary care are heterogeneous, with half of the patients having a normal LVEF. The demographic variables in this study and others¹⁷ suggest that primary care heart failure patients are older and have more comorbidities than participants in randomized clinical trials of ACE inhibitors.¹⁸ These findings make assessments of quality of care difficult at best and impossible with certain methodologies. Primary care physicians’ use of ACE inhibitors in systolic heart failure patients is higher here than reported in other published studies Table 4. Finally, the accurate assessment of quality for chronic disease management in primary care is dependent on the use of appropriate methods and measures sensitive to the longitudinal processes of care. Study time frames influence the accuracy of quality assessments.

Previous studies have suggested that the diagnosis of heart failure in primary care is substandard because of overdiagnosis.^6,19 Yet, research is hampered by a lack of specificity in classifying heart failure. ICD-9-CM codes do not currently account for the different classes of heart failure. Although misdiagnosis occurs in practice, our data suggest that the syndrome of heart failure is heterogeneous, with systolic and diastolic heart failure being equally prevalent. This finding is perhaps not surprising given the age of our cohort, and is consistent with other studies suggesting a high incidence of heart failure with normal LVEF.^17,20 Determining the veracity of the diagnosis is vital for measuring quality.

Current views of heart failure may be no more accurate than musings on presbycardia were 50 years ago.²¹ Since primary care heart failure patients are older and have more comorbidities than participants in randomized clinical trials, important questions are raised about the generalizability and effectiveness of interventions on the basis of our best scientific evidence.

In Table 4 we summarize other studies that have assessed physician performance with heart failure patients. The performance standard for LVEF measurement in our study is similar to that found in a recent outpatient study²² that used the same ICD-9-CM selection criteria. Three other outpatient studies^19,23,24 were completed in the United Kingdom, where access to LVEF tests may be more limited than in the United States. Our findings reiterate the importance of an early measure of LVEF to classify heart failure, because a physician’s use of ACE inhibitors was strongly associated with documentation of a low LVEF. Among the 18% of patients in the initial study sample who had no documented measure of LVEF, the performance rate for ACE inhibitor use (78%) was significantly lower than the threshold rate recommended for systolic heart failure patients. It is unlikely that clinical characteristics could distinguish between heart failure classes.³¹

Misclassification of patients with heart failure is an important concern, both clinically and within administrative databases, and is complicated by comorbidities and uncertainties about the disease process over time. Many studies that suggest low physician compliance with ACE inhibitor prescription did not classify heart failure patients according to LVEF status.^3,4,5,24 In our study and in most others that classified heart failure by LVEF, however, there were substantially higher rates of ACE inhibitor use for patients with systolic heart failure.^{25-28, 22} Also, the studies reporting the lowest compliance rates had less specific sample selection criteria than our study, using more ICD-9-CM codes.^4,5,29 Two studies suggesting low compliance were based on physician self-report.^3,5 Although the National Ambulatory Medical Care Survey provides insightful snapshots of physician practices, it may underestimate pharmaceutical use over time for chronic diseases; ACE inhibitors were not linked to measures of LVEF for heart failure.⁵ The questionnaire used by Edep³ and others to assess physician performance presented standard descriptions of heart failure patients with low LVEF but then asked physicians to respond on the basis of their population of heart failure patients, not the specific patient presented. According to our data, those primary care physicians may have accurately reflected their use of ACE inhibitors with their population of heart failure patients, since a large proportion may have had normal LVEFs.

Compared with studies that did classify by LVEF, the compliance of New York family physicians with the AHCPR clinical guideline recommendation for ACE inhibitor use is higher than that found in an academic setting²² and in 2 studies of Medicare patients hospitalized for heart failure.^27,29 Other studies^25,26,30 have examined ACE inhibitor use for heart failure patients by internists and cardiologists and also found lower rates of use in the hospital setting. Our study could reflect actual change in clinician behavior since the dissemination of the guideline. However, it more likely reflects the degree to which our study controlled for the classification of heart failure according to LVEF and the time frame for our observation.³¹ The overall compliance rate for use of ACE inhibitors was 74% among patients with heart failure, verified through chart review. Yet, among patients with a confirmed LVEF · 40% and corrected for patients with contraindications, the performance rate was 91%.

This research raises important questions about the measures and methods for studying quality in primary care. Our attempt to apply a rigorous method of guideline adherence measurement to primary care settings resulted in measurable review criteria that revealed the complexities of care over time. Despite the emergence of evidence-based medicine, there remains significant uncertainty in the day-to-day care of patients with chronic disease. The diagnostic uncertainty of systolic heart failure has been emphasized, but we would also emphasize that uncertainty surrounds the complex care of the elderly with multiple comorbidities. Our study did not address these issues. Moreover, the application of treatment efficacy studies from younger patients to effectiveness in primary care senior populations raises concerns about the external validity of these randomized clinical trials.

Also, cross-sectional assessments of quality miss the purpose and process of longitudinal physician-patient relationships and underestimate the potential for diagnoses and therapeutic approaches to evolve over time. Our criteria for accepting physician performance as appropriate were likely more lenient because of the time frame used for compliance. For example, the dose of an ACE inhibitor had to remain constant for at least 6 months before we declared that the target dose had not been reached. Our experience in primary care settings suggests that the appropriateness for certain interventions, such as medication changes, is highly time sensitive, yet we know little of the contributors to timing of interventions in primary care.

Limitations

Though the 25 practices represent different types of offices in rural, suburban, and urban sites, the generalizability of our findings is limited, and larger studies should replicate this work, specifically with respect to the clarification of the syndrome of heart failure in primary care. The representativeness of the participating practices should be questioned. These practices are participants in a practice-based research network and were more likely to teach medical students; therefore, they may have been more up-to-date about heart failure, assessing LVEF, and using ACE inhibitors. However, we analyzed data from an earlier adherence survey of physicians in New York State¹³ and found no difference in the use of guidelines or in physician knowledge of the heart failure clinical practice guideline by participants and nonparticipants.

For studies of this magnitude, errors in data collection and data entry are possible. Our quality checks reduced this bias, as did a re-examination of the medical records at a later date for changes in care. Questions also arise about the validity and reliability of the quality measures. The reliability of some of the review criteria is hindered by the complexities of physician decision making and the inadequacies in documentation. Although we measured multiple indicators of quality, only 2 were solid with good interrater reliability and scientific evidence supporting them.

Although assessing LVEF and ACE inhibitor prescriptions does approximate a standard of technical quality that evidence increasingly asserts improves patient outcomes, most physicians might argue that even these do not measure quality. Other activities, such as patient education about low-salt diets, exercise, and medication compliance, are important, but concerns about the quality of the data for these measures limit their utility for judging quality of care.¹⁶ Also, examination of patient care should better evaluate the causes of variability, especially patient and other nonclinical factors that might supersede the technical standards established. For example, other work has suggested that physicians are less likely to order an echocardiogram if patient-centered nondisease factors become a priority.³²

We believe the most reliable, accurate, and valid performance measures for systolic heart failure are those for pharmaceutic use and measures of LVEF, but the optimum time frame for observation requires further study. All other measures are suspect because of the variability of chart documentation, complexity of decision making, and timing of actions. Despite these difficulties, our study does establish benchmarks for comparison, and thus may serve as a foundation for others to attempt quality studies in primary care. Finally, our study does not answer the most important question of whether adherence to the guideline translates into improved outcomes.

We used review criteria translated from an evidence-based clinical guideline to evaluate the quality of care for primary care patients with heart failure in upstate New York. Primary care physicians should critically examine their practices for testing LVEF in patients suspected to have heart failure, as this appears below standard. Performance rates for ACE inhibitor use were above those noted in other studies and were acceptable for patients with documented systolic dysfunction. For patients who did not have a measure of LVEF documented, however, we noted lower quality of care as measured by this disease-specific guideline. Improved dosing of ACE inhibitors is needed to achieve target dosages in heart failure patients, while further study is needed to clarify the syndrome of heart failure in primary care settings.

Acknowledgments

This research was supported by the New York State Department of Health, Health Quality and Information Unit. It also was supported in part by a Family Practice Research Center grant from the American Academy of Family Physicians. We thank Erin Cleary for her assistance with manuscript preparation and Kevin Grumbach, MD, for reviewing an earlier version of the manuscript. We also thank the physicians and staff in the participating practices of the Western New York Practice Based Research Network, including: John Brewer, MD, and Roberta Gebhard, MD, Upper West Side Family Health Center; Arunas A. Budnikas, MD; Daniel Cracium, MD; Farmington Family Practice; John C. Dickinson, MD, and Ronald M. Epstein, MD, Highland Family Medicine Center; Paul A. Frame, MD, Tri-County Family Medicine; Samuel K. Gooldy, MD; Cynthia E. Hadley, MD; Karl F. Hafner, MD; Jeffrey L. Hanson, MD; David P. Haswell, MD, Bassett Healthcare; Peter Kowalski, MD; William R. Kuehling, MD; Herbert A. Laughlin, MD; Thomas P. Lawrence, MD, Cuba Family Health Center; Emanuel Y. Li, MD; North Rochester Family Medicine; David M. Newman, MD; Arun Patel, MD; Raghu Ram, MD; Richard J. Ruh, MD, Orchard Park Family Practice; Timothy Siepel, MD; William Stephan, MD; Louis Lazar Family Medicine Center; and Tri-County Family Medicine Faculty Associates.

Heart failure is a significant health problem in the United States for which primary care and specialist physicians provide care. Gross estimates suggest that more than 1 million hospitalizations and 400,000 new cases occur annually, at a cost of $10 billion.¹ Heart failure is a lethal condition with a mortality rate approaching 50% in 5 years.² Given the seriousness and prevalence of this condition and scientific evidence demonstrating reduced mortality with specific medical interventions, researchers have raised concerns about the care heart failure patients receive in primary care settings.^3-6

Evidence-based clinical practice guidelines were developed to educate physicians about appropriate processes of care.^1,7 Specifically, one guideline published by the Agency for Health Care Policy and Research (AHCPR) has been disseminated through pamphlets and published in the literature for primary care physicians.⁸ Yet, the extent of the dissemination and the effectiveness of applying the guideline in actual practice are unknown.⁹

The evaluation of clinical practice and measures of physician performance require appropriate translation of evidence-based clinical practice guidelines into explicit review criteria.¹⁰ Recommendations for review criteria for this clinical guideline have been published Table 1, and specific adherence rates have been recommended.^9,11 Few studies to date have examined the quality of care delivered in primary care settings using this rigorous methodology.¹²

We examined the quality of care provided to heart failure patients in upstate New York primary care offices. We measured quality using performance rates representing adherence to specific review criteria translated from the AHCPR heart failure clinical practice guideline. We studied 2 research questions: (1) How many heart failure patients in primary care settings found through claims data are actually eligible for measuring quality of care in accordance with an evidence-based guideline for systolic dysfunction? and (2) What are the adherence rates to specific measurable review criteria among this sample of primary care physicians?

Methods

Design and Sampling

We used a retrospective case review study design. Twenty-five practices from a practice-based research network in upstate New York voluntarily participated in a larger quality improvement project for heart failure. We solicited all physicians (n = 226) who had expressed interest in participating in a quality improvement program on a 1996 mail survey,¹³ as well as all physician members of the Western New York Practice-Based Research Network. Practices were selected according to practice location (urban, suburban, or rural), type of practice, order of receipt of physician-signed informed consent, and our goal of enrolling at least 400 patients with heart failure. Twenty-five physicians were selected from 35 respondents who signed informed consent forms. Their practices represented 47 physicians and 12 mid-level providers. We included solo (n = 9), group (n = 16), and hospital-affiliated (n = 5) practices with patient populations representing a broad case mix. They were located in rural (n = 13), urban (n = 4), and suburban (n = 8) sites.

Each practice was asked to provide a list of patients with the International Classification of Diseases (ICD-9-CM) code for congestive heart failure (428.00) from their billing database. Patient lists were generated, and primary care physicians were asked to review the lists to delete any names of deceased patients or those given a misdiagnosis. From 20 practices, every medical record of patients listed with heart failure was examined. In the 4 largest group practices and 1 solo practice, patients were systematically selected by ordering the patient list alphabetically and selecting every nth patient. To meet our goal of enrolling approximately 400 patients, 25 to 40 patients were selected from each of these 5 practices. These samples represented 19% to 55% of all patients on the lists provided by the practices. A total of 420 patients were selected.

Selection Criteria for Heart Failure Patients

Of those with a documented ICD-9-CM code 428.00, patients included were those with 3 or more office visits with heart failure documented in the assessment (suspected heart failure) and for whom another competing diagnosis for volume overload was not later determined (verified heart failure). Of the 420 patients with a billing diagnosis of heart failure, 25 had insufficient documentation in the medical record. Thus, data from 395 records were studied for compliance with the diagnostic criteria for heart failure, as these records reflected physician suspicion of the condition during at least 3 visits. Of these, an additional 57 patient charts were excluded, because an etiology for volume overload was found other than simple left ventricular failure. These records included those patients with valvular heart disease and those with volume overload due to noncardiac etiologies. Thus, we assessed 338 medical records documenting sustained management of heart failure by the primary care physician for their compliance with the treatment review criteria. We report this data for 2 subgroups according to left ventricular ejection fraction (LVEF) test result, to differentiate systolic heart failure from diastolic and unclassified heart failure.

Measures Used

Initially we developed 4 review criteria⁹ using a measurement validity method described by Palmer and colleagues.¹⁰ This method is a systematic and rigorous approach for translating guideline recommendations into measurable review criteria. We expanded our data collection to test 5 additional criteria recommended by researchers at RAND¹¹Table 1. Three of the 9 criteria focused on the diagnosis of heart failure and education; 6 measured pharmacologic management and monitoring of patients known to have systolic heart failure defined as an LVEF 40%. Two of these criteria (laboratory tests and patient counseling) had multiple measures, making a total of 17 adherence measures. Patients’ cardiac functional status was assessed by asking each primary physician to rate their patients using the New York Heart Association (NYHA) classification system.¹⁴ Responses were returned no later than October 1997.

Medical Record Review

In 1996, a separate study enabled the development and testing of the chart extraction form using 99 patients in 4 practices. Minor revisions were made on the basis of recommendations from the nurse chart extractors and the participating physicians. The revised medical record reviews took an average of 1 hour and 15 minutes to complete. They included any data found within the office medical record, such as medication lists, problem lists, progress notes, consultation letters, hospital discharge data, emergency department visits, laboratory results, radiographic data, and old records from other physicians.

The chart extractors first collected data related to the initial date of diagnosis. Next, they recorded all LVEF tests, emergency department visits, and hospitalizations that occurred between the date of initial diagnosis and the date of the medical record review. Finally, between either January 1, 1994, or the date of diagnosis (whichever date was more recent) and the date of the medical record review, all progress notes for office visits were examined for documentation of heart failure evaluation, medications prescribed, and test use. This period provided a potential for 3 years of follow-up. Chart review occurred from December 1996 through March 1997.

To evaluate consistency across reviewers, a second blinded record review was completed using 45 patient records selected randomly. The k statistic was estimated to assess interrater reliability for each review criterion. For these analyses, only the 8 measures (representing 5 criteria) with a k Ž0.4 were used for analyses of adherence. The highest ratios were for the measurement of LVEF (0.57), measurement of renal function (0.63), prescribed trial of angiotensin-converting enzyme (ACE) inhibitors (0.80), and ACE inhibitor at target dose (0.72). In addition to this assessment of data quality, the chart extraction manager evaluated each extraction form; any questionable or missing information was verified during a follow-up chart review.

Analyses

The unit of analysis for both research questions was at the patient level, and weighted performance rates are reported for the total population of heart failure patients in the 25 practices. Initial plans to analyze performance at the physician and practice levels were hindered by the small number of patients in each practice. Seven of the 25 practices had · 10 patients with evidence of heart failure; 13 practices had 11 to 25 patients. Recent evidence suggests that performance rates lack stability with such small numbers.¹⁵ Weighted adherence rates were calculated to adjust for the systematic sampling in the 5 larger practices.

For the first research question (How many heart failure patients found through claims data are actually eligible for measuring quality of care in accordance with an evidence-based guideline for systolic dysfunction?), we calculated the adjusted percentages of patients listed in the administrative datasets who had evidence of heart failure in the chart (more than 3 visits with documentation), had verified heart failure, and had confirmed systolic heart failure (LVEF · 40%). For the second research question (What are the adherence rates to specific measurable review criteria among this sample of primary care physicians?), we used several denominators for the various review criteria. For the 3 diagnostic criteria, we used the estimated total number of patients who had evidence of heart failure documented in their charts. For the first ACE inhibitor review criterion, we evaluated performance separately among the systolic heart failure patients and all other heart failure patients (those with a normal LVEF or no LVEF documented). For the second ACE inhibitor criterion (evaluating dosages), the denominator was the estimated number of patients taking the drug at the time of the chart review.

Results

Tests of 2 proportions were run for the 2 pharmacologic review criteria to determine if the performance rates were significantly higher in the systolic heart failure group than with all other heart failure patients. Also, chi-square tests and comparison-of-mean t tests were conducted to compare descriptors of heart failure presentation and specific comorbidities between these 2 groups (using the original unadjusted sample).

Patient Characteristics in Cohort

The average age of the patients was 76 years (± 11 years), with nearly one fourth of the sample aged 85 years or older. Although nearly half (48%) of the patients had been given their diagnosis less than 2.5 years before the chart-review period, almost one fourth (24%) had received the diagnosis more than 5 years ago. The prevalence of comorbidities was high among these patients. Eighty-six percent of the sample had one or more diseases associated with heart failure. Chronic obstructive pulmonary disease, diabetes, and arthritis were documented in approximately one third of the patient medical records (29%, 35%, and 32%, respectively).

The comparison of heart failure presentation descriptors and comorbidities is presented in Table 2. Systolic heart failure patients were younger at the time of diagnosis and were less likely to have arthritis or osteoporosis listed as a comorbidity. Although the prevalence of coronary artery disease was statistically similar in the 2 groups, significantly more (P = .01) of the systolic heart failure patients had a history of myocardial infarction (55% of patients with low LVEF vs 45% among the others). In all other comparisons, including NYHA functional classification, no difference was found between the groups.

Accuracy of Administrative Databases

Of 740 patients in the billing database with the ICD-9-CM code for heart failure, the adjusted number with suspected heart failure was 661 (89%). Only 572 (77%) had verified heart failure by clinical criteria. In the study sample, a low LVEF consistent with systolic heart failure was found for only 142 patients (37% of those with documented evidence of heart failure), though a normal LVEF was found for an equal number of patients (n = 145, 37%). Thus, we estimate only 31% of all patients labeled with heart failure in administrative databases in these 25 practices had documented systolic dysfunction.

Diagnostic Criteria

The adjusted performance rates based on our weighted sampling for the diagnostic review criteria are presented in Table 3. Within 3 months before or following diagnosis, 60% of all suspected heart failure patients (n = 661 estimated) had a measure of LVEF documented. This rate increased to 67% for the time interval of 6 months before or after diagnosis. When any time frame was disregarded, we found that 82% had an LVEF test documented in the medical record.

Higher adherence rates would have been found for use of LVEF if we had used more specific criteria. Of 72 charts reviewed that did not have any documentation of an LVEF measure,¹³ (18%) showed the test had been ordered but no documentation of the result. Physicians were queried about LVEF testing for the remaining patients. They reported another 10 (14%) patients had LVEF measures taken while in the hospital and 8 (11%) patients refused the test. For our study these 31 patients were all grouped in the nonadherence category. Those patients without any LVEF measurement were significantly older (82 vs 75 years; t test P< .001) and had fewer comorbidities (average = 2.6 vs 3.5; t test P <.001) than patients who had the test.

Measuring initial laboratory evaluation was complicated by uncertainty about the time of diagnosis, place of diagnosis, and the time frame chosen for compliance. Thus, several review criteria had low ks (< 40%) and were not reported. For laboratory evaluation, performance rates ranged from a low of 30% for thyroid function to a high of 72% for renal function tests. Documentation of patient education about diet changes was also low (21% compliance). However, low compliance should be interpreted with caution, as medical record review has been found to be unreliable in assessing patient education.¹⁶

Treatment Review Criteria

The performance rates for treatment review criteria are shown in Table 3. The adjusted rate of ACE inhibitor use in all patients with a diagnosis of heart failure was 74% (n = 421). We report data for compliance with each class of heart failure to illustrate the importance of documenting a low LVEF. The adjusted adherence rate was significantly higher for systolic heart failure patients (91%) than for patients with normal LVEFs or no LVEF measured (62%) (for the test of the difference between 2 population proportions z = 7.88, P <.001). The performance rates for achieving the target dosages were also significantly different in the 2 groups (z = -2.38; P <.01). Eighty-seven percent of the estimated number of systolic heart failure patients taking an ACE inhibitor at the time of the chart audit (n = 421 adjusted and estimated) were achieving the target dose, compared with 94% of the other patients. The 95% confidence interval for this performance rate in the systolic heart failure group did overlap the proposed standard of quality. Of the 267 patients in the initial sample who had been prescribed a trial of ACE inhibitors,³⁷ (11%) met exclusion criteria for not taking one at the time of the chart review.

Discussion

Three important observations come from our study. Heart failure patients in primary care are heterogeneous, with half of the patients having a normal LVEF. The demographic variables in this study and others¹⁷ suggest that primary care heart failure patients are older and have more comorbidities than participants in randomized clinical trials of ACE inhibitors.¹⁸ These findings make assessments of quality of care difficult at best and impossible with certain methodologies. Primary care physicians’ use of ACE inhibitors in systolic heart failure patients is higher here than reported in other published studies Table 4. Finally, the accurate assessment of quality for chronic disease management in primary care is dependent on the use of appropriate methods and measures sensitive to the longitudinal processes of care. Study time frames influence the accuracy of quality assessments.

Previous studies have suggested that the diagnosis of heart failure in primary care is substandard because of overdiagnosis.^6,19 Yet, research is hampered by a lack of specificity in classifying heart failure. ICD-9-CM codes do not currently account for the different classes of heart failure. Although misdiagnosis occurs in practice, our data suggest that the syndrome of heart failure is heterogeneous, with systolic and diastolic heart failure being equally prevalent. This finding is perhaps not surprising given the age of our cohort, and is consistent with other studies suggesting a high incidence of heart failure with normal LVEF.^17,20 Determining the veracity of the diagnosis is vital for measuring quality.

Current views of heart failure may be no more accurate than musings on presbycardia were 50 years ago.²¹ Since primary care heart failure patients are older and have more comorbidities than participants in randomized clinical trials, important questions are raised about the generalizability and effectiveness of interventions on the basis of our best scientific evidence.

In Table 4 we summarize other studies that have assessed physician performance with heart failure patients. The performance standard for LVEF measurement in our study is similar to that found in a recent outpatient study²² that used the same ICD-9-CM selection criteria. Three other outpatient studies^19,23,24 were completed in the United Kingdom, where access to LVEF tests may be more limited than in the United States. Our findings reiterate the importance of an early measure of LVEF to classify heart failure, because a physician’s use of ACE inhibitors was strongly associated with documentation of a low LVEF. Among the 18% of patients in the initial study sample who had no documented measure of LVEF, the performance rate for ACE inhibitor use (78%) was significantly lower than the threshold rate recommended for systolic heart failure patients. It is unlikely that clinical characteristics could distinguish between heart failure classes.³¹

Misclassification of patients with heart failure is an important concern, both clinically and within administrative databases, and is complicated by comorbidities and uncertainties about the disease process over time. Many studies that suggest low physician compliance with ACE inhibitor prescription did not classify heart failure patients according to LVEF status.^3,4,5,24 In our study and in most others that classified heart failure by LVEF, however, there were substantially higher rates of ACE inhibitor use for patients with systolic heart failure.^{25-28, 22} Also, the studies reporting the lowest compliance rates had less specific sample selection criteria than our study, using more ICD-9-CM codes.^4,5,29 Two studies suggesting low compliance were based on physician self-report.^3,5 Although the National Ambulatory Medical Care Survey provides insightful snapshots of physician practices, it may underestimate pharmaceutical use over time for chronic diseases; ACE inhibitors were not linked to measures of LVEF for heart failure.⁵ The questionnaire used by Edep³ and others to assess physician performance presented standard descriptions of heart failure patients with low LVEF but then asked physicians to respond on the basis of their population of heart failure patients, not the specific patient presented. According to our data, those primary care physicians may have accurately reflected their use of ACE inhibitors with their population of heart failure patients, since a large proportion may have had normal LVEFs.

Compared with studies that did classify by LVEF, the compliance of New York family physicians with the AHCPR clinical guideline recommendation for ACE inhibitor use is higher than that found in an academic setting²² and in 2 studies of Medicare patients hospitalized for heart failure.^27,29 Other studies^25,26,30 have examined ACE inhibitor use for heart failure patients by internists and cardiologists and also found lower rates of use in the hospital setting. Our study could reflect actual change in clinician behavior since the dissemination of the guideline. However, it more likely reflects the degree to which our study controlled for the classification of heart failure according to LVEF and the time frame for our observation.³¹ The overall compliance rate for use of ACE inhibitors was 74% among patients with heart failure, verified through chart review. Yet, among patients with a confirmed LVEF · 40% and corrected for patients with contraindications, the performance rate was 91%.

This research raises important questions about the measures and methods for studying quality in primary care. Our attempt to apply a rigorous method of guideline adherence measurement to primary care settings resulted in measurable review criteria that revealed the complexities of care over time. Despite the emergence of evidence-based medicine, there remains significant uncertainty in the day-to-day care of patients with chronic disease. The diagnostic uncertainty of systolic heart failure has been emphasized, but we would also emphasize that uncertainty surrounds the complex care of the elderly with multiple comorbidities. Our study did not address these issues. Moreover, the application of treatment efficacy studies from younger patients to effectiveness in primary care senior populations raises concerns about the external validity of these randomized clinical trials.

Also, cross-sectional assessments of quality miss the purpose and process of longitudinal physician-patient relationships and underestimate the potential for diagnoses and therapeutic approaches to evolve over time. Our criteria for accepting physician performance as appropriate were likely more lenient because of the time frame used for compliance. For example, the dose of an ACE inhibitor had to remain constant for at least 6 months before we declared that the target dose had not been reached. Our experience in primary care settings suggests that the appropriateness for certain interventions, such as medication changes, is highly time sensitive, yet we know little of the contributors to timing of interventions in primary care.

Limitations

Though the 25 practices represent different types of offices in rural, suburban, and urban sites, the generalizability of our findings is limited, and larger studies should replicate this work, specifically with respect to the clarification of the syndrome of heart failure in primary care. The representativeness of the participating practices should be questioned. These practices are participants in a practice-based research network and were more likely to teach medical students; therefore, they may have been more up-to-date about heart failure, assessing LVEF, and using ACE inhibitors. However, we analyzed data from an earlier adherence survey of physicians in New York State¹³ and found no difference in the use of guidelines or in physician knowledge of the heart failure clinical practice guideline by participants and nonparticipants.

For studies of this magnitude, errors in data collection and data entry are possible. Our quality checks reduced this bias, as did a re-examination of the medical records at a later date for changes in care. Questions also arise about the validity and reliability of the quality measures. The reliability of some of the review criteria is hindered by the complexities of physician decision making and the inadequacies in documentation. Although we measured multiple indicators of quality, only 2 were solid with good interrater reliability and scientific evidence supporting them.

Although assessing LVEF and ACE inhibitor prescriptions does approximate a standard of technical quality that evidence increasingly asserts improves patient outcomes, most physicians might argue that even these do not measure quality. Other activities, such as patient education about low-salt diets, exercise, and medication compliance, are important, but concerns about the quality of the data for these measures limit their utility for judging quality of care.¹⁶ Also, examination of patient care should better evaluate the causes of variability, especially patient and other nonclinical factors that might supersede the technical standards established. For example, other work has suggested that physicians are less likely to order an echocardiogram if patient-centered nondisease factors become a priority.³²

We believe the most reliable, accurate, and valid performance measures for systolic heart failure are those for pharmaceutic use and measures of LVEF, but the optimum time frame for observation requires further study. All other measures are suspect because of the variability of chart documentation, complexity of decision making, and timing of actions. Despite these difficulties, our study does establish benchmarks for comparison, and thus may serve as a foundation for others to attempt quality studies in primary care. Finally, our study does not answer the most important question of whether adherence to the guideline translates into improved outcomes.

We used review criteria translated from an evidence-based clinical guideline to evaluate the quality of care for primary care patients with heart failure in upstate New York. Primary care physicians should critically examine their practices for testing LVEF in patients suspected to have heart failure, as this appears below standard. Performance rates for ACE inhibitor use were above those noted in other studies and were acceptable for patients with documented systolic dysfunction. For patients who did not have a measure of LVEF documented, however, we noted lower quality of care as measured by this disease-specific guideline. Improved dosing of ACE inhibitors is needed to achieve target dosages in heart failure patients, while further study is needed to clarify the syndrome of heart failure in primary care settings.

Acknowledgments

This research was supported by the New York State Department of Health, Health Quality and Information Unit. It also was supported in part by a Family Practice Research Center grant from the American Academy of Family Physicians. We thank Erin Cleary for her assistance with manuscript preparation and Kevin Grumbach, MD, for reviewing an earlier version of the manuscript. We also thank the physicians and staff in the participating practices of the Western New York Practice Based Research Network, including: John Brewer, MD, and Roberta Gebhard, MD, Upper West Side Family Health Center; Arunas A. Budnikas, MD; Daniel Cracium, MD; Farmington Family Practice; John C. Dickinson, MD, and Ronald M. Epstein, MD, Highland Family Medicine Center; Paul A. Frame, MD, Tri-County Family Medicine; Samuel K. Gooldy, MD; Cynthia E. Hadley, MD; Karl F. Hafner, MD; Jeffrey L. Hanson, MD; David P. Haswell, MD, Bassett Healthcare; Peter Kowalski, MD; William R. Kuehling, MD; Herbert A. Laughlin, MD; Thomas P. Lawrence, MD, Cuba Family Health Center; Emanuel Y. Li, MD; North Rochester Family Medicine; David M. Newman, MD; Arun Patel, MD; Raghu Ram, MD; Richard J. Ruh, MD, Orchard Park Family Practice; Timothy Siepel, MD; William Stephan, MD; Louis Lazar Family Medicine Center; and Tri-County Family Medicine Faculty Associates.