Original Research

The¹³C-urea breath test provides accurate, noninvasive diagnosis of active Helicobacter pylori infection and can document posttherapy cure. This study evaluated point-of-care testing with onsite sample analysis with the use of a desktop infrared spectrophotometer. Ambulatory patients (N = 320) underwent ¹³C-urea breath testing, and breath samples were analyzed immediately by clinic staff with no prior breath testing experience. Duplicate samples were sent to a reference laboratory, and the results of both methods were compared. Point-of-care testing was simple, with an overall agreement of 99.1%. Accurate near-patient ¹³C-urea breath testing is now practical in the primary care setting even when done by inexperienced personnel.

Helicobacter pylori infection is etiologically associated with chronic gastritis, peptic ulcer disease, and gastric cancer.^1-5Helicobacter pylori infection is also a consideration in the evaluation of uninvestigated or undifferentiated dyspepsia.^4-7 The steps in the management of H pylori infection include diagnosis, choice of appropriate therapy, and confirmation of cure.^4,5,8 Diagnosis and confirmation of cure require diagnostic testing, and the recent consensus has been in the direction of noninvasive testing.^4,5

The urea breath test (UBT) is generally considered the clinical, gold standard, noninvasive test for detection of active H pylori infection.⁹ The intragastric hydrolysis of orally administered urea by H pylori urease produces a change in the isotopic ratio (¹³CO₂/¹²CO₂) in the breath.¹⁰ The ¹³C-UBT has been approved by the US Food and Drug Administration for pre- and posttherapy testing.^11,12

Recently the ¹³C-UBT has been shortened and simplified by the use of a citric acid test meal and elimination of the requirement of a 4-hour fast.¹³ Use of infrared mass spectrophotometry for measuring ¹³CO₂ enrichment in breath samples makes it theoretically possible to do office-based breath testing with onsite analysis.^14-16 Infrared spectrophotometry has been shown in comparative studies to be an accurate method for assessing ¹³CO₂ enrichment in breath.^17-19 No comparative studies of the 2 analytical methods have been done in the United States.

This study examined the utility of UBT testing by comparing infrared spectrophotometry with traditional gas isotope ratio mass spectrometry in the primary care environment. Our hypothesis was that the results obtained from the primary care clinics would be as accurate as those obtained from the commercial laboratory or the more experienced hospital-based clinical laboratory.

Methods

This was a multicenter, prospective study designed to compare a new infrared spectrophotometer (UBiTIR300, Otsuka Pharmaceuticals, Tokyo, Japan) for measuring ¹³CO₂ enrichment in breath with gas isotope ratio mass spectrometry (ABCA, Europa Scientific, Cheshire, UK). Subjects were recruited at the offices or clinics of 4 physicians’ including an indigent care primary care clinic, a hospital-based gastroenterology clinic, a private practice internal medicine office, an academic family medicine clinic, and a tertiary care clinical laboratory site. The study was done between July and September 2001. Consecutive patients were enrolled in the study if they expressed interest in participating and met the inclusion and exclusion criteria. Each site was provided with an infrared spectrophotometer, a breath gas transfer device, and commercially available UBT breath test collection kits (Meretek Diagnostics, Nashville, TN). Each kit contained a ¹³C-urea solution (125 mg in 75 mL of water), test meal pudding, and a specimen return box containing 4 bar-coded, evacuated 10-mL sample tubes. Before enrollment, all study personnel received approximately 1 hour of training in the performance of the test and use of the equipment.

Study procedures

Subjects included in the study were medically stable, ambulatory patients between 18 and 75 years old who were asymptomatic or experiencing dyspepsia. Potential subjects were excluded from study participation if they took bismuth preparations, antibiotics (ie, amoxicillin, tetracycline, metronidazole, clarithromycin, or azithromycin) or any anti-ulcer medication in dosages indicated for ulcer disease (ie, proton pump inhibitors, type 2 histamine blockers, or misoprostol) within 2 weeks before the study breath test. Exclusion criteria also included participation in a drug study within 4 weeks, treatment for eradication of H pylori within 4 weeks of the study breath test, or a history of gastric surgery or vagotomy for ulcer disease except simple closure of a gastric perforation.

The protocol was approved by the local Institutional Review Board for Human Studies, and all subjects provided written informed consent.

Testing began with a minimum 4-hour fast from solid food. Breath samples were obtained in disposable, balloonlike, breath collection bags designed for use with the infrared spectrophotometer. One sample was obtained immediately before ingestion of the ¹³C-urea test solution, and the second was collected 30 minutes after substrate ingestion. Paired sample aliquots were taken for separate analyses. Results from local infrared instruments were blinded to the central laboratory.

 

Statistical analyses

Each primary care site was asked to enroll subjects until 80 positive cases were identified from among all sites. The clinical laboratory site tested a minimum of 30 positive and 30 negative cases. The primary endpoint was the percentage of agreement (overall and within positive and negative cases separately) of results from both methods. Delta-over-baseline (DOB) enrichment values below 2.4 per mil were deemed negative and values greater than or equal to 2.4 per mil were deemed positive. The predicate reference method was gas isotope ratio mass spectrometry. Equivalence was defined by the percentage of agreement for positive and negative cases based on gas isotope ratio mass spectrometry results of at least 95%, and the lower limit of the 95% confidence interval was based on a percentage of agreement of at least 90% for positive cases.

Results

The primary care centers enrolled 258 subjects and the clinical laboratory enrolled 64 subjects, for a total enrollment of 322. The subjects’ mean age was 41.5 years (range, 18–70 years), with 88 black non-Hispanics, 106 Hispanics, 92 whites, 32 Asian/Pacific Islanders, and 4 other ethnic groups. There were 215 women and 107 men. Approximately 18% had active or previous gastrointestinal ailments, including previously diagnosed H pylori infection (11%) and peptic ulcer disease (3%).

There was excellent agreement between methods (Table), with an overall agreement of 99% (95% confidence interval, 97.3–99.7). Two subjects were excluded from the analysis of the primary and secondary endpoints because 1 or both assay values were missing. The data showed close correlation between methods among all sites (Figure). Evaluation by the personnel who performed the tests and analyses indicated that the office procedure was easy and nonintrusive.

There were 3 disagreements between the results obtained with the devices, 1 from the gastroenterology site and 2 from the clinical laboratory site. All results were near the cutoff value.

TABLE
Comparison of results between IRMS and GIRMS*

	GIRMS, %
IRMS, %	Negative	Positive	Total
Negative	1	115	116
Positive	202	2	204
Total	203	117	320
*Overall agreement, 99.1% (95% confidence interval, 97.3–99.7); positive agreement, 98.2% (95% confidence interval, 94.2–99.7); negative agreement, 99.5% (95% confidence interval, 97.3–99.9).
Kappa statistic = 0.98.
GIRMS, gas isotope ratio mass spectrometry; IRMS, infrared mass spectrophotometry.

FIGURE
Comparison of results between 2 methods of analysis of ¹³CO₂ breath sample enrichments

Discussion

¹³C-urea breath testing is an accurate diagnostic method for the detection of active H pylori infection^12,19-21 and point-of-care assessment of curative therapy. This study confirmed the hypothesis that the infrared instrument is an easy to operate alternative to the original sendout analyses. Rapid turnaround allows for decisions regarding therapy to be made at the time of care. The currently approved UBT in the United States does not require fasting from solid food for longer than 1 hour.

Noninvasive alternatives to UBT include serology and stool antigen testing. Serology assays cannot discriminate between active and recent past infections. Stool antigen testing requires patient compliance with specimen collection and is a sendout test. In general, although studies using pretreatment stool antigen tests have shown sensitivity and specificity comparable to those of histology or UBT, it has become evident that there can be considerable lot-to-lot variation in stool antigen tests.²² The most likely explanation is that the polyclonal serum used for the capture antibody is obtained from rabbits and thus difficult to standardize.²³ Stool antigen testing has also proven to be less reliable when used soon after the end of therapy, and it is now generally recommended that one must wait 6 or 8 weeks after therapy when using the stool antigen test to confirm eradication. For example, a recent study had a false negative rate of 12.5% (95% confidence interval, 1.5–33).²⁴Recent recommendations are that the UBT is preferred where available.⁴

The US Food and Drug Administration recently cleared the UBiT-IR300 instrument for use with the commercial ¹³C-UBT. The costs of the UBT are but a fraction of those of endoscopy, not including indirect patient costs. Office-based testing has a separate reimbursement for testing, and overall the costs appear less than those of the stool antigen test. Economic impact studies comparing the tests are planned. Office-based infrared analysis for ¹³C makes near-patient or point-of-care UBT and analysis practical and should make accurate diagnosis of active H pylori infection readily available.

Acknowledgments

The authors thank the clinical study coordinators, including Ms Rebecca Garza, Ms Flora Godard, and Mr Zachary Patton for their conscientious efforts.

The¹³C-urea breath test provides accurate, noninvasive diagnosis of active Helicobacter pylori infection and can document posttherapy cure. This study evaluated point-of-care testing with onsite sample analysis with the use of a desktop infrared spectrophotometer. Ambulatory patients (N = 320) underwent ¹³C-urea breath testing, and breath samples were analyzed immediately by clinic staff with no prior breath testing experience. Duplicate samples were sent to a reference laboratory, and the results of both methods were compared. Point-of-care testing was simple, with an overall agreement of 99.1%. Accurate near-patient ¹³C-urea breath testing is now practical in the primary care setting even when done by inexperienced personnel.

Helicobacter pylori infection is etiologically associated with chronic gastritis, peptic ulcer disease, and gastric cancer.^1-5Helicobacter pylori infection is also a consideration in the evaluation of uninvestigated or undifferentiated dyspepsia.^4-7 The steps in the management of H pylori infection include diagnosis, choice of appropriate therapy, and confirmation of cure.^4,5,8 Diagnosis and confirmation of cure require diagnostic testing, and the recent consensus has been in the direction of noninvasive testing.^4,5

The urea breath test (UBT) is generally considered the clinical, gold standard, noninvasive test for detection of active H pylori infection.⁹ The intragastric hydrolysis of orally administered urea by H pylori urease produces a change in the isotopic ratio (¹³CO₂/¹²CO₂) in the breath.¹⁰ The ¹³C-UBT has been approved by the US Food and Drug Administration for pre- and posttherapy testing.^11,12

Recently the ¹³C-UBT has been shortened and simplified by the use of a citric acid test meal and elimination of the requirement of a 4-hour fast.¹³ Use of infrared mass spectrophotometry for measuring ¹³CO₂ enrichment in breath samples makes it theoretically possible to do office-based breath testing with onsite analysis.^14-16 Infrared spectrophotometry has been shown in comparative studies to be an accurate method for assessing ¹³CO₂ enrichment in breath.^17-19 No comparative studies of the 2 analytical methods have been done in the United States.

This study examined the utility of UBT testing by comparing infrared spectrophotometry with traditional gas isotope ratio mass spectrometry in the primary care environment. Our hypothesis was that the results obtained from the primary care clinics would be as accurate as those obtained from the commercial laboratory or the more experienced hospital-based clinical laboratory.

Methods

This was a multicenter, prospective study designed to compare a new infrared spectrophotometer (UBiTIR300, Otsuka Pharmaceuticals, Tokyo, Japan) for measuring ¹³CO₂ enrichment in breath with gas isotope ratio mass spectrometry (ABCA, Europa Scientific, Cheshire, UK). Subjects were recruited at the offices or clinics of 4 physicians’ including an indigent care primary care clinic, a hospital-based gastroenterology clinic, a private practice internal medicine office, an academic family medicine clinic, and a tertiary care clinical laboratory site. The study was done between July and September 2001. Consecutive patients were enrolled in the study if they expressed interest in participating and met the inclusion and exclusion criteria. Each site was provided with an infrared spectrophotometer, a breath gas transfer device, and commercially available UBT breath test collection kits (Meretek Diagnostics, Nashville, TN). Each kit contained a ¹³C-urea solution (125 mg in 75 mL of water), test meal pudding, and a specimen return box containing 4 bar-coded, evacuated 10-mL sample tubes. Before enrollment, all study personnel received approximately 1 hour of training in the performance of the test and use of the equipment.

Study procedures

Subjects included in the study were medically stable, ambulatory patients between 18 and 75 years old who were asymptomatic or experiencing dyspepsia. Potential subjects were excluded from study participation if they took bismuth preparations, antibiotics (ie, amoxicillin, tetracycline, metronidazole, clarithromycin, or azithromycin) or any anti-ulcer medication in dosages indicated for ulcer disease (ie, proton pump inhibitors, type 2 histamine blockers, or misoprostol) within 2 weeks before the study breath test. Exclusion criteria also included participation in a drug study within 4 weeks, treatment for eradication of H pylori within 4 weeks of the study breath test, or a history of gastric surgery or vagotomy for ulcer disease except simple closure of a gastric perforation.

The protocol was approved by the local Institutional Review Board for Human Studies, and all subjects provided written informed consent.

Testing began with a minimum 4-hour fast from solid food. Breath samples were obtained in disposable, balloonlike, breath collection bags designed for use with the infrared spectrophotometer. One sample was obtained immediately before ingestion of the ¹³C-urea test solution, and the second was collected 30 minutes after substrate ingestion. Paired sample aliquots were taken for separate analyses. Results from local infrared instruments were blinded to the central laboratory.

 

Statistical analyses

Each primary care site was asked to enroll subjects until 80 positive cases were identified from among all sites. The clinical laboratory site tested a minimum of 30 positive and 30 negative cases. The primary endpoint was the percentage of agreement (overall and within positive and negative cases separately) of results from both methods. Delta-over-baseline (DOB) enrichment values below 2.4 per mil were deemed negative and values greater than or equal to 2.4 per mil were deemed positive. The predicate reference method was gas isotope ratio mass spectrometry. Equivalence was defined by the percentage of agreement for positive and negative cases based on gas isotope ratio mass spectrometry results of at least 95%, and the lower limit of the 95% confidence interval was based on a percentage of agreement of at least 90% for positive cases.

Results

The primary care centers enrolled 258 subjects and the clinical laboratory enrolled 64 subjects, for a total enrollment of 322. The subjects’ mean age was 41.5 years (range, 18–70 years), with 88 black non-Hispanics, 106 Hispanics, 92 whites, 32 Asian/Pacific Islanders, and 4 other ethnic groups. There were 215 women and 107 men. Approximately 18% had active or previous gastrointestinal ailments, including previously diagnosed H pylori infection (11%) and peptic ulcer disease (3%).

There was excellent agreement between methods (Table), with an overall agreement of 99% (95% confidence interval, 97.3–99.7). Two subjects were excluded from the analysis of the primary and secondary endpoints because 1 or both assay values were missing. The data showed close correlation between methods among all sites (Figure). Evaluation by the personnel who performed the tests and analyses indicated that the office procedure was easy and nonintrusive.

There were 3 disagreements between the results obtained with the devices, 1 from the gastroenterology site and 2 from the clinical laboratory site. All results were near the cutoff value.

TABLE
Comparison of results between IRMS and GIRMS*

	GIRMS, %
IRMS, %	Negative	Positive	Total
Negative	1	115	116
Positive	202	2	204
Total	203	117	320
*Overall agreement, 99.1% (95% confidence interval, 97.3–99.7); positive agreement, 98.2% (95% confidence interval, 94.2–99.7); negative agreement, 99.5% (95% confidence interval, 97.3–99.9).
Kappa statistic = 0.98.
GIRMS, gas isotope ratio mass spectrometry; IRMS, infrared mass spectrophotometry.

FIGURE
Comparison of results between 2 methods of analysis of ¹³CO₂ breath sample enrichments

Discussion

¹³C-urea breath testing is an accurate diagnostic method for the detection of active H pylori infection^12,19-21 and point-of-care assessment of curative therapy. This study confirmed the hypothesis that the infrared instrument is an easy to operate alternative to the original sendout analyses. Rapid turnaround allows for decisions regarding therapy to be made at the time of care. The currently approved UBT in the United States does not require fasting from solid food for longer than 1 hour.

Noninvasive alternatives to UBT include serology and stool antigen testing. Serology assays cannot discriminate between active and recent past infections. Stool antigen testing requires patient compliance with specimen collection and is a sendout test. In general, although studies using pretreatment stool antigen tests have shown sensitivity and specificity comparable to those of histology or UBT, it has become evident that there can be considerable lot-to-lot variation in stool antigen tests.²² The most likely explanation is that the polyclonal serum used for the capture antibody is obtained from rabbits and thus difficult to standardize.²³ Stool antigen testing has also proven to be less reliable when used soon after the end of therapy, and it is now generally recommended that one must wait 6 or 8 weeks after therapy when using the stool antigen test to confirm eradication. For example, a recent study had a false negative rate of 12.5% (95% confidence interval, 1.5–33).²⁴Recent recommendations are that the UBT is preferred where available.⁴

The US Food and Drug Administration recently cleared the UBiT-IR300 instrument for use with the commercial ¹³C-UBT. The costs of the UBT are but a fraction of those of endoscopy, not including indirect patient costs. Office-based testing has a separate reimbursement for testing, and overall the costs appear less than those of the stool antigen test. Economic impact studies comparing the tests are planned. Office-based infrared analysis for ¹³C makes near-patient or point-of-care UBT and analysis practical and should make accurate diagnosis of active H pylori infection readily available.

Acknowledgments

The authors thank the clinical study coordinators, including Ms Rebecca Garza, Ms Flora Godard, and Mr Zachary Patton for their conscientious efforts.

The¹³C-urea breath test provides accurate, noninvasive diagnosis of active Helicobacter pylori infection and can document posttherapy cure. This study evaluated point-of-care testing with onsite sample analysis with the use of a desktop infrared spectrophotometer. Ambulatory patients (N = 320) underwent ¹³C-urea breath testing, and breath samples were analyzed immediately by clinic staff with no prior breath testing experience. Duplicate samples were sent to a reference laboratory, and the results of both methods were compared. Point-of-care testing was simple, with an overall agreement of 99.1%. Accurate near-patient ¹³C-urea breath testing is now practical in the primary care setting even when done by inexperienced personnel.

Helicobacter pylori infection is etiologically associated with chronic gastritis, peptic ulcer disease, and gastric cancer.^1-5Helicobacter pylori infection is also a consideration in the evaluation of uninvestigated or undifferentiated dyspepsia.^4-7 The steps in the management of H pylori infection include diagnosis, choice of appropriate therapy, and confirmation of cure.^4,5,8 Diagnosis and confirmation of cure require diagnostic testing, and the recent consensus has been in the direction of noninvasive testing.^4,5

The urea breath test (UBT) is generally considered the clinical, gold standard, noninvasive test for detection of active H pylori infection.⁹ The intragastric hydrolysis of orally administered urea by H pylori urease produces a change in the isotopic ratio (¹³CO₂/¹²CO₂) in the breath.¹⁰ The ¹³C-UBT has been approved by the US Food and Drug Administration for pre- and posttherapy testing.^11,12

Recently the ¹³C-UBT has been shortened and simplified by the use of a citric acid test meal and elimination of the requirement of a 4-hour fast.¹³ Use of infrared mass spectrophotometry for measuring ¹³CO₂ enrichment in breath samples makes it theoretically possible to do office-based breath testing with onsite analysis.^14-16 Infrared spectrophotometry has been shown in comparative studies to be an accurate method for assessing ¹³CO₂ enrichment in breath.^17-19 No comparative studies of the 2 analytical methods have been done in the United States.

This study examined the utility of UBT testing by comparing infrared spectrophotometry with traditional gas isotope ratio mass spectrometry in the primary care environment. Our hypothesis was that the results obtained from the primary care clinics would be as accurate as those obtained from the commercial laboratory or the more experienced hospital-based clinical laboratory.

Methods

This was a multicenter, prospective study designed to compare a new infrared spectrophotometer (UBiTIR300, Otsuka Pharmaceuticals, Tokyo, Japan) for measuring ¹³CO₂ enrichment in breath with gas isotope ratio mass spectrometry (ABCA, Europa Scientific, Cheshire, UK). Subjects were recruited at the offices or clinics of 4 physicians’ including an indigent care primary care clinic, a hospital-based gastroenterology clinic, a private practice internal medicine office, an academic family medicine clinic, and a tertiary care clinical laboratory site. The study was done between July and September 2001. Consecutive patients were enrolled in the study if they expressed interest in participating and met the inclusion and exclusion criteria. Each site was provided with an infrared spectrophotometer, a breath gas transfer device, and commercially available UBT breath test collection kits (Meretek Diagnostics, Nashville, TN). Each kit contained a ¹³C-urea solution (125 mg in 75 mL of water), test meal pudding, and a specimen return box containing 4 bar-coded, evacuated 10-mL sample tubes. Before enrollment, all study personnel received approximately 1 hour of training in the performance of the test and use of the equipment.

Study procedures

Subjects included in the study were medically stable, ambulatory patients between 18 and 75 years old who were asymptomatic or experiencing dyspepsia. Potential subjects were excluded from study participation if they took bismuth preparations, antibiotics (ie, amoxicillin, tetracycline, metronidazole, clarithromycin, or azithromycin) or any anti-ulcer medication in dosages indicated for ulcer disease (ie, proton pump inhibitors, type 2 histamine blockers, or misoprostol) within 2 weeks before the study breath test. Exclusion criteria also included participation in a drug study within 4 weeks, treatment for eradication of H pylori within 4 weeks of the study breath test, or a history of gastric surgery or vagotomy for ulcer disease except simple closure of a gastric perforation.

The protocol was approved by the local Institutional Review Board for Human Studies, and all subjects provided written informed consent.

Testing began with a minimum 4-hour fast from solid food. Breath samples were obtained in disposable, balloonlike, breath collection bags designed for use with the infrared spectrophotometer. One sample was obtained immediately before ingestion of the ¹³C-urea test solution, and the second was collected 30 minutes after substrate ingestion. Paired sample aliquots were taken for separate analyses. Results from local infrared instruments were blinded to the central laboratory.

 

Statistical analyses

Each primary care site was asked to enroll subjects until 80 positive cases were identified from among all sites. The clinical laboratory site tested a minimum of 30 positive and 30 negative cases. The primary endpoint was the percentage of agreement (overall and within positive and negative cases separately) of results from both methods. Delta-over-baseline (DOB) enrichment values below 2.4 per mil were deemed negative and values greater than or equal to 2.4 per mil were deemed positive. The predicate reference method was gas isotope ratio mass spectrometry. Equivalence was defined by the percentage of agreement for positive and negative cases based on gas isotope ratio mass spectrometry results of at least 95%, and the lower limit of the 95% confidence interval was based on a percentage of agreement of at least 90% for positive cases.

Results

The primary care centers enrolled 258 subjects and the clinical laboratory enrolled 64 subjects, for a total enrollment of 322. The subjects’ mean age was 41.5 years (range, 18–70 years), with 88 black non-Hispanics, 106 Hispanics, 92 whites, 32 Asian/Pacific Islanders, and 4 other ethnic groups. There were 215 women and 107 men. Approximately 18% had active or previous gastrointestinal ailments, including previously diagnosed H pylori infection (11%) and peptic ulcer disease (3%).

There was excellent agreement between methods (Table), with an overall agreement of 99% (95% confidence interval, 97.3–99.7). Two subjects were excluded from the analysis of the primary and secondary endpoints because 1 or both assay values were missing. The data showed close correlation between methods among all sites (Figure). Evaluation by the personnel who performed the tests and analyses indicated that the office procedure was easy and nonintrusive.

There were 3 disagreements between the results obtained with the devices, 1 from the gastroenterology site and 2 from the clinical laboratory site. All results were near the cutoff value.

TABLE
Comparison of results between IRMS and GIRMS*

	GIRMS, %
IRMS, %	Negative	Positive	Total
Negative	1	115	116
Positive	202	2	204
Total	203	117	320
*Overall agreement, 99.1% (95% confidence interval, 97.3–99.7); positive agreement, 98.2% (95% confidence interval, 94.2–99.7); negative agreement, 99.5% (95% confidence interval, 97.3–99.9).
Kappa statistic = 0.98.
GIRMS, gas isotope ratio mass spectrometry; IRMS, infrared mass spectrophotometry.

FIGURE
Comparison of results between 2 methods of analysis of ¹³CO₂ breath sample enrichments

Discussion

¹³C-urea breath testing is an accurate diagnostic method for the detection of active H pylori infection^12,19-21 and point-of-care assessment of curative therapy. This study confirmed the hypothesis that the infrared instrument is an easy to operate alternative to the original sendout analyses. Rapid turnaround allows for decisions regarding therapy to be made at the time of care. The currently approved UBT in the United States does not require fasting from solid food for longer than 1 hour.

Noninvasive alternatives to UBT include serology and stool antigen testing. Serology assays cannot discriminate between active and recent past infections. Stool antigen testing requires patient compliance with specimen collection and is a sendout test. In general, although studies using pretreatment stool antigen tests have shown sensitivity and specificity comparable to those of histology or UBT, it has become evident that there can be considerable lot-to-lot variation in stool antigen tests.²² The most likely explanation is that the polyclonal serum used for the capture antibody is obtained from rabbits and thus difficult to standardize.²³ Stool antigen testing has also proven to be less reliable when used soon after the end of therapy, and it is now generally recommended that one must wait 6 or 8 weeks after therapy when using the stool antigen test to confirm eradication. For example, a recent study had a false negative rate of 12.5% (95% confidence interval, 1.5–33).²⁴Recent recommendations are that the UBT is preferred where available.⁴

The US Food and Drug Administration recently cleared the UBiT-IR300 instrument for use with the commercial ¹³C-UBT. The costs of the UBT are but a fraction of those of endoscopy, not including indirect patient costs. Office-based testing has a separate reimbursement for testing, and overall the costs appear less than those of the stool antigen test. Economic impact studies comparing the tests are planned. Office-based infrared analysis for ¹³C makes near-patient or point-of-care UBT and analysis practical and should make accurate diagnosis of active H pylori infection readily available.

Acknowledgments

The authors thank the clinical study coordinators, including Ms Rebecca Garza, Ms Flora Godard, and Mr Zachary Patton for their conscientious efforts.

We conducted a cross-sectional study in Italy among men at least 50 years old and women at least 40 years old who consecutively visited their general practitioners. Patients were asked about the frequency of symptoms of overactive bladder and urinary incontinence. A total of 9613 men (mean age, 64.8 years; range, 50–98 years) and 13,365 women (mean age, 60.3 years; range, 40–98 years) were identified by 774 general practitioners. The frequencies of overactive bladder were 3.0% (95% confidence interval, 2.7–3.5) in men and 1.1% (95% confidence interval, 0.9–1.3) in women. The corresponding frequencies for urinary incontinence were 8.3% (95% confidence interval, 7.7–8.9) in men and 10.2% (95% confidence interval, 9.6–10.8) in women.

The reported prevalence of urinary incontinence ranges from 10% to 50%.^11-9 The differences can be explained in part by the age distribution of the populations considered, with higher rates in studies including more older subjects.^2,10 Further, some studies were conducted in selected populations.^7,11

With regard to diagnosis, although urinary incontinence should be an objectively proven condition of involuntary loss of urine, the criteria for its definition vary.¹² Differences also have been reported in the frequency of various types of urinary incontinence (urge, stress, and mixed incontinences),^11,13-16 and data are scant on the frequency of overactive bladder without urinary incontinence.¹⁶ We therefore conducted a study in Italian men at least 50 years old and women at least 40 years old to determine the prevalence of overactive bladder and various types of urinary incontinence.¹⁴

Materials And Methods

Eligible subjects were consecutive men at least 50 years old and women at least 40 years old who asked to be seen by their general practitioners during the study period. There were no exclusion criteria. Age cutoffs were chosen because of the very low rate of urinary incontinence in men younger than 50 years and in women younger than 40 years.²

Participating general practitioners were invited to take part in the study on the basis of lists of general practitioners specifically interested in epidemiologic studies and affiliated with the Gruppo Interdisciplinare di Studio Incontinenza Urinaria (Interdisciplinary Group for the Study of Incontinence). Each general practitioner could decide when to start recruitment (generally soon after agreeing to participate) and stopped when 50 cases had been identified, or after 5 days. The motivation for participating was purely an interest in the data collection. All participating physicians were general practitioners, without any specific interest in urogynecological problems.

We obtained demographic information from each subject. The subjects were then asked: Have you had any involuntary urinary loss during the past 3 months? On average, do you urinate more than 8 times a day and/or more than once during the night? Have you any urgency symptoms? If the subject answered yes to the first question, that subject was defined as having urinary incontinence; if the subject answered yes to the second and third questions, that subject was considered to have overactive bladder.

Subjects with incontinence were interviewed further with the questionnaire proposed by Wein and colleagues^17,18 for diagnosis of type of urinary incontinence: stress incontinence, mixed incontinence, or urge incontinence. The questionnaire provides a presumptive diagnosis of stress and urge incontinence on the basis of the presence or absence of the following symptoms: urgency, frequency with urgency, leaking during physical activity, amount of urinary leakage with each episode of incontinence, ability to reach the toilet in time after an urge to void, and nocturia. The criteria for overactive bladder without urinary incontinence (ie, patients reporting an urge to urinate more than 8 times a day and/or more than once during the night and/or with urgency symptoms) and various types of urinary incontinence were the criteria accepted by the main Italian research groups on urinary incontinence.

Informed consent was obtained from each subject. Participation in the study did not commit the patient to any instrumental examination or laboratory tests. Confidence intervals (CI) of the estimated percentages of frequency of urinary incontinence were based on the Poisson approximation. Statistical differences in the frequency of urinary incontinence among strata of age and sex were analyzed with the standard chi square test comparing observed and expected events and, when appropriate, using the test for trend.

Results

A total of 9858 men and 13,671 women were identified by 774 general practitioners, representing approximately 1.5% of all Italian general practitioners. Of those, 26.6% were in northern Italy, 17.4% were in central Italy, and 56.0% were in southern Italy. Each general practitioner identified a mean of 30 subjects (range, 15–50). A total of 245 men (2.5%) and 306 women (2.2%) refused to enter the study. Thus the present report included information on 9613 men (mean age, 64.8 years; range, 50–98 years) and 13,365 women (mean age, 60.3 years; range, 40–98 years).

 

The frequencies of overactive bladder without incontinence were 3.0% (95% CI, 2.7–3.5) in men and 1.1% (95% CI, 0.9–1.3) in women. The corresponding frequencies for urinary incontinence were 8.3% (95% CI, 7.7–8.9) in men and 10.2% (95% CI, 9.6–10.8) in women (Table 1).

The frequencies of overactive bladder and urinary incontinence increased with age in both sexes. For example, the frequencies of overactive bladder were 2.0% and 1.1% in men and women 51 to 60 years but 3.4% and 1.4% in subjects older than 70 years. The frequency of urinary incontinence also increased with age; this trend was statistically significant (chi square with 1 df, adjusted for sex, P < .05).

Table 2 shows the distribution of subjects with urinary problems stratified by sex, age, and type of problem. Mixed incontinence was the most frequent condition in men older than 60 years, and overactive bladder was more frequent in younger men. The relative frequency of stress incontinence tended to decrease with age. In women, stress incontinence and mixed incontinence were the most common causes of urinary incontinence in all age strata. The frequency of stress incontinence decreased with age, whereas that of mixed incontinence increased.

TABLE 1
Frequency of overactive bladder and urinary incontinence according to sex and age

Age, y	Men*			Women*
	No urinary problem	OB only	UI	No urinary problem	OB only	UI
40–50				3122 (93.6)	23 (0.7)	190 (5.7)
51–60	3160 (93.8)	69 (2.0)	139 (4.1)	323 (90.6)	38 (1.1)	299 (8.4)
61–70	3071 (88.8)	116 (3.4)	273 (7.9)	3180 (87.5)	52 (1.4)	403 (11.1)
>70	2298 (82.5)	100 (3.4)	387 (13.6)	2320 (82.2)	39 (1.4)	463 (16.4)
Total	8529 (88.7)	285 (3.0)	799 (8.3)	11858 (88.7)	150 (1.1)	1358 (10.2)
*Data are presented as number (%) of subjects.OB, overactive bladder; UI, urinary incontinence.

TABLE 2
Frequency of overactive bladder and various types of urinary incontinence in strata by sex and age*

Age, y	Men†				Women†
	OB only	UGI	SI	MXI	OB only	UGI	SI	MXI
40–50					23 (11.2)	31 (15.1)	93 (45.4)	58 (28.3)
51–60	69 (35.5)	30 (16.4)	33 (18.0)	55 (30.1)	38 (11.4)	59 (17.8)	127 (38.3)	108 (32.5)
61–70	116 (33.1)	63 (18.0)	37 (10.6)	134 (38.3)	52 (11.9)	72 (16.5)	169 (37.8)	147 (33.7)
>70	100 (24.1)	69 (15.7)	52 (12.5)	198 (47.7)	37 (7.6)	89 (18.3)	143 (29.4)	217 (44.7)
Total	285 (29.7)	162 (16.9)	122 (12.7)	391 (40.7)	150 (10.3)	251 (17.2)	532 (36.4)	530 (36.2)
*These totals are not the same as those in Table 1 due to missing values of subjects with OB or urinary incontinence.
†Data are presented as number (%) of subjects.
MXI, mixed incontinence; OB, overactive bladder; SI, stress incontinence; UGI, urge incontinence.

Discussion

Before discussing the results, the study limitations must be considered. The study population consisted of men at least 50 years and women at least 40 years identified among patients who asked to be seen by their general practitioners during the study period, but not among all patients registered with these physicians. The general practitioners were not randomly identified among all Italian general practitioners, so their patients cannot be formally considered representative of the Italian population. Nevertheless, general practitioners participating in this study were placed throughout the main areas of the country. The strengths of the study included the opportunity to analyze the prevalence of urinary incontinence and overactive bladder in a large series of subjects with the use of standard methods for recording the symptoms and data collection. Further, the interview was conducted by physicians well known to the subjects, which should increase the reliability of diagnosis, particularly of various types of urinary incontinence.

The limitations of a patient’s history in the diagnosis of type of urinary incontinence are widely recognized. In a review of the literature, a clinical history indicating stress or urge incontinence, when compared with a urodynamically based diagnosis, showed sensitivities of 0.9 and 0.4 and specificities of 0.5 and 0.6, respectively, in clinical studies.¹⁹ In epidemiologic studies with self-reported information, these values might be lower. Any misclassification of the type of incontinence would tend to reduce the differences in the frequency of different types.

The frequency of urinary incontinence in this population was consistent with that reported in studies conducted in European and North American areas and in Italy in the general population. For example, the prevalences of urinary incontinence in women 50 to 60 years were approximately 18% in Denmark²⁰ and 12% to 17% in the United Kingdom.^1,4 An Italian study of 2767 women found a prevalence of 11.8% for urinary incontinence in women 51 to 60 years old.

In contrast, the rates of urinary incontinence reported in this study were slightly lower than those reported by Lagace and colleagues in a similar ambulatory setting in the United States.⁶ They found prevalences of 35% for urinary incontinence in women 50 to 59 years and 5% in men in the same age group.

Among men the prevalence of urinary incontinence was similar to or slightly lower than that reported in the general population.¹ In the study by Bortolotti and associates, the overall prevalence of urinary incontinence in men older than 50 years was 2.3%, a figure somewhat lower than in the present study.¹⁴

 

An interesting finding of this study was the prevalence of overactive bladder without urinary incontinence, a condition rarely analyzed. It appears less frequently than urinary incontinence, particularly in women, and causes approximately 30% of urinary problems in men but only 10% in women. However, considering the frequencies of overactive bladder without urinary incontinence and urge incontinence, the conditions due to detrusor overactivity likely account for a large proportion of urinary problems.

In conclusion, this study, in a large number of subjects, provided an estimate of the prevalence of overactive bladder and urinary incontinence among people attending their general practitioners in Italy. It also emphasized the importance of detrusor overactivity-related conditions as a cause of urinary problems in this population, in both sexes and all ages.

MEMBERS OF THE GRUPPO INTERDISCIPLINARE DI STUDIO INCONTINENZA URINARIA

Walter Artibani, Cattedra di Urologia, Università di Verona; Francesco Benvenuti, Unità Operativa di Geriatria, Ospedale INRCA c/o Presidio IOT, Firenze; Roberto Carone, Divisione di Urologia, Centro Rieducazionale Funzionale, Torino; Francesco Catanzaro, Divisione di Urologia, Multimedica, Sesto San Giovanni (MI); Claudio Cricelli, SIMG, Firenze; Paolo Di Benedetto, Centro di Riabilitazione, Ospedale Santoro, Trieste; Vincenzo Giambanco, Divisione II di Ostetricia e Ginecologia, Ospedale Civico, Palermo; Gian Battista Massi, Clinica Ostetrica e Ginecologica, Università di Firenze; Rodolfo Milani, Divisione di Ginecologia, Ospedale Bassini, Cinisello Balsamo (MI); and Alberto Zanollo, Divisione di Urologia, Ospedale Civile, Magenta, Milano.

We conducted a cross-sectional study in Italy among men at least 50 years old and women at least 40 years old who consecutively visited their general practitioners. Patients were asked about the frequency of symptoms of overactive bladder and urinary incontinence. A total of 9613 men (mean age, 64.8 years; range, 50–98 years) and 13,365 women (mean age, 60.3 years; range, 40–98 years) were identified by 774 general practitioners. The frequencies of overactive bladder were 3.0% (95% confidence interval, 2.7–3.5) in men and 1.1% (95% confidence interval, 0.9–1.3) in women. The corresponding frequencies for urinary incontinence were 8.3% (95% confidence interval, 7.7–8.9) in men and 10.2% (95% confidence interval, 9.6–10.8) in women.

The reported prevalence of urinary incontinence ranges from 10% to 50%.^11-9 The differences can be explained in part by the age distribution of the populations considered, with higher rates in studies including more older subjects.^2,10 Further, some studies were conducted in selected populations.^7,11

With regard to diagnosis, although urinary incontinence should be an objectively proven condition of involuntary loss of urine, the criteria for its definition vary.¹² Differences also have been reported in the frequency of various types of urinary incontinence (urge, stress, and mixed incontinences),^11,13-16 and data are scant on the frequency of overactive bladder without urinary incontinence.¹⁶ We therefore conducted a study in Italian men at least 50 years old and women at least 40 years old to determine the prevalence of overactive bladder and various types of urinary incontinence.¹⁴

Materials And Methods

Eligible subjects were consecutive men at least 50 years old and women at least 40 years old who asked to be seen by their general practitioners during the study period. There were no exclusion criteria. Age cutoffs were chosen because of the very low rate of urinary incontinence in men younger than 50 years and in women younger than 40 years.²

Participating general practitioners were invited to take part in the study on the basis of lists of general practitioners specifically interested in epidemiologic studies and affiliated with the Gruppo Interdisciplinare di Studio Incontinenza Urinaria (Interdisciplinary Group for the Study of Incontinence). Each general practitioner could decide when to start recruitment (generally soon after agreeing to participate) and stopped when 50 cases had been identified, or after 5 days. The motivation for participating was purely an interest in the data collection. All participating physicians were general practitioners, without any specific interest in urogynecological problems.

We obtained demographic information from each subject. The subjects were then asked: Have you had any involuntary urinary loss during the past 3 months? On average, do you urinate more than 8 times a day and/or more than once during the night? Have you any urgency symptoms? If the subject answered yes to the first question, that subject was defined as having urinary incontinence; if the subject answered yes to the second and third questions, that subject was considered to have overactive bladder.

Subjects with incontinence were interviewed further with the questionnaire proposed by Wein and colleagues^17,18 for diagnosis of type of urinary incontinence: stress incontinence, mixed incontinence, or urge incontinence. The questionnaire provides a presumptive diagnosis of stress and urge incontinence on the basis of the presence or absence of the following symptoms: urgency, frequency with urgency, leaking during physical activity, amount of urinary leakage with each episode of incontinence, ability to reach the toilet in time after an urge to void, and nocturia. The criteria for overactive bladder without urinary incontinence (ie, patients reporting an urge to urinate more than 8 times a day and/or more than once during the night and/or with urgency symptoms) and various types of urinary incontinence were the criteria accepted by the main Italian research groups on urinary incontinence.

Informed consent was obtained from each subject. Participation in the study did not commit the patient to any instrumental examination or laboratory tests. Confidence intervals (CI) of the estimated percentages of frequency of urinary incontinence were based on the Poisson approximation. Statistical differences in the frequency of urinary incontinence among strata of age and sex were analyzed with the standard chi square test comparing observed and expected events and, when appropriate, using the test for trend.

Results

A total of 9858 men and 13,671 women were identified by 774 general practitioners, representing approximately 1.5% of all Italian general practitioners. Of those, 26.6% were in northern Italy, 17.4% were in central Italy, and 56.0% were in southern Italy. Each general practitioner identified a mean of 30 subjects (range, 15–50). A total of 245 men (2.5%) and 306 women (2.2%) refused to enter the study. Thus the present report included information on 9613 men (mean age, 64.8 years; range, 50–98 years) and 13,365 women (mean age, 60.3 years; range, 40–98 years).

 

The frequencies of overactive bladder without incontinence were 3.0% (95% CI, 2.7–3.5) in men and 1.1% (95% CI, 0.9–1.3) in women. The corresponding frequencies for urinary incontinence were 8.3% (95% CI, 7.7–8.9) in men and 10.2% (95% CI, 9.6–10.8) in women (Table 1).

The frequencies of overactive bladder and urinary incontinence increased with age in both sexes. For example, the frequencies of overactive bladder were 2.0% and 1.1% in men and women 51 to 60 years but 3.4% and 1.4% in subjects older than 70 years. The frequency of urinary incontinence also increased with age; this trend was statistically significant (chi square with 1 df, adjusted for sex, P < .05).

Table 2 shows the distribution of subjects with urinary problems stratified by sex, age, and type of problem. Mixed incontinence was the most frequent condition in men older than 60 years, and overactive bladder was more frequent in younger men. The relative frequency of stress incontinence tended to decrease with age. In women, stress incontinence and mixed incontinence were the most common causes of urinary incontinence in all age strata. The frequency of stress incontinence decreased with age, whereas that of mixed incontinence increased.

TABLE 1
Frequency of overactive bladder and urinary incontinence according to sex and age

Age, y	Men*			Women*
	No urinary problem	OB only	UI	No urinary problem	OB only	UI
40–50				3122 (93.6)	23 (0.7)	190 (5.7)
51–60	3160 (93.8)	69 (2.0)	139 (4.1)	323 (90.6)	38 (1.1)	299 (8.4)
61–70	3071 (88.8)	116 (3.4)	273 (7.9)	3180 (87.5)	52 (1.4)	403 (11.1)
>70	2298 (82.5)	100 (3.4)	387 (13.6)	2320 (82.2)	39 (1.4)	463 (16.4)
Total	8529 (88.7)	285 (3.0)	799 (8.3)	11858 (88.7)	150 (1.1)	1358 (10.2)
*Data are presented as number (%) of subjects.OB, overactive bladder; UI, urinary incontinence.

TABLE 2
Frequency of overactive bladder and various types of urinary incontinence in strata by sex and age*

Age, y	Men†				Women†
	OB only	UGI	SI	MXI	OB only	UGI	SI	MXI
40–50					23 (11.2)	31 (15.1)	93 (45.4)	58 (28.3)
51–60	69 (35.5)	30 (16.4)	33 (18.0)	55 (30.1)	38 (11.4)	59 (17.8)	127 (38.3)	108 (32.5)
61–70	116 (33.1)	63 (18.0)	37 (10.6)	134 (38.3)	52 (11.9)	72 (16.5)	169 (37.8)	147 (33.7)
>70	100 (24.1)	69 (15.7)	52 (12.5)	198 (47.7)	37 (7.6)	89 (18.3)	143 (29.4)	217 (44.7)
Total	285 (29.7)	162 (16.9)	122 (12.7)	391 (40.7)	150 (10.3)	251 (17.2)	532 (36.4)	530 (36.2)
*These totals are not the same as those in Table 1 due to missing values of subjects with OB or urinary incontinence.
†Data are presented as number (%) of subjects.
MXI, mixed incontinence; OB, overactive bladder; SI, stress incontinence; UGI, urge incontinence.

Discussion

Before discussing the results, the study limitations must be considered. The study population consisted of men at least 50 years and women at least 40 years identified among patients who asked to be seen by their general practitioners during the study period, but not among all patients registered with these physicians. The general practitioners were not randomly identified among all Italian general practitioners, so their patients cannot be formally considered representative of the Italian population. Nevertheless, general practitioners participating in this study were placed throughout the main areas of the country. The strengths of the study included the opportunity to analyze the prevalence of urinary incontinence and overactive bladder in a large series of subjects with the use of standard methods for recording the symptoms and data collection. Further, the interview was conducted by physicians well known to the subjects, which should increase the reliability of diagnosis, particularly of various types of urinary incontinence.

The limitations of a patient’s history in the diagnosis of type of urinary incontinence are widely recognized. In a review of the literature, a clinical history indicating stress or urge incontinence, when compared with a urodynamically based diagnosis, showed sensitivities of 0.9 and 0.4 and specificities of 0.5 and 0.6, respectively, in clinical studies.¹⁹ In epidemiologic studies with self-reported information, these values might be lower. Any misclassification of the type of incontinence would tend to reduce the differences in the frequency of different types.

The frequency of urinary incontinence in this population was consistent with that reported in studies conducted in European and North American areas and in Italy in the general population. For example, the prevalences of urinary incontinence in women 50 to 60 years were approximately 18% in Denmark²⁰ and 12% to 17% in the United Kingdom.^1,4 An Italian study of 2767 women found a prevalence of 11.8% for urinary incontinence in women 51 to 60 years old.

In contrast, the rates of urinary incontinence reported in this study were slightly lower than those reported by Lagace and colleagues in a similar ambulatory setting in the United States.⁶ They found prevalences of 35% for urinary incontinence in women 50 to 59 years and 5% in men in the same age group.

Among men the prevalence of urinary incontinence was similar to or slightly lower than that reported in the general population.¹ In the study by Bortolotti and associates, the overall prevalence of urinary incontinence in men older than 50 years was 2.3%, a figure somewhat lower than in the present study.¹⁴

 

An interesting finding of this study was the prevalence of overactive bladder without urinary incontinence, a condition rarely analyzed. It appears less frequently than urinary incontinence, particularly in women, and causes approximately 30% of urinary problems in men but only 10% in women. However, considering the frequencies of overactive bladder without urinary incontinence and urge incontinence, the conditions due to detrusor overactivity likely account for a large proportion of urinary problems.

In conclusion, this study, in a large number of subjects, provided an estimate of the prevalence of overactive bladder and urinary incontinence among people attending their general practitioners in Italy. It also emphasized the importance of detrusor overactivity-related conditions as a cause of urinary problems in this population, in both sexes and all ages.

MEMBERS OF THE GRUPPO INTERDISCIPLINARE DI STUDIO INCONTINENZA URINARIA

Walter Artibani, Cattedra di Urologia, Università di Verona; Francesco Benvenuti, Unità Operativa di Geriatria, Ospedale INRCA c/o Presidio IOT, Firenze; Roberto Carone, Divisione di Urologia, Centro Rieducazionale Funzionale, Torino; Francesco Catanzaro, Divisione di Urologia, Multimedica, Sesto San Giovanni (MI); Claudio Cricelli, SIMG, Firenze; Paolo Di Benedetto, Centro di Riabilitazione, Ospedale Santoro, Trieste; Vincenzo Giambanco, Divisione II di Ostetricia e Ginecologia, Ospedale Civico, Palermo; Gian Battista Massi, Clinica Ostetrica e Ginecologica, Università di Firenze; Rodolfo Milani, Divisione di Ginecologia, Ospedale Bassini, Cinisello Balsamo (MI); and Alberto Zanollo, Divisione di Urologia, Ospedale Civile, Magenta, Milano.

We conducted a cross-sectional study in Italy among men at least 50 years old and women at least 40 years old who consecutively visited their general practitioners. Patients were asked about the frequency of symptoms of overactive bladder and urinary incontinence. A total of 9613 men (mean age, 64.8 years; range, 50–98 years) and 13,365 women (mean age, 60.3 years; range, 40–98 years) were identified by 774 general practitioners. The frequencies of overactive bladder were 3.0% (95% confidence interval, 2.7–3.5) in men and 1.1% (95% confidence interval, 0.9–1.3) in women. The corresponding frequencies for urinary incontinence were 8.3% (95% confidence interval, 7.7–8.9) in men and 10.2% (95% confidence interval, 9.6–10.8) in women.

The reported prevalence of urinary incontinence ranges from 10% to 50%.^11-9 The differences can be explained in part by the age distribution of the populations considered, with higher rates in studies including more older subjects.^2,10 Further, some studies were conducted in selected populations.^7,11

With regard to diagnosis, although urinary incontinence should be an objectively proven condition of involuntary loss of urine, the criteria for its definition vary.¹² Differences also have been reported in the frequency of various types of urinary incontinence (urge, stress, and mixed incontinences),^11,13-16 and data are scant on the frequency of overactive bladder without urinary incontinence.¹⁶ We therefore conducted a study in Italian men at least 50 years old and women at least 40 years old to determine the prevalence of overactive bladder and various types of urinary incontinence.¹⁴

Materials And Methods

Eligible subjects were consecutive men at least 50 years old and women at least 40 years old who asked to be seen by their general practitioners during the study period. There were no exclusion criteria. Age cutoffs were chosen because of the very low rate of urinary incontinence in men younger than 50 years and in women younger than 40 years.²

Participating general practitioners were invited to take part in the study on the basis of lists of general practitioners specifically interested in epidemiologic studies and affiliated with the Gruppo Interdisciplinare di Studio Incontinenza Urinaria (Interdisciplinary Group for the Study of Incontinence). Each general practitioner could decide when to start recruitment (generally soon after agreeing to participate) and stopped when 50 cases had been identified, or after 5 days. The motivation for participating was purely an interest in the data collection. All participating physicians were general practitioners, without any specific interest in urogynecological problems.

We obtained demographic information from each subject. The subjects were then asked: Have you had any involuntary urinary loss during the past 3 months? On average, do you urinate more than 8 times a day and/or more than once during the night? Have you any urgency symptoms? If the subject answered yes to the first question, that subject was defined as having urinary incontinence; if the subject answered yes to the second and third questions, that subject was considered to have overactive bladder.

Subjects with incontinence were interviewed further with the questionnaire proposed by Wein and colleagues^17,18 for diagnosis of type of urinary incontinence: stress incontinence, mixed incontinence, or urge incontinence. The questionnaire provides a presumptive diagnosis of stress and urge incontinence on the basis of the presence or absence of the following symptoms: urgency, frequency with urgency, leaking during physical activity, amount of urinary leakage with each episode of incontinence, ability to reach the toilet in time after an urge to void, and nocturia. The criteria for overactive bladder without urinary incontinence (ie, patients reporting an urge to urinate more than 8 times a day and/or more than once during the night and/or with urgency symptoms) and various types of urinary incontinence were the criteria accepted by the main Italian research groups on urinary incontinence.

Informed consent was obtained from each subject. Participation in the study did not commit the patient to any instrumental examination or laboratory tests. Confidence intervals (CI) of the estimated percentages of frequency of urinary incontinence were based on the Poisson approximation. Statistical differences in the frequency of urinary incontinence among strata of age and sex were analyzed with the standard chi square test comparing observed and expected events and, when appropriate, using the test for trend.

Results

A total of 9858 men and 13,671 women were identified by 774 general practitioners, representing approximately 1.5% of all Italian general practitioners. Of those, 26.6% were in northern Italy, 17.4% were in central Italy, and 56.0% were in southern Italy. Each general practitioner identified a mean of 30 subjects (range, 15–50). A total of 245 men (2.5%) and 306 women (2.2%) refused to enter the study. Thus the present report included information on 9613 men (mean age, 64.8 years; range, 50–98 years) and 13,365 women (mean age, 60.3 years; range, 40–98 years).

 

The frequencies of overactive bladder without incontinence were 3.0% (95% CI, 2.7–3.5) in men and 1.1% (95% CI, 0.9–1.3) in women. The corresponding frequencies for urinary incontinence were 8.3% (95% CI, 7.7–8.9) in men and 10.2% (95% CI, 9.6–10.8) in women (Table 1).

The frequencies of overactive bladder and urinary incontinence increased with age in both sexes. For example, the frequencies of overactive bladder were 2.0% and 1.1% in men and women 51 to 60 years but 3.4% and 1.4% in subjects older than 70 years. The frequency of urinary incontinence also increased with age; this trend was statistically significant (chi square with 1 df, adjusted for sex, P < .05).

Table 2 shows the distribution of subjects with urinary problems stratified by sex, age, and type of problem. Mixed incontinence was the most frequent condition in men older than 60 years, and overactive bladder was more frequent in younger men. The relative frequency of stress incontinence tended to decrease with age. In women, stress incontinence and mixed incontinence were the most common causes of urinary incontinence in all age strata. The frequency of stress incontinence decreased with age, whereas that of mixed incontinence increased.

TABLE 1
Frequency of overactive bladder and urinary incontinence according to sex and age

Age, y	Men*			Women*
	No urinary problem	OB only	UI	No urinary problem	OB only	UI
40–50				3122 (93.6)	23 (0.7)	190 (5.7)
51–60	3160 (93.8)	69 (2.0)	139 (4.1)	323 (90.6)	38 (1.1)	299 (8.4)
61–70	3071 (88.8)	116 (3.4)	273 (7.9)	3180 (87.5)	52 (1.4)	403 (11.1)
>70	2298 (82.5)	100 (3.4)	387 (13.6)	2320 (82.2)	39 (1.4)	463 (16.4)
Total	8529 (88.7)	285 (3.0)	799 (8.3)	11858 (88.7)	150 (1.1)	1358 (10.2)
*Data are presented as number (%) of subjects.OB, overactive bladder; UI, urinary incontinence.

TABLE 2
Frequency of overactive bladder and various types of urinary incontinence in strata by sex and age*

Age, y	Men†				Women†
	OB only	UGI	SI	MXI	OB only	UGI	SI	MXI
40–50					23 (11.2)	31 (15.1)	93 (45.4)	58 (28.3)
51–60	69 (35.5)	30 (16.4)	33 (18.0)	55 (30.1)	38 (11.4)	59 (17.8)	127 (38.3)	108 (32.5)
61–70	116 (33.1)	63 (18.0)	37 (10.6)	134 (38.3)	52 (11.9)	72 (16.5)	169 (37.8)	147 (33.7)
>70	100 (24.1)	69 (15.7)	52 (12.5)	198 (47.7)	37 (7.6)	89 (18.3)	143 (29.4)	217 (44.7)
Total	285 (29.7)	162 (16.9)	122 (12.7)	391 (40.7)	150 (10.3)	251 (17.2)	532 (36.4)	530 (36.2)
*These totals are not the same as those in Table 1 due to missing values of subjects with OB or urinary incontinence.
†Data are presented as number (%) of subjects.
MXI, mixed incontinence; OB, overactive bladder; SI, stress incontinence; UGI, urge incontinence.

Discussion

Before discussing the results, the study limitations must be considered. The study population consisted of men at least 50 years and women at least 40 years identified among patients who asked to be seen by their general practitioners during the study period, but not among all patients registered with these physicians. The general practitioners were not randomly identified among all Italian general practitioners, so their patients cannot be formally considered representative of the Italian population. Nevertheless, general practitioners participating in this study were placed throughout the main areas of the country. The strengths of the study included the opportunity to analyze the prevalence of urinary incontinence and overactive bladder in a large series of subjects with the use of standard methods for recording the symptoms and data collection. Further, the interview was conducted by physicians well known to the subjects, which should increase the reliability of diagnosis, particularly of various types of urinary incontinence.

The limitations of a patient’s history in the diagnosis of type of urinary incontinence are widely recognized. In a review of the literature, a clinical history indicating stress or urge incontinence, when compared with a urodynamically based diagnosis, showed sensitivities of 0.9 and 0.4 and specificities of 0.5 and 0.6, respectively, in clinical studies.¹⁹ In epidemiologic studies with self-reported information, these values might be lower. Any misclassification of the type of incontinence would tend to reduce the differences in the frequency of different types.

The frequency of urinary incontinence in this population was consistent with that reported in studies conducted in European and North American areas and in Italy in the general population. For example, the prevalences of urinary incontinence in women 50 to 60 years were approximately 18% in Denmark²⁰ and 12% to 17% in the United Kingdom.^1,4 An Italian study of 2767 women found a prevalence of 11.8% for urinary incontinence in women 51 to 60 years old.

In contrast, the rates of urinary incontinence reported in this study were slightly lower than those reported by Lagace and colleagues in a similar ambulatory setting in the United States.⁶ They found prevalences of 35% for urinary incontinence in women 50 to 59 years and 5% in men in the same age group.

Among men the prevalence of urinary incontinence was similar to or slightly lower than that reported in the general population.¹ In the study by Bortolotti and associates, the overall prevalence of urinary incontinence in men older than 50 years was 2.3%, a figure somewhat lower than in the present study.¹⁴

 

An interesting finding of this study was the prevalence of overactive bladder without urinary incontinence, a condition rarely analyzed. It appears less frequently than urinary incontinence, particularly in women, and causes approximately 30% of urinary problems in men but only 10% in women. However, considering the frequencies of overactive bladder without urinary incontinence and urge incontinence, the conditions due to detrusor overactivity likely account for a large proportion of urinary problems.

In conclusion, this study, in a large number of subjects, provided an estimate of the prevalence of overactive bladder and urinary incontinence among people attending their general practitioners in Italy. It also emphasized the importance of detrusor overactivity-related conditions as a cause of urinary problems in this population, in both sexes and all ages.

MEMBERS OF THE GRUPPO INTERDISCIPLINARE DI STUDIO INCONTINENZA URINARIA

Walter Artibani, Cattedra di Urologia, Università di Verona; Francesco Benvenuti, Unità Operativa di Geriatria, Ospedale INRCA c/o Presidio IOT, Firenze; Roberto Carone, Divisione di Urologia, Centro Rieducazionale Funzionale, Torino; Francesco Catanzaro, Divisione di Urologia, Multimedica, Sesto San Giovanni (MI); Claudio Cricelli, SIMG, Firenze; Paolo Di Benedetto, Centro di Riabilitazione, Ospedale Santoro, Trieste; Vincenzo Giambanco, Divisione II di Ostetricia e Ginecologia, Ospedale Civico, Palermo; Gian Battista Massi, Clinica Ostetrica e Ginecologica, Università di Firenze; Rodolfo Milani, Divisione di Ginecologia, Ospedale Bassini, Cinisello Balsamo (MI); and Alberto Zanollo, Divisione di Urologia, Ospedale Civile, Magenta, Milano.

Spirituality and religion are embedded within contemporary American culture¹ and have become an increasingly important part of the patient experience of health and illness.² There is growing interest in examining the association of spirituality, religion, and health-related outcomes in the United States, particularly in the areas of health behavior and promotion³ and psychoneuroimmunology.⁴ Despite this interest, the absence of operational definitions of spirituality and religion, the contamination of spirituality items with measures of religion, and the lack of valid and reliable instruments that gauge these constructs continue to be major limitations to work in this area.⁵

Conceptually, religion or religiosity is often viewed in terms of the various organized, individual, and attitudinal manifestations of different faith traditions, and spirituality connotes and expresses a sense of meaning, purpose, or power from within or from a transcendent source.⁶ There is no shortage of instruments that measure dimensions of either construct, and researchers from the fields of sociology,⁷ psychology,⁶ and pastoral theology and chaplaincy⁸ have developed a variety of scales of religion and spirituality.⁹ It remains unclear, however, whether these constructs can be extended to health care settings or whether these instruments are applicable and useful as measures of individual or population health. For example, frequency of religious service attendance is often a single-item measure used as an independent variable in studies of health outcomes, such as health status.¹⁰ Although service attendance is associated with self-reported health in community-dwelling elderly individuals, the effect of this activity on perceived health disappears when functional status is controlled.¹⁰ Therefore, can religious service attendance be considered an independent variable, or is it simply a proxy of functional status within a geriatric population?¹¹

This example highlights the importance of context in the use of any measure of religion or spirituality. It also points to the health-related quality of life field as a useful orientation for conceptualizing spirituality and religion in health care settings. Health-related quality of life, an individual’s or group’s perception of health over time, is predicated on the assumption that a patient’s experiences, beliefs, expectations, and perceptions directly influence the physical, psychological, and social domains of health.¹² Spirituality and religion have been proposed as mediators of 1 characteristic of psychological health, subjective well-being, in 4 ways: by ensuring social support and integration within a community, by establishing personal relationships with a divine other, by promoting a salubrious personal lifestyle that is congruent with a personal faith tradition, and by providing systems of meaning and existential coherence.¹³

To identify and describe elements of spirituality that are linked to subjective well-being, our prior qualitative work explored the patient perspective. We found that patients consider spirituality in predominantly cognitive terms and incorporate the domains of life scheme and positive intentionality, or self-efficacy, as primary components Figure 1.¹⁴ In addition to suggesting a dynamic conceptual framework, this research supported the assumption that patients associate spirituality with well-being largely through the provision of systems of meaning and coherence.

The current study builds on this work and describes the development and evaluation of a brief research instrument, the Spirituality Index of Well-Being (SIWB), which is designed to measure the effect of spirituality on subjective well-being. Several assumptions guided our study design and analysis. First, we recognized that there are no global yet parsimonious instruments that capture the complexity and depth of spirituality in any context, health care or otherwise. Second, based on our qualitative work, we viewed spirituality as subsumed within the psychological rather than within the social or physical domain. Third, we considered the SIWB as a health-related quality of life measure, one to be used in studies of individual or population health, rather than as an assessment tool.

 

From the cultural and social perspectives, spirituality and religion are especially salient in the lives of minority elderly,^15,16 particularly within the settings of serious illness and end-of-life care.¹⁷ From a population health perspective, increased life expectancy in the United States highlights the importance of health-related quality of life assessment in the areas of chronic illness, aging, and end-of-life care, and Healthy People 2010 has identified quality of life improvement as a specific public health objective.¹⁸ By bridging both perspectives, the SIWB has the potential to add a unique and patient-centered dimension to health-related quality of life research.

Methods

Scale and item development

The SIWB was designed as a research tool to measure the effect of patient-reported spirituality on subjective well-being. Our understanding of spirituality and the stimulus material for the index have been described elsewhere.¹⁴ In brief, a congruent, meaningful life scheme and a high degree of positive intentionality or self-efficacy promote personal agency, an intermediary between spirituality and subjective well-being Figure.

Life scheme is similar to the construct of sense of coherence, which was described by Antonovsky as a positive, pervasive way of viewing the world, and one’s life in it, lending elements of comprehensibility, manageability, and meaningfulness.¹⁹ Positive intentionality shares characteristics with self-efficacy, which is an individual’s belief in the capacity to organize and perform activities that are required for a prescribed goal.²⁰ Self-efficacy beliefs are domain and task specific, and participants in our focus group study depicted these beliefs within the context of overcoming threatened or actual changes to their functioning.

Forty items, 20 for the life scheme domain and 20 for the self-efficacy domain, were developed by investigators who conducted the qualitative study (T.P.D., B.B.F.). The scale was prefaced by the question, “Which statement best describes your feelings and choices,” and each item was a statement accompanied by a 5-point Likert scale response from “strongly agree” to “strongly disagree,” with the midpoint representing “neither agree nor disagree.” Item content consisted of positive and negative statements regarding life scheme (eg, “I haven’t yet found my life’s purpose”) and personal self-efficacy (eg, “Despite any problem that I may face, I can get through the day”).

Study population

Participants were 65 years or older and enrolled in a cohort study to assess the ability of performance measures to predict future health service use, health status, and functional status. Recruitment for the parent study occurred between April and November 1996 from primary care sites within the Veteran’s Affairs network and a Medicare health management organization serving the Kansas City metropolitan area. The study population represented the cohort 36 months after enrollment.

Measures

Demographic information. Participants had the following demographic information collected: age, sex, race, and education level.

Health and functional status. Subjective health status was measured by the EuroQol, a recognized quality-of-life measure,²¹ in addition to a single-item measure of global health from the Years of Healthy Life (YOHL) Scale.²² The Physical Functioning Index of the Medical Outcomes Study Short Form 36 was used to assess functional status.²³

Mental health status. We measured mental health status with the Geriatric Depression Scale (GDS), a 15-item instrument with a dichotomous (yes/no) response format.²⁴ Items from the fear of death domain of the Death Attitude Profile Scale-Revised (DAP-R) were selected as an additional proxy of psychological well-being.²⁵

Religiosity. Five items derived from questions developed by the National Opinion Research Center²⁶ were preferentially selected according to a previously tested and validated model of religiosity.²⁷ Frequency of religious or spiritual service attendance was used to assess organizational religiosity, and frequency of private prayer or spiritual practice was used to measure nonorganizational religiosity. Three items were used to measure subjective or intrinsic religiosity: self-reported strength of religious or spiritual orientation, closeness to God (or a Higher Force), and frequency of affective spiritual experiences..

Data analysis

Item reduction and reliability testing. The initial 40-item pool was reduced to 20 life scheme items and 14 self-efficacy items based on subject response and feedback during survey administration. Items that subjects could not understand or answer by self-report were removed.

First, internal reliability analyses were conducted for each subscale (life scheme, self-efficacy) and for the SIWB scale with a goal of producing high internal consistency as measured by the Cronbach’s α (eg, > .70). Items that contributed to lower internal reliability were discarded, which removed 1 self-efficacy item and 6 life scheme items from the scale.

To further refine the SIWB and its subscales, the remaining items were subjected to principal components analysis by using Varimax rotation. After rotation, the 2 largest factors were readily interpretable, with items loading as expected: self-efficacy items loading on the first factor and life scheme items loading on the second factor. From each factor, the top 6 items ranked by loading magnitude were selected for inclusion into the final scale.

 

Internal reliabilities for the subscales (6 items each) and the SIWB scale (12 items total) were calculated. A maximum likelihood factor analysis with Varimax rotation also was conducted to verify that a 2-factor solution remained for the reduced 12-item scale.

Validity testing. Well-being is conceptually subsumed within the psychological domain of quality of life measures and is comprised of the dimensions of positive affect (affective) and subjective perceptions of general health and life satisfaction (cognitive).¹² As a result, we determined concurrent construct validity by correlating the 2 6-item subscale scores and the total SIWB score with summed scores from the fear of death items from the DAP-R, the GDS, YOHL, the Physical Functioning Index from the SF-36, and the EuroQol. We anticipated positive correlations of the SIWB with physical functioning (SF-36) and quality of life (EuroQol) and inverse correlations with fear of death (DAP-R), depression (GDS), and self-reported poor health status (YOHL). Discriminant validity was examined by correlating the SIWB subscale and total scores with the religiosity measure. All analyses were performed with the Statistical Package for the Social Sciences version 9.0 (SPSS, Chicago, IL, 1996).

Results

Study population

Two hundred seventy-seven patients were in the final cohort and participated in the study Table 1. The mean age of the study population was 74 years, with a range of 65 to 90 years. Most participants (66%) were 75 years or younger, and the population was evenly distributed between males and females. Participants were predominantly white (78%), reported a wide range of education levels, and had a mean physical function score (SF-36) of 62.92 and a mean health status score (EuroQol) of 0.77.

Internal consistency and factor analysis

Twelve items, 6 each from the self-efficacy and life scheme subscales, remained from the original 40 items after item reduction; initial reliability testing and factor analysis were performed. This 12-item measure of the SIWB produced a coefficient α of .87, indicating good internal consistency. The 6-item subscales also demonstrated good reliability: .83 for self-efficacy and .80 for life scheme.

Results of factor analysis with individual items and item loadings for the final SIWB scale are presented in Table 2. A confirmatory approach anticipated 2 factors, which was based on our conceptual framework. Factor analysis found that 2 factors, reasonably named self-efficacy and life scheme, accounted for a substantial proportion of the variance in responses. The eigenvalue for the self-efficacy factor was 2.88, accounting for 24.04% of the total variance. The eigenvalue for the life scheme factor was 2.35, accounting for 19.57% of the total variance. A Pearson chi-square goodness of fit test of the difference between the actual and reproduced correlation patterns was not significant (51.72; df = 43; P = .17), which suggested that a 2-factor solution was reasonable. Table 3 contains the descriptive statistics for the SIWB scale and its subscales.

Validity testing

To provide a more consistent and intuitive interpretation of scores and correlations, SIWB total and subscale scores were produced by reverse scoring and summing items. As a result, higher SIWB scores indicated a greater degree of spirituality or its components. Correlations between the summed SIWB and subscale scores and other health-related measures of well-being are presented in Table 4. The SIWB and its subscales had significant and expected correlations in direction and magnitude with other measures related to subjective well-being. Fear of death and depression (GDS) had the highest inverse correlations with the SIWB and its subscales. Subjective perceptions of general health and life satisfaction, as measured by self-reports of poor health status (YOHL), functional quality of life (EuroQol), and physical functioning (SF-36) had significant correlations with the SIWB.

Although the life scheme subscale did have a significant but small correlation with a previously validated measure of religiosity, the total SIWB scale and self-efficacy subscale did not have a significant correlation with religiosity.

Discussion

The purpose of this study was to evaluate a brief research instrument designed to measure the effect of spirituality on subjective well-being in a patient population. Instruments that are developed to measure health-related quality of life are evaluated according to several criteria, most notably their degree of validity and reliability.²⁸ The SIWB demonstrated very good reliability with good internal consistency for the total and subscales as assessed by α coefficient in a geriatric patient population.

The construct spirituality has multiple dimensions and connotations in health-related settings,²⁹ which challenge the validity testing of any spirituality instrument. We chose a qualitative approach, rather than the use of experts or preexisting measures in health services research, pastoral theology and chaplaincy, and the social sciences, to conceptualize how patients understand and define spirituality, in particular as if affects their well-being. This approach also provided stimulus material for SIWB item selection and scale construction.

 

In our conceptual framework, spirituality within a health context is a state that is comprised primarily of the domains of life scheme and self-efficacy. Patients who report high self-efficacy beliefs regarding their functioning and who view their lives as purposeful and meaningful should score higher on measures of subjective well-being than those who do not hold such beliefs or attitudes. The use of concurrent construct validity testing allowed us to test this assumption through the correlation of SIWB scores with other established proxies of subjective well-being. Face validity may suggest that the SIWB is a measure of affective or cognitive states (eg, depression) or a proxy for self-efficacy and alienation rather than spirituality. Concurrent construct validity testing provided a means to determine the independence of the SIWB from an accepted measure of depression, the GDS.

Although the pilot version of the SIWB consisted of 40 items with positive and negative statements regarding life scheme and personal self-efficacy, only negative items remained after validity and reliability testing. One explanation for the exclusion of positive statements from the SIWB may involve the predominance of a specific component of subjective well-being in older persons, a low level of negative affect. There are several additional components of subjective well-being (eg, positive affect, satisfaction with work or other domains, and life satisfaction),³⁰ that may not be as salient or as operational in an older population.

However, the SIWB consistently had significant and expected correlations in direction and magnitude with other established measures related to subjective well-being. Spirituality had the highest inverse correlations with fear of death, depression, and perceived health status, which are supportive of affective and cognitive dimensions of subjective well-being in our instrument. A modest correlation with the GDS also suggested that the SIWB is a measure that is independent of depression.

Discriminant validity testing was used to differentiate the SIWB from religiosity. The total SIWB scale did not have a significant correlation with a measure of religiosity that has been used in a geriatric population,²⁷ although the life scheme subscale did have a significant but small (r = .18) correlation. The distinction between conceptualizations of religiosity and spirituality is a major consideration in measurement development,³¹ and there are other measures of spirituality that have been used in clinical and research settings. Virtually all are contaminated by the inclusion of items that assess religiosity.⁹ For example, the Functional Assessment of Chronic Illness Therapy-Spiritual Well-Being Scale contains items that measure the comfort and strength derived from religious faith, in addition to a sense of meaning, purpose, and peace in life.³² The Systems of Belief Inventory, which was designed for use in quality of life and psychosocial research examining illness adjustment, measures religious and spiritual beliefs and practices and the social support that accompanies those beliefs and practices.³³

The Spiritual Well-Being Scale has been used widely in health care settings and consists of 2 subscales: a religious well-being subscale and an existential well-being subscale.³⁴ Religious well-being is conceptualized as the quality of one’s relationship with God, whereas existential well-being includes characteristics such as life purpose, life satisfaction, and positive and negative life experiences. Scores from the Spiritual Well-Being Scale have been inversely correlated with measures of psychological well-being.

However, much of this unpublished research has been compromised by ceiling effects or an inability to detect differences in those who score high on the scale, particularly in religious populations³⁵ and by a lack of peer review.³⁶

Our study has several limitations. Our conceptualization of spirituality is a new construct based on qualitative research, and the study purpose was to evaluate the psychometric properties of a new instrument to measure this construct. As a result, we did not analyze or report normative data about the SIWB. Spirituality may have conceptual overlap with existing constructs, such as self-efficacy and alienation, and we did not evaluate the independence of our scale against these constructs. The SIWB was embedded in the final cohort of a longitudinal study, and we were unable to perform test-retest reliability to determine the stability and the responsiveness or sensitivity of the instrument over time. Due to subject burden, the parent study limited the inclusion of additional measures and the quality-of-life instruments were selected a priori.

Our cross-sectional design also did not allow us to draw any definitive conclusions about the causal relations of the variables. The study population consisted primarily of predominantly white, older patients with some functional limitations, and the generalizability of our findings to other populations is uncertain. However, good theory development and item construction from prior qualitative studies, a high α coefficient, and factor analysis support the validity and reliability of our measure.

 

In summary, the SIWB appears to be a valid and reliable measure of patient subjective well-being, one that is uncontaminated by the inclusion of religiosity. This instrument may be used in observational studies of chronic illness, aging, and end-of-life care that use spirituality as an explanatory or predictor variable of well-being. Future validation studies with multiple, diverse populations and a longitudinal design are needed to refine, modify, or verify the SIWB as an additional, complementary instrument of wellbeing.

Acknowledgments

We thank Lynn Maxwell, Annette Becker, Danielle Sirchak, Donna Clausen, June Belt, Marjoire Frank, and Lisa Rogers for their dedicated service in this study.

Spirituality and religion are embedded within contemporary American culture¹ and have become an increasingly important part of the patient experience of health and illness.² There is growing interest in examining the association of spirituality, religion, and health-related outcomes in the United States, particularly in the areas of health behavior and promotion³ and psychoneuroimmunology.⁴ Despite this interest, the absence of operational definitions of spirituality and religion, the contamination of spirituality items with measures of religion, and the lack of valid and reliable instruments that gauge these constructs continue to be major limitations to work in this area.⁵

Conceptually, religion or religiosity is often viewed in terms of the various organized, individual, and attitudinal manifestations of different faith traditions, and spirituality connotes and expresses a sense of meaning, purpose, or power from within or from a transcendent source.⁶ There is no shortage of instruments that measure dimensions of either construct, and researchers from the fields of sociology,⁷ psychology,⁶ and pastoral theology and chaplaincy⁸ have developed a variety of scales of religion and spirituality.⁹ It remains unclear, however, whether these constructs can be extended to health care settings or whether these instruments are applicable and useful as measures of individual or population health. For example, frequency of religious service attendance is often a single-item measure used as an independent variable in studies of health outcomes, such as health status.¹⁰ Although service attendance is associated with self-reported health in community-dwelling elderly individuals, the effect of this activity on perceived health disappears when functional status is controlled.¹⁰ Therefore, can religious service attendance be considered an independent variable, or is it simply a proxy of functional status within a geriatric population?¹¹

This example highlights the importance of context in the use of any measure of religion or spirituality. It also points to the health-related quality of life field as a useful orientation for conceptualizing spirituality and religion in health care settings. Health-related quality of life, an individual’s or group’s perception of health over time, is predicated on the assumption that a patient’s experiences, beliefs, expectations, and perceptions directly influence the physical, psychological, and social domains of health.¹² Spirituality and religion have been proposed as mediators of 1 characteristic of psychological health, subjective well-being, in 4 ways: by ensuring social support and integration within a community, by establishing personal relationships with a divine other, by promoting a salubrious personal lifestyle that is congruent with a personal faith tradition, and by providing systems of meaning and existential coherence.¹³

To identify and describe elements of spirituality that are linked to subjective well-being, our prior qualitative work explored the patient perspective. We found that patients consider spirituality in predominantly cognitive terms and incorporate the domains of life scheme and positive intentionality, or self-efficacy, as primary components Figure 1.¹⁴ In addition to suggesting a dynamic conceptual framework, this research supported the assumption that patients associate spirituality with well-being largely through the provision of systems of meaning and coherence.

The current study builds on this work and describes the development and evaluation of a brief research instrument, the Spirituality Index of Well-Being (SIWB), which is designed to measure the effect of spirituality on subjective well-being. Several assumptions guided our study design and analysis. First, we recognized that there are no global yet parsimonious instruments that capture the complexity and depth of spirituality in any context, health care or otherwise. Second, based on our qualitative work, we viewed spirituality as subsumed within the psychological rather than within the social or physical domain. Third, we considered the SIWB as a health-related quality of life measure, one to be used in studies of individual or population health, rather than as an assessment tool.

 

From the cultural and social perspectives, spirituality and religion are especially salient in the lives of minority elderly,^15,16 particularly within the settings of serious illness and end-of-life care.¹⁷ From a population health perspective, increased life expectancy in the United States highlights the importance of health-related quality of life assessment in the areas of chronic illness, aging, and end-of-life care, and Healthy People 2010 has identified quality of life improvement as a specific public health objective.¹⁸ By bridging both perspectives, the SIWB has the potential to add a unique and patient-centered dimension to health-related quality of life research.

Methods

Scale and item development

The SIWB was designed as a research tool to measure the effect of patient-reported spirituality on subjective well-being. Our understanding of spirituality and the stimulus material for the index have been described elsewhere.¹⁴ In brief, a congruent, meaningful life scheme and a high degree of positive intentionality or self-efficacy promote personal agency, an intermediary between spirituality and subjective well-being Figure.

Life scheme is similar to the construct of sense of coherence, which was described by Antonovsky as a positive, pervasive way of viewing the world, and one’s life in it, lending elements of comprehensibility, manageability, and meaningfulness.¹⁹ Positive intentionality shares characteristics with self-efficacy, which is an individual’s belief in the capacity to organize and perform activities that are required for a prescribed goal.²⁰ Self-efficacy beliefs are domain and task specific, and participants in our focus group study depicted these beliefs within the context of overcoming threatened or actual changes to their functioning.

Forty items, 20 for the life scheme domain and 20 for the self-efficacy domain, were developed by investigators who conducted the qualitative study (T.P.D., B.B.F.). The scale was prefaced by the question, “Which statement best describes your feelings and choices,” and each item was a statement accompanied by a 5-point Likert scale response from “strongly agree” to “strongly disagree,” with the midpoint representing “neither agree nor disagree.” Item content consisted of positive and negative statements regarding life scheme (eg, “I haven’t yet found my life’s purpose”) and personal self-efficacy (eg, “Despite any problem that I may face, I can get through the day”).

Study population

Participants were 65 years or older and enrolled in a cohort study to assess the ability of performance measures to predict future health service use, health status, and functional status. Recruitment for the parent study occurred between April and November 1996 from primary care sites within the Veteran’s Affairs network and a Medicare health management organization serving the Kansas City metropolitan area. The study population represented the cohort 36 months after enrollment.

Measures

Demographic information. Participants had the following demographic information collected: age, sex, race, and education level.

Health and functional status. Subjective health status was measured by the EuroQol, a recognized quality-of-life measure,²¹ in addition to a single-item measure of global health from the Years of Healthy Life (YOHL) Scale.²² The Physical Functioning Index of the Medical Outcomes Study Short Form 36 was used to assess functional status.²³

Mental health status. We measured mental health status with the Geriatric Depression Scale (GDS), a 15-item instrument with a dichotomous (yes/no) response format.²⁴ Items from the fear of death domain of the Death Attitude Profile Scale-Revised (DAP-R) were selected as an additional proxy of psychological well-being.²⁵

Religiosity. Five items derived from questions developed by the National Opinion Research Center²⁶ were preferentially selected according to a previously tested and validated model of religiosity.²⁷ Frequency of religious or spiritual service attendance was used to assess organizational religiosity, and frequency of private prayer or spiritual practice was used to measure nonorganizational religiosity. Three items were used to measure subjective or intrinsic religiosity: self-reported strength of religious or spiritual orientation, closeness to God (or a Higher Force), and frequency of affective spiritual experiences..

Data analysis

Item reduction and reliability testing. The initial 40-item pool was reduced to 20 life scheme items and 14 self-efficacy items based on subject response and feedback during survey administration. Items that subjects could not understand or answer by self-report were removed.

First, internal reliability analyses were conducted for each subscale (life scheme, self-efficacy) and for the SIWB scale with a goal of producing high internal consistency as measured by the Cronbach’s α (eg, > .70). Items that contributed to lower internal reliability were discarded, which removed 1 self-efficacy item and 6 life scheme items from the scale.

To further refine the SIWB and its subscales, the remaining items were subjected to principal components analysis by using Varimax rotation. After rotation, the 2 largest factors were readily interpretable, with items loading as expected: self-efficacy items loading on the first factor and life scheme items loading on the second factor. From each factor, the top 6 items ranked by loading magnitude were selected for inclusion into the final scale.

 

Internal reliabilities for the subscales (6 items each) and the SIWB scale (12 items total) were calculated. A maximum likelihood factor analysis with Varimax rotation also was conducted to verify that a 2-factor solution remained for the reduced 12-item scale.

Validity testing. Well-being is conceptually subsumed within the psychological domain of quality of life measures and is comprised of the dimensions of positive affect (affective) and subjective perceptions of general health and life satisfaction (cognitive).¹² As a result, we determined concurrent construct validity by correlating the 2 6-item subscale scores and the total SIWB score with summed scores from the fear of death items from the DAP-R, the GDS, YOHL, the Physical Functioning Index from the SF-36, and the EuroQol. We anticipated positive correlations of the SIWB with physical functioning (SF-36) and quality of life (EuroQol) and inverse correlations with fear of death (DAP-R), depression (GDS), and self-reported poor health status (YOHL). Discriminant validity was examined by correlating the SIWB subscale and total scores with the religiosity measure. All analyses were performed with the Statistical Package for the Social Sciences version 9.0 (SPSS, Chicago, IL, 1996).

Results

Study population

Two hundred seventy-seven patients were in the final cohort and participated in the study Table 1. The mean age of the study population was 74 years, with a range of 65 to 90 years. Most participants (66%) were 75 years or younger, and the population was evenly distributed between males and females. Participants were predominantly white (78%), reported a wide range of education levels, and had a mean physical function score (SF-36) of 62.92 and a mean health status score (EuroQol) of 0.77.

Internal consistency and factor analysis

Twelve items, 6 each from the self-efficacy and life scheme subscales, remained from the original 40 items after item reduction; initial reliability testing and factor analysis were performed. This 12-item measure of the SIWB produced a coefficient α of .87, indicating good internal consistency. The 6-item subscales also demonstrated good reliability: .83 for self-efficacy and .80 for life scheme.

Results of factor analysis with individual items and item loadings for the final SIWB scale are presented in Table 2. A confirmatory approach anticipated 2 factors, which was based on our conceptual framework. Factor analysis found that 2 factors, reasonably named self-efficacy and life scheme, accounted for a substantial proportion of the variance in responses. The eigenvalue for the self-efficacy factor was 2.88, accounting for 24.04% of the total variance. The eigenvalue for the life scheme factor was 2.35, accounting for 19.57% of the total variance. A Pearson chi-square goodness of fit test of the difference between the actual and reproduced correlation patterns was not significant (51.72; df = 43; P = .17), which suggested that a 2-factor solution was reasonable. Table 3 contains the descriptive statistics for the SIWB scale and its subscales.

Validity testing

To provide a more consistent and intuitive interpretation of scores and correlations, SIWB total and subscale scores were produced by reverse scoring and summing items. As a result, higher SIWB scores indicated a greater degree of spirituality or its components. Correlations between the summed SIWB and subscale scores and other health-related measures of well-being are presented in Table 4. The SIWB and its subscales had significant and expected correlations in direction and magnitude with other measures related to subjective well-being. Fear of death and depression (GDS) had the highest inverse correlations with the SIWB and its subscales. Subjective perceptions of general health and life satisfaction, as measured by self-reports of poor health status (YOHL), functional quality of life (EuroQol), and physical functioning (SF-36) had significant correlations with the SIWB.

Although the life scheme subscale did have a significant but small correlation with a previously validated measure of religiosity, the total SIWB scale and self-efficacy subscale did not have a significant correlation with religiosity.

Discussion

The purpose of this study was to evaluate a brief research instrument designed to measure the effect of spirituality on subjective well-being in a patient population. Instruments that are developed to measure health-related quality of life are evaluated according to several criteria, most notably their degree of validity and reliability.²⁸ The SIWB demonstrated very good reliability with good internal consistency for the total and subscales as assessed by α coefficient in a geriatric patient population.

The construct spirituality has multiple dimensions and connotations in health-related settings,²⁹ which challenge the validity testing of any spirituality instrument. We chose a qualitative approach, rather than the use of experts or preexisting measures in health services research, pastoral theology and chaplaincy, and the social sciences, to conceptualize how patients understand and define spirituality, in particular as if affects their well-being. This approach also provided stimulus material for SIWB item selection and scale construction.

 

In our conceptual framework, spirituality within a health context is a state that is comprised primarily of the domains of life scheme and self-efficacy. Patients who report high self-efficacy beliefs regarding their functioning and who view their lives as purposeful and meaningful should score higher on measures of subjective well-being than those who do not hold such beliefs or attitudes. The use of concurrent construct validity testing allowed us to test this assumption through the correlation of SIWB scores with other established proxies of subjective well-being. Face validity may suggest that the SIWB is a measure of affective or cognitive states (eg, depression) or a proxy for self-efficacy and alienation rather than spirituality. Concurrent construct validity testing provided a means to determine the independence of the SIWB from an accepted measure of depression, the GDS.

Although the pilot version of the SIWB consisted of 40 items with positive and negative statements regarding life scheme and personal self-efficacy, only negative items remained after validity and reliability testing. One explanation for the exclusion of positive statements from the SIWB may involve the predominance of a specific component of subjective well-being in older persons, a low level of negative affect. There are several additional components of subjective well-being (eg, positive affect, satisfaction with work or other domains, and life satisfaction),³⁰ that may not be as salient or as operational in an older population.

However, the SIWB consistently had significant and expected correlations in direction and magnitude with other established measures related to subjective well-being. Spirituality had the highest inverse correlations with fear of death, depression, and perceived health status, which are supportive of affective and cognitive dimensions of subjective well-being in our instrument. A modest correlation with the GDS also suggested that the SIWB is a measure that is independent of depression.

Discriminant validity testing was used to differentiate the SIWB from religiosity. The total SIWB scale did not have a significant correlation with a measure of religiosity that has been used in a geriatric population,²⁷ although the life scheme subscale did have a significant but small (r = .18) correlation. The distinction between conceptualizations of religiosity and spirituality is a major consideration in measurement development,³¹ and there are other measures of spirituality that have been used in clinical and research settings. Virtually all are contaminated by the inclusion of items that assess religiosity.⁹ For example, the Functional Assessment of Chronic Illness Therapy-Spiritual Well-Being Scale contains items that measure the comfort and strength derived from religious faith, in addition to a sense of meaning, purpose, and peace in life.³² The Systems of Belief Inventory, which was designed for use in quality of life and psychosocial research examining illness adjustment, measures religious and spiritual beliefs and practices and the social support that accompanies those beliefs and practices.³³

The Spiritual Well-Being Scale has been used widely in health care settings and consists of 2 subscales: a religious well-being subscale and an existential well-being subscale.³⁴ Religious well-being is conceptualized as the quality of one’s relationship with God, whereas existential well-being includes characteristics such as life purpose, life satisfaction, and positive and negative life experiences. Scores from the Spiritual Well-Being Scale have been inversely correlated with measures of psychological well-being.

However, much of this unpublished research has been compromised by ceiling effects or an inability to detect differences in those who score high on the scale, particularly in religious populations³⁵ and by a lack of peer review.³⁶

Our study has several limitations. Our conceptualization of spirituality is a new construct based on qualitative research, and the study purpose was to evaluate the psychometric properties of a new instrument to measure this construct. As a result, we did not analyze or report normative data about the SIWB. Spirituality may have conceptual overlap with existing constructs, such as self-efficacy and alienation, and we did not evaluate the independence of our scale against these constructs. The SIWB was embedded in the final cohort of a longitudinal study, and we were unable to perform test-retest reliability to determine the stability and the responsiveness or sensitivity of the instrument over time. Due to subject burden, the parent study limited the inclusion of additional measures and the quality-of-life instruments were selected a priori.

Our cross-sectional design also did not allow us to draw any definitive conclusions about the causal relations of the variables. The study population consisted primarily of predominantly white, older patients with some functional limitations, and the generalizability of our findings to other populations is uncertain. However, good theory development and item construction from prior qualitative studies, a high α coefficient, and factor analysis support the validity and reliability of our measure.

 

In summary, the SIWB appears to be a valid and reliable measure of patient subjective well-being, one that is uncontaminated by the inclusion of religiosity. This instrument may be used in observational studies of chronic illness, aging, and end-of-life care that use spirituality as an explanatory or predictor variable of well-being. Future validation studies with multiple, diverse populations and a longitudinal design are needed to refine, modify, or verify the SIWB as an additional, complementary instrument of wellbeing.

Acknowledgments

We thank Lynn Maxwell, Annette Becker, Danielle Sirchak, Donna Clausen, June Belt, Marjoire Frank, and Lisa Rogers for their dedicated service in this study.

Spirituality and religion are embedded within contemporary American culture¹ and have become an increasingly important part of the patient experience of health and illness.² There is growing interest in examining the association of spirituality, religion, and health-related outcomes in the United States, particularly in the areas of health behavior and promotion³ and psychoneuroimmunology.⁴ Despite this interest, the absence of operational definitions of spirituality and religion, the contamination of spirituality items with measures of religion, and the lack of valid and reliable instruments that gauge these constructs continue to be major limitations to work in this area.⁵

Conceptually, religion or religiosity is often viewed in terms of the various organized, individual, and attitudinal manifestations of different faith traditions, and spirituality connotes and expresses a sense of meaning, purpose, or power from within or from a transcendent source.⁶ There is no shortage of instruments that measure dimensions of either construct, and researchers from the fields of sociology,⁷ psychology,⁶ and pastoral theology and chaplaincy⁸ have developed a variety of scales of religion and spirituality.⁹ It remains unclear, however, whether these constructs can be extended to health care settings or whether these instruments are applicable and useful as measures of individual or population health. For example, frequency of religious service attendance is often a single-item measure used as an independent variable in studies of health outcomes, such as health status.¹⁰ Although service attendance is associated with self-reported health in community-dwelling elderly individuals, the effect of this activity on perceived health disappears when functional status is controlled.¹⁰ Therefore, can religious service attendance be considered an independent variable, or is it simply a proxy of functional status within a geriatric population?¹¹

This example highlights the importance of context in the use of any measure of religion or spirituality. It also points to the health-related quality of life field as a useful orientation for conceptualizing spirituality and religion in health care settings. Health-related quality of life, an individual’s or group’s perception of health over time, is predicated on the assumption that a patient’s experiences, beliefs, expectations, and perceptions directly influence the physical, psychological, and social domains of health.¹² Spirituality and religion have been proposed as mediators of 1 characteristic of psychological health, subjective well-being, in 4 ways: by ensuring social support and integration within a community, by establishing personal relationships with a divine other, by promoting a salubrious personal lifestyle that is congruent with a personal faith tradition, and by providing systems of meaning and existential coherence.¹³

To identify and describe elements of spirituality that are linked to subjective well-being, our prior qualitative work explored the patient perspective. We found that patients consider spirituality in predominantly cognitive terms and incorporate the domains of life scheme and positive intentionality, or self-efficacy, as primary components Figure 1.¹⁴ In addition to suggesting a dynamic conceptual framework, this research supported the assumption that patients associate spirituality with well-being largely through the provision of systems of meaning and coherence.

The current study builds on this work and describes the development and evaluation of a brief research instrument, the Spirituality Index of Well-Being (SIWB), which is designed to measure the effect of spirituality on subjective well-being. Several assumptions guided our study design and analysis. First, we recognized that there are no global yet parsimonious instruments that capture the complexity and depth of spirituality in any context, health care or otherwise. Second, based on our qualitative work, we viewed spirituality as subsumed within the psychological rather than within the social or physical domain. Third, we considered the SIWB as a health-related quality of life measure, one to be used in studies of individual or population health, rather than as an assessment tool.

 

From the cultural and social perspectives, spirituality and religion are especially salient in the lives of minority elderly,^15,16 particularly within the settings of serious illness and end-of-life care.¹⁷ From a population health perspective, increased life expectancy in the United States highlights the importance of health-related quality of life assessment in the areas of chronic illness, aging, and end-of-life care, and Healthy People 2010 has identified quality of life improvement as a specific public health objective.¹⁸ By bridging both perspectives, the SIWB has the potential to add a unique and patient-centered dimension to health-related quality of life research.

Methods

Scale and item development

The SIWB was designed as a research tool to measure the effect of patient-reported spirituality on subjective well-being. Our understanding of spirituality and the stimulus material for the index have been described elsewhere.¹⁴ In brief, a congruent, meaningful life scheme and a high degree of positive intentionality or self-efficacy promote personal agency, an intermediary between spirituality and subjective well-being Figure.

Life scheme is similar to the construct of sense of coherence, which was described by Antonovsky as a positive, pervasive way of viewing the world, and one’s life in it, lending elements of comprehensibility, manageability, and meaningfulness.¹⁹ Positive intentionality shares characteristics with self-efficacy, which is an individual’s belief in the capacity to organize and perform activities that are required for a prescribed goal.²⁰ Self-efficacy beliefs are domain and task specific, and participants in our focus group study depicted these beliefs within the context of overcoming threatened or actual changes to their functioning.

Forty items, 20 for the life scheme domain and 20 for the self-efficacy domain, were developed by investigators who conducted the qualitative study (T.P.D., B.B.F.). The scale was prefaced by the question, “Which statement best describes your feelings and choices,” and each item was a statement accompanied by a 5-point Likert scale response from “strongly agree” to “strongly disagree,” with the midpoint representing “neither agree nor disagree.” Item content consisted of positive and negative statements regarding life scheme (eg, “I haven’t yet found my life’s purpose”) and personal self-efficacy (eg, “Despite any problem that I may face, I can get through the day”).

Study population

Participants were 65 years or older and enrolled in a cohort study to assess the ability of performance measures to predict future health service use, health status, and functional status. Recruitment for the parent study occurred between April and November 1996 from primary care sites within the Veteran’s Affairs network and a Medicare health management organization serving the Kansas City metropolitan area. The study population represented the cohort 36 months after enrollment.

Measures

Demographic information. Participants had the following demographic information collected: age, sex, race, and education level.

Health and functional status. Subjective health status was measured by the EuroQol, a recognized quality-of-life measure,²¹ in addition to a single-item measure of global health from the Years of Healthy Life (YOHL) Scale.²² The Physical Functioning Index of the Medical Outcomes Study Short Form 36 was used to assess functional status.²³

Mental health status. We measured mental health status with the Geriatric Depression Scale (GDS), a 15-item instrument with a dichotomous (yes/no) response format.²⁴ Items from the fear of death domain of the Death Attitude Profile Scale-Revised (DAP-R) were selected as an additional proxy of psychological well-being.²⁵

Religiosity. Five items derived from questions developed by the National Opinion Research Center²⁶ were preferentially selected according to a previously tested and validated model of religiosity.²⁷ Frequency of religious or spiritual service attendance was used to assess organizational religiosity, and frequency of private prayer or spiritual practice was used to measure nonorganizational religiosity. Three items were used to measure subjective or intrinsic religiosity: self-reported strength of religious or spiritual orientation, closeness to God (or a Higher Force), and frequency of affective spiritual experiences..

Data analysis

Item reduction and reliability testing. The initial 40-item pool was reduced to 20 life scheme items and 14 self-efficacy items based on subject response and feedback during survey administration. Items that subjects could not understand or answer by self-report were removed.

First, internal reliability analyses were conducted for each subscale (life scheme, self-efficacy) and for the SIWB scale with a goal of producing high internal consistency as measured by the Cronbach’s α (eg, > .70). Items that contributed to lower internal reliability were discarded, which removed 1 self-efficacy item and 6 life scheme items from the scale.

To further refine the SIWB and its subscales, the remaining items were subjected to principal components analysis by using Varimax rotation. After rotation, the 2 largest factors were readily interpretable, with items loading as expected: self-efficacy items loading on the first factor and life scheme items loading on the second factor. From each factor, the top 6 items ranked by loading magnitude were selected for inclusion into the final scale.

 

Internal reliabilities for the subscales (6 items each) and the SIWB scale (12 items total) were calculated. A maximum likelihood factor analysis with Varimax rotation also was conducted to verify that a 2-factor solution remained for the reduced 12-item scale.

Validity testing. Well-being is conceptually subsumed within the psychological domain of quality of life measures and is comprised of the dimensions of positive affect (affective) and subjective perceptions of general health and life satisfaction (cognitive).¹² As a result, we determined concurrent construct validity by correlating the 2 6-item subscale scores and the total SIWB score with summed scores from the fear of death items from the DAP-R, the GDS, YOHL, the Physical Functioning Index from the SF-36, and the EuroQol. We anticipated positive correlations of the SIWB with physical functioning (SF-36) and quality of life (EuroQol) and inverse correlations with fear of death (DAP-R), depression (GDS), and self-reported poor health status (YOHL). Discriminant validity was examined by correlating the SIWB subscale and total scores with the religiosity measure. All analyses were performed with the Statistical Package for the Social Sciences version 9.0 (SPSS, Chicago, IL, 1996).

Results

Study population

Two hundred seventy-seven patients were in the final cohort and participated in the study Table 1. The mean age of the study population was 74 years, with a range of 65 to 90 years. Most participants (66%) were 75 years or younger, and the population was evenly distributed between males and females. Participants were predominantly white (78%), reported a wide range of education levels, and had a mean physical function score (SF-36) of 62.92 and a mean health status score (EuroQol) of 0.77.

Internal consistency and factor analysis

Twelve items, 6 each from the self-efficacy and life scheme subscales, remained from the original 40 items after item reduction; initial reliability testing and factor analysis were performed. This 12-item measure of the SIWB produced a coefficient α of .87, indicating good internal consistency. The 6-item subscales also demonstrated good reliability: .83 for self-efficacy and .80 for life scheme.

Results of factor analysis with individual items and item loadings for the final SIWB scale are presented in Table 2. A confirmatory approach anticipated 2 factors, which was based on our conceptual framework. Factor analysis found that 2 factors, reasonably named self-efficacy and life scheme, accounted for a substantial proportion of the variance in responses. The eigenvalue for the self-efficacy factor was 2.88, accounting for 24.04% of the total variance. The eigenvalue for the life scheme factor was 2.35, accounting for 19.57% of the total variance. A Pearson chi-square goodness of fit test of the difference between the actual and reproduced correlation patterns was not significant (51.72; df = 43; P = .17), which suggested that a 2-factor solution was reasonable. Table 3 contains the descriptive statistics for the SIWB scale and its subscales.

Validity testing

To provide a more consistent and intuitive interpretation of scores and correlations, SIWB total and subscale scores were produced by reverse scoring and summing items. As a result, higher SIWB scores indicated a greater degree of spirituality or its components. Correlations between the summed SIWB and subscale scores and other health-related measures of well-being are presented in Table 4. The SIWB and its subscales had significant and expected correlations in direction and magnitude with other measures related to subjective well-being. Fear of death and depression (GDS) had the highest inverse correlations with the SIWB and its subscales. Subjective perceptions of general health and life satisfaction, as measured by self-reports of poor health status (YOHL), functional quality of life (EuroQol), and physical functioning (SF-36) had significant correlations with the SIWB.

Although the life scheme subscale did have a significant but small correlation with a previously validated measure of religiosity, the total SIWB scale and self-efficacy subscale did not have a significant correlation with religiosity.

Discussion

The purpose of this study was to evaluate a brief research instrument designed to measure the effect of spirituality on subjective well-being in a patient population. Instruments that are developed to measure health-related quality of life are evaluated according to several criteria, most notably their degree of validity and reliability.²⁸ The SIWB demonstrated very good reliability with good internal consistency for the total and subscales as assessed by α coefficient in a geriatric patient population.

The construct spirituality has multiple dimensions and connotations in health-related settings,²⁹ which challenge the validity testing of any spirituality instrument. We chose a qualitative approach, rather than the use of experts or preexisting measures in health services research, pastoral theology and chaplaincy, and the social sciences, to conceptualize how patients understand and define spirituality, in particular as if affects their well-being. This approach also provided stimulus material for SIWB item selection and scale construction.

 

In our conceptual framework, spirituality within a health context is a state that is comprised primarily of the domains of life scheme and self-efficacy. Patients who report high self-efficacy beliefs regarding their functioning and who view their lives as purposeful and meaningful should score higher on measures of subjective well-being than those who do not hold such beliefs or attitudes. The use of concurrent construct validity testing allowed us to test this assumption through the correlation of SIWB scores with other established proxies of subjective well-being. Face validity may suggest that the SIWB is a measure of affective or cognitive states (eg, depression) or a proxy for self-efficacy and alienation rather than spirituality. Concurrent construct validity testing provided a means to determine the independence of the SIWB from an accepted measure of depression, the GDS.

Although the pilot version of the SIWB consisted of 40 items with positive and negative statements regarding life scheme and personal self-efficacy, only negative items remained after validity and reliability testing. One explanation for the exclusion of positive statements from the SIWB may involve the predominance of a specific component of subjective well-being in older persons, a low level of negative affect. There are several additional components of subjective well-being (eg, positive affect, satisfaction with work or other domains, and life satisfaction),³⁰ that may not be as salient or as operational in an older population.

However, the SIWB consistently had significant and expected correlations in direction and magnitude with other established measures related to subjective well-being. Spirituality had the highest inverse correlations with fear of death, depression, and perceived health status, which are supportive of affective and cognitive dimensions of subjective well-being in our instrument. A modest correlation with the GDS also suggested that the SIWB is a measure that is independent of depression.

Discriminant validity testing was used to differentiate the SIWB from religiosity. The total SIWB scale did not have a significant correlation with a measure of religiosity that has been used in a geriatric population,²⁷ although the life scheme subscale did have a significant but small (r = .18) correlation. The distinction between conceptualizations of religiosity and spirituality is a major consideration in measurement development,³¹ and there are other measures of spirituality that have been used in clinical and research settings. Virtually all are contaminated by the inclusion of items that assess religiosity.⁹ For example, the Functional Assessment of Chronic Illness Therapy-Spiritual Well-Being Scale contains items that measure the comfort and strength derived from religious faith, in addition to a sense of meaning, purpose, and peace in life.³² The Systems of Belief Inventory, which was designed for use in quality of life and psychosocial research examining illness adjustment, measures religious and spiritual beliefs and practices and the social support that accompanies those beliefs and practices.³³

The Spiritual Well-Being Scale has been used widely in health care settings and consists of 2 subscales: a religious well-being subscale and an existential well-being subscale.³⁴ Religious well-being is conceptualized as the quality of one’s relationship with God, whereas existential well-being includes characteristics such as life purpose, life satisfaction, and positive and negative life experiences. Scores from the Spiritual Well-Being Scale have been inversely correlated with measures of psychological well-being.

However, much of this unpublished research has been compromised by ceiling effects or an inability to detect differences in those who score high on the scale, particularly in religious populations³⁵ and by a lack of peer review.³⁶

Our study has several limitations. Our conceptualization of spirituality is a new construct based on qualitative research, and the study purpose was to evaluate the psychometric properties of a new instrument to measure this construct. As a result, we did not analyze or report normative data about the SIWB. Spirituality may have conceptual overlap with existing constructs, such as self-efficacy and alienation, and we did not evaluate the independence of our scale against these constructs. The SIWB was embedded in the final cohort of a longitudinal study, and we were unable to perform test-retest reliability to determine the stability and the responsiveness or sensitivity of the instrument over time. Due to subject burden, the parent study limited the inclusion of additional measures and the quality-of-life instruments were selected a priori.

Our cross-sectional design also did not allow us to draw any definitive conclusions about the causal relations of the variables. The study population consisted primarily of predominantly white, older patients with some functional limitations, and the generalizability of our findings to other populations is uncertain. However, good theory development and item construction from prior qualitative studies, a high α coefficient, and factor analysis support the validity and reliability of our measure.

 

In summary, the SIWB appears to be a valid and reliable measure of patient subjective well-being, one that is uncontaminated by the inclusion of religiosity. This instrument may be used in observational studies of chronic illness, aging, and end-of-life care that use spirituality as an explanatory or predictor variable of well-being. Future validation studies with multiple, diverse populations and a longitudinal design are needed to refine, modify, or verify the SIWB as an additional, complementary instrument of wellbeing.

Acknowledgments

We thank Lynn Maxwell, Annette Becker, Danielle Sirchak, Donna Clausen, June Belt, Marjoire Frank, and Lisa Rogers for their dedicated service in this study.

The randomized controlled trial (RCT) has become the most important test of therapeutic benefit.¹ When evaluating an RCT, readers should determine whether the analysis was by intention to treat (ITT).^1-4 ITT analysis, often described as “once randomized, always analyzed,”⁵ is the practice of attributing all participants to the group to which they were randomized, regardless of what subsequently occurred.^2,6,7 ITT analysis avoids the problems created by omitting dropouts and noncompliant patients, which can negate randomization, introduce bias, and overestimate clinical effectiveness.^2,8

Surveys of the literature found that ITT analysis was reported in 7% to 48% of RCTs^8-10; however, reporting ITT analysis does not guarantee that the analysis was conducted properly or that the results promoted by the authors were derived from the ITT analysis. For articles reporting ITT analysis of an RCT, we specifically examined which participants were included in the analysis.

Methods

We searched MEDLINE for abstracts that included the text words “intention to treat” or “intent to treat,” limiting the results to randomized controlled trials published in English during 1999. We entered the resultant studies in a database (FileMaker Pro 4.0; FileMaker, Inc., Santa Clara, CA), ordered them using the database’s random number function, and reviewed the first 100 eligible studies.

Two of us (in a rotating fashion cycling through each pair-wise combination of the 6 authors) were systematically assigned to review each article. We used a structured form (available on request from the authors) to evaluate each article for the number of subjects randomized, the number in the ITT analysis, the number in the primary analysis, which categories of subjects were in the ITT analysis, and where ITT was defined within the article. We defined the primary analysis as the most prominently featured outcome in the abstract. Two of us (R.L.K., J.J.S.) independently assessed whether articles contained a diagram showing the flow of participants through each stage, a feature strongly recommended in the Consolidated Standards of Reporting Trials (CONSORT).³ All coauthors discussed discrepant results and made final determinations using majority voting. We conducted all analyses using SAS software (SAS system for Windows, Release 8.0; SAS Institute, Cary, NC).

Results

The MEDLINE search identified 335 studies. We reviewed 129 articles to obtain 100 eligible studies (Figure). We excluded articles for the following reasons: the words “intent” or “intention” and “treat” did not refer to a method of analysis (14); the study was not an RCT (5); the study involved randomized groups (eg, villages or hospitals) rather than individuals (4); the study presented a secondary analysis of a previously published study (4); the study used a crossover design (1) or described a trial protocol without results (1). The paired reviewers agreed on all abstracted data elements for 83 articles; there was a disagreement on one or more items for the remainder, which were determined by committee.

Of 100 studies selected, 42 included all randomized subjects in the ITT analysis (Table). Among those studies that excluded randomized patients from analysis, the most common reasons given were that the patients received no follow-up after randomization (16) or received none of the allocated treatment (14). For 13 studies, we could not determine which categories of participants were excluded from the ITT analysis.

We used the number of subjects randomized, the number in the ITT analysis, and the number in the primary analysis to determine the proportion of randomized subjects in both the ITT and primary analyses. Ideally, 100% of randomized subjects should be included in the ITT analysis. The proportion of randomized subjects included in the ITT analysis could be determined for 92 studies, and ranged from 69% to 100%, with a median of 99%. Nineteen of the 92 studies (21%) excluded more than 5% of randomized participants from the ITT analysis, while 10 studies (11%) excluded more than 10%.

We could determine the proportion of randomized participants in the primary analysis in 93 studies; it varied from 49% to 100%, with a median of 98.7%. Ten of the 93 studies (11%) excluded more than 20% of participants from the primary analysis. In 16 of the 93 studies (17%), a non-ITT analysis (eg, “per protocol”) was presented as the primary analysis. In these studies, an average of 80.1% (median, 82.4%; range, 49.0% to 92.4%) of randomized patients were included in the primary analysis.

 

Fifty-six studies included a definition of the ITT population, primarily within the methods (38) and results (18) sections. Of the 42 studies where all randomized subjects were analyzed, 20 included definitions of ITT. Diagrams showing the flow of participants through each trial were present in 41 of 100 articles, including 1 on a journal’s web site. An additional 8 articles had diagrams that showed patient flow without giving the number of patients. Presence of a flow diagram was not related to whether or not all randomized subjects were included in the ITT analysis (36% vs 45% respectively, P = .37). Of the 31 articles from journals that participate in CONSORT, 29 included flow diagrams, compared with 12 of the 69 articles from journals that do not participate in CONSORT (P < .0001).

TABLE
Categories of randomized patients excluded from ITT analysis*

Category	Number of studies
All randomized subjects were analyzed (true intention to treat)	42
Some randomized subjects were excluded	58
Subjects found not to meet entry criteria	12
Subjects who did not receive any of the assigned treatment	14
Subjects who received some but not all of the assigned treatment	1
Subjects with no follow-up after randomization	16
Subjects with some but not all follow-up achieved	1
Subjects who dropped out for selected reasons	4
Subjects with specific protocol violations	2
Subjects with protocol violations but details not given	2
Other	9
Author needs to be contacted to determine who was in the ITT group	13
*Reports of 100 randomized trials were analyzed. Studies could have more than one group of excluded subjects. ITT, intention to treat.

Discussion

The hallmark of ITT analysis is that all randomized subjects are analyzed.⁷ In more than half of the articles we examined, this was not the case. Analysis of only certain subgroups of patients is sometimes appropriate, but an explanation should be provided whenever subjects are left out of any analysis. For example, we examined a report of a trial that was stopped based on the results of an interim analysis, thus excluding subjects who were randomized after the interim analysis.¹¹ This type of exclusion, based on an a priori decision rather than individual characteristics or behavior, is less likely to bias results.

While all the articles in our sample reported analysis by ITT, many authors did not define the term, even when they excluded some randomized subjects from the ITT analysis. In these cases, the reader is left to infer which subjects were excluded based on information given in the text, figures, and tables.

Despite numerous recommendations for detailed reporting of RCT methods,^1-4 many articles were vague and lacked detail. We could not determine which categories of participants were excluded from the ITT analysis in 13 articles. In 8 of the 100 articles we examined, we could not determine how many subjects were randomized or included in the ITT or primary analysis. Four of these 8 articles were in journals that endorsed the CONSORT statement. All were published well after the initial CONSORT statement was released in 1996.¹

The number of randomized subjects excluded from the ITT analysis was usually small. It is unlikely that excluding up to 1% of subjects had a major effect on the results. In 11% of our sample, however, more than 10% of randomized subjects were excluded. Exclusions of this magnitude have significant potential to alter the findings. When outcome data can’t be determined and the outcome is categorical (eg, alive/dead), it can be helpful to produce best-case and worst-case scenarios in which patients lost to follow-up are arbitrarily ascribed good or bad outcomes. These extremes delimit the potential effect of the exclusions on results.¹² Similarly, missing continuous outcomes (eg, weight change) can be assigned specific values to determine the potential impact on the results.

We assessed only articles that mentioned ITT in the abstract, so we probably missed some studies that used ITT analysis; however, we doubt that this caused us to significantly underestimate accurate use of the term ITT. The articles came from a wide spectrum of journals (62), of which 21 were listed in the Abridged Index Medicus subset. The 17 articles requiring a committee vote described the analytic process in terms that were often vague and ambiguous. In these cases, we cannot be certain that we correctly interpreted the authors’ methods; most readers would have similar difficulties.

We found considerable variation in how the term ITT was used in reports of RCTs. Fewer than half of the reports we examined included all randomized subjects in the ITT analysis. While exclusions were negligible in many cases, more than 10% of the subjects were excluded in 10% of the trials. In 7 trials, including some drawn from journals that endorse the CONSORT statement, it was not even possible to determine the number of subjects included in the ITT analysis. These problems highlight the continued need for better reporting of clinical trials.

The randomized controlled trial (RCT) has become the most important test of therapeutic benefit.¹ When evaluating an RCT, readers should determine whether the analysis was by intention to treat (ITT).^1-4 ITT analysis, often described as “once randomized, always analyzed,”⁵ is the practice of attributing all participants to the group to which they were randomized, regardless of what subsequently occurred.^2,6,7 ITT analysis avoids the problems created by omitting dropouts and noncompliant patients, which can negate randomization, introduce bias, and overestimate clinical effectiveness.^2,8

Surveys of the literature found that ITT analysis was reported in 7% to 48% of RCTs^8-10; however, reporting ITT analysis does not guarantee that the analysis was conducted properly or that the results promoted by the authors were derived from the ITT analysis. For articles reporting ITT analysis of an RCT, we specifically examined which participants were included in the analysis.

Methods

We searched MEDLINE for abstracts that included the text words “intention to treat” or “intent to treat,” limiting the results to randomized controlled trials published in English during 1999. We entered the resultant studies in a database (FileMaker Pro 4.0; FileMaker, Inc., Santa Clara, CA), ordered them using the database’s random number function, and reviewed the first 100 eligible studies.

Two of us (in a rotating fashion cycling through each pair-wise combination of the 6 authors) were systematically assigned to review each article. We used a structured form (available on request from the authors) to evaluate each article for the number of subjects randomized, the number in the ITT analysis, the number in the primary analysis, which categories of subjects were in the ITT analysis, and where ITT was defined within the article. We defined the primary analysis as the most prominently featured outcome in the abstract. Two of us (R.L.K., J.J.S.) independently assessed whether articles contained a diagram showing the flow of participants through each stage, a feature strongly recommended in the Consolidated Standards of Reporting Trials (CONSORT).³ All coauthors discussed discrepant results and made final determinations using majority voting. We conducted all analyses using SAS software (SAS system for Windows, Release 8.0; SAS Institute, Cary, NC).

Results

The MEDLINE search identified 335 studies. We reviewed 129 articles to obtain 100 eligible studies (Figure). We excluded articles for the following reasons: the words “intent” or “intention” and “treat” did not refer to a method of analysis (14); the study was not an RCT (5); the study involved randomized groups (eg, villages or hospitals) rather than individuals (4); the study presented a secondary analysis of a previously published study (4); the study used a crossover design (1) or described a trial protocol without results (1). The paired reviewers agreed on all abstracted data elements for 83 articles; there was a disagreement on one or more items for the remainder, which were determined by committee.

Of 100 studies selected, 42 included all randomized subjects in the ITT analysis (Table). Among those studies that excluded randomized patients from analysis, the most common reasons given were that the patients received no follow-up after randomization (16) or received none of the allocated treatment (14). For 13 studies, we could not determine which categories of participants were excluded from the ITT analysis.

We used the number of subjects randomized, the number in the ITT analysis, and the number in the primary analysis to determine the proportion of randomized subjects in both the ITT and primary analyses. Ideally, 100% of randomized subjects should be included in the ITT analysis. The proportion of randomized subjects included in the ITT analysis could be determined for 92 studies, and ranged from 69% to 100%, with a median of 99%. Nineteen of the 92 studies (21%) excluded more than 5% of randomized participants from the ITT analysis, while 10 studies (11%) excluded more than 10%.

We could determine the proportion of randomized participants in the primary analysis in 93 studies; it varied from 49% to 100%, with a median of 98.7%. Ten of the 93 studies (11%) excluded more than 20% of participants from the primary analysis. In 16 of the 93 studies (17%), a non-ITT analysis (eg, “per protocol”) was presented as the primary analysis. In these studies, an average of 80.1% (median, 82.4%; range, 49.0% to 92.4%) of randomized patients were included in the primary analysis.

 

Fifty-six studies included a definition of the ITT population, primarily within the methods (38) and results (18) sections. Of the 42 studies where all randomized subjects were analyzed, 20 included definitions of ITT. Diagrams showing the flow of participants through each trial were present in 41 of 100 articles, including 1 on a journal’s web site. An additional 8 articles had diagrams that showed patient flow without giving the number of patients. Presence of a flow diagram was not related to whether or not all randomized subjects were included in the ITT analysis (36% vs 45% respectively, P = .37). Of the 31 articles from journals that participate in CONSORT, 29 included flow diagrams, compared with 12 of the 69 articles from journals that do not participate in CONSORT (P < .0001).

TABLE
Categories of randomized patients excluded from ITT analysis*

Category	Number of studies
All randomized subjects were analyzed (true intention to treat)	42
Some randomized subjects were excluded	58
Subjects found not to meet entry criteria	12
Subjects who did not receive any of the assigned treatment	14
Subjects who received some but not all of the assigned treatment	1
Subjects with no follow-up after randomization	16
Subjects with some but not all follow-up achieved	1
Subjects who dropped out for selected reasons	4
Subjects with specific protocol violations	2
Subjects with protocol violations but details not given	2
Other	9
Author needs to be contacted to determine who was in the ITT group	13
*Reports of 100 randomized trials were analyzed. Studies could have more than one group of excluded subjects. ITT, intention to treat.

Discussion

The hallmark of ITT analysis is that all randomized subjects are analyzed.⁷ In more than half of the articles we examined, this was not the case. Analysis of only certain subgroups of patients is sometimes appropriate, but an explanation should be provided whenever subjects are left out of any analysis. For example, we examined a report of a trial that was stopped based on the results of an interim analysis, thus excluding subjects who were randomized after the interim analysis.¹¹ This type of exclusion, based on an a priori decision rather than individual characteristics or behavior, is less likely to bias results.

While all the articles in our sample reported analysis by ITT, many authors did not define the term, even when they excluded some randomized subjects from the ITT analysis. In these cases, the reader is left to infer which subjects were excluded based on information given in the text, figures, and tables.

Despite numerous recommendations for detailed reporting of RCT methods,^1-4 many articles were vague and lacked detail. We could not determine which categories of participants were excluded from the ITT analysis in 13 articles. In 8 of the 100 articles we examined, we could not determine how many subjects were randomized or included in the ITT or primary analysis. Four of these 8 articles were in journals that endorsed the CONSORT statement. All were published well after the initial CONSORT statement was released in 1996.¹

The number of randomized subjects excluded from the ITT analysis was usually small. It is unlikely that excluding up to 1% of subjects had a major effect on the results. In 11% of our sample, however, more than 10% of randomized subjects were excluded. Exclusions of this magnitude have significant potential to alter the findings. When outcome data can’t be determined and the outcome is categorical (eg, alive/dead), it can be helpful to produce best-case and worst-case scenarios in which patients lost to follow-up are arbitrarily ascribed good or bad outcomes. These extremes delimit the potential effect of the exclusions on results.¹² Similarly, missing continuous outcomes (eg, weight change) can be assigned specific values to determine the potential impact on the results.

We assessed only articles that mentioned ITT in the abstract, so we probably missed some studies that used ITT analysis; however, we doubt that this caused us to significantly underestimate accurate use of the term ITT. The articles came from a wide spectrum of journals (62), of which 21 were listed in the Abridged Index Medicus subset. The 17 articles requiring a committee vote described the analytic process in terms that were often vague and ambiguous. In these cases, we cannot be certain that we correctly interpreted the authors’ methods; most readers would have similar difficulties.

We found considerable variation in how the term ITT was used in reports of RCTs. Fewer than half of the reports we examined included all randomized subjects in the ITT analysis. While exclusions were negligible in many cases, more than 10% of the subjects were excluded in 10% of the trials. In 7 trials, including some drawn from journals that endorse the CONSORT statement, it was not even possible to determine the number of subjects included in the ITT analysis. These problems highlight the continued need for better reporting of clinical trials.

The randomized controlled trial (RCT) has become the most important test of therapeutic benefit.¹ When evaluating an RCT, readers should determine whether the analysis was by intention to treat (ITT).^1-4 ITT analysis, often described as “once randomized, always analyzed,”⁵ is the practice of attributing all participants to the group to which they were randomized, regardless of what subsequently occurred.^2,6,7 ITT analysis avoids the problems created by omitting dropouts and noncompliant patients, which can negate randomization, introduce bias, and overestimate clinical effectiveness.^2,8

Surveys of the literature found that ITT analysis was reported in 7% to 48% of RCTs^8-10; however, reporting ITT analysis does not guarantee that the analysis was conducted properly or that the results promoted by the authors were derived from the ITT analysis. For articles reporting ITT analysis of an RCT, we specifically examined which participants were included in the analysis.

Methods

We searched MEDLINE for abstracts that included the text words “intention to treat” or “intent to treat,” limiting the results to randomized controlled trials published in English during 1999. We entered the resultant studies in a database (FileMaker Pro 4.0; FileMaker, Inc., Santa Clara, CA), ordered them using the database’s random number function, and reviewed the first 100 eligible studies.

Two of us (in a rotating fashion cycling through each pair-wise combination of the 6 authors) were systematically assigned to review each article. We used a structured form (available on request from the authors) to evaluate each article for the number of subjects randomized, the number in the ITT analysis, the number in the primary analysis, which categories of subjects were in the ITT analysis, and where ITT was defined within the article. We defined the primary analysis as the most prominently featured outcome in the abstract. Two of us (R.L.K., J.J.S.) independently assessed whether articles contained a diagram showing the flow of participants through each stage, a feature strongly recommended in the Consolidated Standards of Reporting Trials (CONSORT).³ All coauthors discussed discrepant results and made final determinations using majority voting. We conducted all analyses using SAS software (SAS system for Windows, Release 8.0; SAS Institute, Cary, NC).

Results

The MEDLINE search identified 335 studies. We reviewed 129 articles to obtain 100 eligible studies (Figure). We excluded articles for the following reasons: the words “intent” or “intention” and “treat” did not refer to a method of analysis (14); the study was not an RCT (5); the study involved randomized groups (eg, villages or hospitals) rather than individuals (4); the study presented a secondary analysis of a previously published study (4); the study used a crossover design (1) or described a trial protocol without results (1). The paired reviewers agreed on all abstracted data elements for 83 articles; there was a disagreement on one or more items for the remainder, which were determined by committee.

Of 100 studies selected, 42 included all randomized subjects in the ITT analysis (Table). Among those studies that excluded randomized patients from analysis, the most common reasons given were that the patients received no follow-up after randomization (16) or received none of the allocated treatment (14). For 13 studies, we could not determine which categories of participants were excluded from the ITT analysis.

We used the number of subjects randomized, the number in the ITT analysis, and the number in the primary analysis to determine the proportion of randomized subjects in both the ITT and primary analyses. Ideally, 100% of randomized subjects should be included in the ITT analysis. The proportion of randomized subjects included in the ITT analysis could be determined for 92 studies, and ranged from 69% to 100%, with a median of 99%. Nineteen of the 92 studies (21%) excluded more than 5% of randomized participants from the ITT analysis, while 10 studies (11%) excluded more than 10%.

We could determine the proportion of randomized participants in the primary analysis in 93 studies; it varied from 49% to 100%, with a median of 98.7%. Ten of the 93 studies (11%) excluded more than 20% of participants from the primary analysis. In 16 of the 93 studies (17%), a non-ITT analysis (eg, “per protocol”) was presented as the primary analysis. In these studies, an average of 80.1% (median, 82.4%; range, 49.0% to 92.4%) of randomized patients were included in the primary analysis.

 

Fifty-six studies included a definition of the ITT population, primarily within the methods (38) and results (18) sections. Of the 42 studies where all randomized subjects were analyzed, 20 included definitions of ITT. Diagrams showing the flow of participants through each trial were present in 41 of 100 articles, including 1 on a journal’s web site. An additional 8 articles had diagrams that showed patient flow without giving the number of patients. Presence of a flow diagram was not related to whether or not all randomized subjects were included in the ITT analysis (36% vs 45% respectively, P = .37). Of the 31 articles from journals that participate in CONSORT, 29 included flow diagrams, compared with 12 of the 69 articles from journals that do not participate in CONSORT (P < .0001).

TABLE
Categories of randomized patients excluded from ITT analysis*

Category	Number of studies
All randomized subjects were analyzed (true intention to treat)	42
Some randomized subjects were excluded	58
Subjects found not to meet entry criteria	12
Subjects who did not receive any of the assigned treatment	14
Subjects who received some but not all of the assigned treatment	1
Subjects with no follow-up after randomization	16
Subjects with some but not all follow-up achieved	1
Subjects who dropped out for selected reasons	4
Subjects with specific protocol violations	2
Subjects with protocol violations but details not given	2
Other	9
Author needs to be contacted to determine who was in the ITT group	13
*Reports of 100 randomized trials were analyzed. Studies could have more than one group of excluded subjects. ITT, intention to treat.

Discussion

The hallmark of ITT analysis is that all randomized subjects are analyzed.⁷ In more than half of the articles we examined, this was not the case. Analysis of only certain subgroups of patients is sometimes appropriate, but an explanation should be provided whenever subjects are left out of any analysis. For example, we examined a report of a trial that was stopped based on the results of an interim analysis, thus excluding subjects who were randomized after the interim analysis.¹¹ This type of exclusion, based on an a priori decision rather than individual characteristics or behavior, is less likely to bias results.

While all the articles in our sample reported analysis by ITT, many authors did not define the term, even when they excluded some randomized subjects from the ITT analysis. In these cases, the reader is left to infer which subjects were excluded based on information given in the text, figures, and tables.

Despite numerous recommendations for detailed reporting of RCT methods,^1-4 many articles were vague and lacked detail. We could not determine which categories of participants were excluded from the ITT analysis in 13 articles. In 8 of the 100 articles we examined, we could not determine how many subjects were randomized or included in the ITT or primary analysis. Four of these 8 articles were in journals that endorsed the CONSORT statement. All were published well after the initial CONSORT statement was released in 1996.¹

The number of randomized subjects excluded from the ITT analysis was usually small. It is unlikely that excluding up to 1% of subjects had a major effect on the results. In 11% of our sample, however, more than 10% of randomized subjects were excluded. Exclusions of this magnitude have significant potential to alter the findings. When outcome data can’t be determined and the outcome is categorical (eg, alive/dead), it can be helpful to produce best-case and worst-case scenarios in which patients lost to follow-up are arbitrarily ascribed good or bad outcomes. These extremes delimit the potential effect of the exclusions on results.¹² Similarly, missing continuous outcomes (eg, weight change) can be assigned specific values to determine the potential impact on the results.

We assessed only articles that mentioned ITT in the abstract, so we probably missed some studies that used ITT analysis; however, we doubt that this caused us to significantly underestimate accurate use of the term ITT. The articles came from a wide spectrum of journals (62), of which 21 were listed in the Abridged Index Medicus subset. The 17 articles requiring a committee vote described the analytic process in terms that were often vague and ambiguous. In these cases, we cannot be certain that we correctly interpreted the authors’ methods; most readers would have similar difficulties.

We found considerable variation in how the term ITT was used in reports of RCTs. Fewer than half of the reports we examined included all randomized subjects in the ITT analysis. While exclusions were negligible in many cases, more than 10% of the subjects were excluded in 10% of the trials. In 7 trials, including some drawn from journals that endorse the CONSORT statement, it was not even possible to determine the number of subjects included in the ITT analysis. These problems highlight the continued need for better reporting of clinical trials.

Every primary care clinician in the United States knows the frustration of lost charts, misplaced reports, and messages from patients that should have been answered yesterday. These are some of the common frustrations and failures in day-to-day clinical practice. Many clinicians also know the guilt, shame, and self-doubt that occur when patients suffer a serious complication or die due to a mistake made by the clinician, health care team, or health care system. Between the common frustrations of practice and the rare patient death due to an error lies a large chasm, a rarely explored territory of relationships, causes and effects, and mitigating factors. Looking backward from a catastrophic patient outcome rarely goes beyond blaming the immediate person “at fault.” ¹ Looking forward from common charting errors rarely goes beyond a conclusion to be “more careful.”

Hospital-based research has categorized preventable adverse outcomes and some process errors associated with them,^2-4 but this has not been done in primary care.⁵ There are difficulties in studying errors in the primary care setting: care takes place in many locations; involves multiple visits; is provided in person, by phone, by mail, and even by computer; and involves interactions with many health care workers. However, it is important to study errors in primary care⁶ because it is the location of most health care visits in the United States.⁷

A classification or taxonomy of errors and preventable adverse events is an important first step in improving patient care. Prevalence and epidemiology studies, clinical and system interventions, and even individual practice group databases of errors and adverse events⁸ can more easily be developed if there is a beginning classification system. Just as clinicians use a differential diagnostic list for analyzing symptoms or a list of risk factors for assessing disease, so, too, can clinicians use a classification and listing of process errors and preventable adverse events to “diagnose” and “prevent” patient harm from medical care. Many taxonomies of medical error do exist and have been used in hospital accreditation or malpractice contexts for some time.⁹ These taxonomies have not been generally available for purposes other than their intended use, ie, to help their developers understand the data they were dealing with, and because these data do not originate from primary care practice, it remains unknown how well the taxonomies might meet the needs of family physicians and other primary care researchers.

 

The purpose of this study was to use published data from original research to understand and classify process errors and preventable adverse events associated with primary medical care. Through a systematic review and synthesis of the medical literature, we developed a classification of medical errors relevant to primary care.

Methods

To identify eligible published English-language original research articles, we searched MEDLINE and the Cochrane Library from 1965 through March 2001 with the MESH search term medical errors, modified by adding family practice, primary health care, physicians/family, or ambulatory care to the primary term. Published bibliographies from the National Patient Safety Foundation (NPSF) and the Institute for Healthcare Improvement (IHI) were also reviewed. The Web sites of the American Academy of Family Physicians, the American College of Physicians–American Society of Internal Medicine, the Institute of Medicine, the NPSF, and the IHI were also reviewed for unpublished reports, presentations, and leads to other sites, journals, or investigators with relevant work. Additional papers were identified from the references of the papers reviewed, from seminal papers in the field, and from discussion with others working in the field of patient safety or quality improvement in primary care.

We reviewed titles of 379 articles identified by electronic searches for inclusion. We excluded papers if they related to comparisons of different approaches to diagnosis or treatment of specific diseases, the evaluation of teaching or research tools, or exclusively to hospitalized patients. If there was uncertainty as to the appropriateness of an article, we read the abstract. We reviewed complete papers if they appeared from the title and abstract to report original research involving a broad assessment of medical errors and preventable adverse events in primary care. Data relating to topic, study quality, and research results were abstracted from identified papers. Both authors performed independent MEDLINE searches and reviewed citations in the papers. To broaden the search for potential studies, one author searched Web sites and NPSF and IHI bibliographies. Both authors agreed on the inclusion of the chosen studies, appraised them independently, and abstracted key classification components. One author (N.C.E.) initially prepared the classification system presented here; it was then reviewed by both authors and revised after their discussions.

Results

Four original research studies directly studied and described medical errors and preventable adverse events in primary care.^10-13 Three other studies peripherally addressed primary care medical errors as part of an investigation with another central focus^14-16 (Table 1).

TABLE 1
Primary care studies describing medical error

Study	Research purpose	Definition of error	Method	Pertinent results
Primary care studies directly describing medical error
Bhasale et al10	Describe incidents occurring in general practice	An unintended event, no matter how seemingly trivial or commonplace, that could have harmed or did harm a patient	Self-report by 324 Australian sentinel research network FPs using reporting cards	805 incidents reported, 76% preventable; categories were drug management, non-drug management, diagnosis, and equipment; causes included communication, actions of others, and clinical judgment errors
Ely et al12	Describe the causes to which family physicians attribute errors	Act or omission for which the physician felt responsible and which had serious consequences for the patient	30-min interviews with 53 randomly chosen Iowa FPs	53 errors reported: delayed diagnoses, surgical and medical treatment mishaps; causes included physical stressors, process of care factors, patient related factors, and physician characteristics
Dovey et al11	Describe medical errors reported by FPs	Something in one’s practice that should not have happened, that was not anticipated, and that makes one say, “I don’t want it to happen again”	Self-report by 42 American research network FPs using electronic and reply card reporting	330 reported errors, 83% from health care system and 13% from knowledge and skills; subcategories were office administration, investigations, treatments, communication, execution of clinical tasks, misdiagnosis, and wrong treatment decision
Fischer et al13	Describe the prevalence of adverse events in a risk management database	Incidents resulting in, or having the potential for, physical, emotional, or financial liability for the patient	Review of incident reports entered by 8 primary care clinics into risk management database	Prevalence of adverse events was 3.7/100,000 clinic visits, 83% were preventable; categories included diagnostic, treatment, and preventive and other errors
Primary care studies peripherally describing medical error
Holden et al15	Determine patterns of death and potential preventive factors		Formal review of all patient deaths in a group of general practices	5.1% of deaths due to preventable FP factors; 2 main categories were delay of diagnosis and treatment and lack of prevention with aspirin therapy
Gandhi et al14communication	Evaluate primary care and specialist inter physician		Surveys in academic medical center	Main issues for doctors were lack of timeliness and inadequate content
Britten et al16	Describe misunderstandings between patients and FPs		Qualitative study using 5 data sources	14 categories of misunderstandings were identified
FP, family physician.

 

Outcome measures

Bhasale and colleagues¹⁰ and Fischer and coworkers¹³ collected patient outcome data; they specifically examined incidents that had “harmed” patients or had “potential for harm.” Ely and associates¹² also studied incidents causing patients harm by investigating possible causes of these incidents. Dovey and colleagues¹¹ reported physician-observed errors regardless of whether they were associated with an adverse event. Britten and coworkers¹⁶ analyzed misunderstandings between patients and physicians that had adverse consequences for taking medicines. Gandhi and associates¹⁴ described communication between primary care physicians and specialists. Holden and colleagues¹⁵ investigated deaths in general practices. All these studies attempted some categorization of medical errors. Bhasale and associates¹⁰ and Fischer and colleagues¹³ defined 4 incident cate gories and then assessed preventability. Dovey and coworkers¹¹ and Ely and associates¹² placed medical errors into categories, and Bhasale and colleagues¹⁰ listed a number of contributing factors. Britten and coworkers¹⁶ and Gandhi and associates¹⁴ categorized clinician communication problems. Holden and colleagues¹⁵ classified clinician actions that led to preventable deaths.

Due to the multiple methods used in the 7 studies and the descriptive nature of the studies, a standard assessment of quality and quantitative synthesis of data were not possible. Six studies used practicing community-based primary care physicians as their main study group. The study by Gandhi and coworkers, of communication between primary care physicians and specialists,¹⁴ was performed in an academic institution.

Classification system

We derived the following classification system (outlined in Tables 2 and 3) from the errors and preventable adverse events reported in these 7 studies.^10-16Table 2 defines the three main categories of preventable adverse events related by primary care physicians: diagnosis, treatment, and preventive services. These offer descriptors of what went wrong in the care of the patient but not of the level of harm. For example, a patient who was prescribed and took an incorrect drug has experienced a preventable adverse event. As a consequence, that patient may suffer no ill effects (a near miss), may die from anaphylaxis, or may experience some intermediate outcome (such as a rash).

Table 3 outlines “process errors” that clarify why something went wrong. For example, Why was the patient prescribed an incorrect drug? The answer may lie with a clinician factor (the doctor took an inadequate history), a communication factor (not dealing with a language or cultural barrier), an administrative factor (the medical chart was missing), or a blunt end factor (Medicare regulations). Often, multiple factors may be involved.

TABLE 2
Classification of preventable adverse events in primary care

Diagnosis
Related to symptoms
Misdiagnosis
Missed diagnosis
Delayed diagnosis
Related to prevention
Misdiagnosis
Missed diagnosis
Delayed diagnosis
Treatment
Drug
Incorrect drug
Incorrect dose
Delayed administration
Omitted administration
Non-drug
Inappropriate
Delayed
Omitted
Procedural complication
Preventive services
Inappropriate
Delayed
Omitted
Procedural complication

TABLE 3
Classification of process errors in primary care

Clinician factors
Clinical judgment
Procedural skills error
Communication factors
Clinician–patient
Clinician–clinician or health care system personnel
Administration factors
Clinician
Pharmacy
Ancillary providers (physical therapy, occupational therapy, etc)
Office setting
Blunt end factors
Personal and family issues of clinicians and staff
Insurance company regulations
Government regulations
Funding and employers
Physical size and location of practice
General health care system

Discussion

The results of this literature synthesis are important for 3 main reasons. First, they offer a summary of the current state of published research. Second, by synthesizing the results of this small body of literature, we were able to develop a working classification system of preventable adverse events (what went wrong) and process errors (why did it go wrong). Third, this classification may clarify the relations between patient safety, process errors, and preventable adverse events in primary care.

Other published classification systems of medical errors and preventable adverse events range from sparse (3 categories with 19 root causes)¹⁷ to dense (80 categories with more than 12,000 branching trees).¹⁸They generally derive from studies of safety in non-medical industries¹⁷ or from studies emphasizing hospital care.^2,18 In a recent review of the medical literature, Wilson and Sheikh noted the lack of a typology of medical errors in primary care and reasoned that the key safety issues in primary care are in the arenas of diagnosis, prescribing, communication, and organizational change.⁵ Their conclusions are congruent with ours, and our more structured classification system contains these arenas.

The classification in Table 3 was generated from research in primary care settings by using data from practicing family physicians and general practitioners. (A more complete version of Table 3 may be found at http://www.jfponline.com.) If the classification is valid and useful, it should assist clinicians and researchers in understanding how process errors and preventable adverse events happen during the practice of primary care. Models assist us in understanding these relations. Among previously proposed models are the “Swiss Cheese”¹⁹ and the “Toxic Cascades.”²⁰ The Swiss Cheese model postulates that barriers exist to prevent adverse events, but they are like slices of Swiss cheese with many holes (or errors) in them. Adverse events happen when the holes in many layers temporarily line up. The Toxic Cascades model conceptualizes 4 levels of threats to patient safety: trickles, which leave little trace of their existence; creeks, which have potential seriousness; rivers, which are the actual errors that harm patients; and torrents, which are errors that lead to a patient’s death or serious injury. From our classification, we can define some of the holes in the Swiss Cheese and name many trickles and creeks in primary care Toxic Cascades.

 

However, we found a striking gap in the literature of an absence of discussion of the contribution of patient factors to medical errors, despite a logic suggesting these are important issues.^21,22 A new model of patient safety dynamics should incorporate features of these models and add patient issues. Our proposed “Hourglass” model, derived from the classification system, incorporates 4 potential components of preventable adverse events in the primary care setting: 2 relating mainly to the primary health care system (process errors and patient safety factors) and 2 relating mainly to patients (patient risk factors for adverse events and patient-controlled patient safety factors; Figure). At the top of the hourglass, patient encounters enter like pieces of sand that flow through a health care system full of process errors that happen regularly. But, as in the Swiss Cheese model, there are barriers (patient safety factors) stopping these process errors from becoming preventable adverse events. Unfortunately, these barriers sometimes allow errors to slip through and a bad outcome results. Luckily, only a small number of patient encounters likely exits the primary health care system with a preventable adverse event, as demonstrated by the narrow part of the hourglass.

Outside the doctor’s office, factors in the patient’s milieu influence the probability of a preventable adverse event occurring. We postulate an experience analogous to that within the health care system. There are more factors increasing a patient’s likelihood of suffering a preventable adverse event,²³ but there are also patient-controlled factors serving as barriers against errors and their consequences. These are not well researched²⁴ but occur, for example, when a patient receives a blue pill from the pharmacy that had been pink in the past. The patient may prevent an adverse event by not taking the pill and double-checking with the clinician and pharmacist.

The order in which various process errors and safety factors interact with each other likely varies with each encounter and episode. Interactions within the classification suggest that, for any episode of disease or preventive care, the hourglass gets shaken and turned over numerous times as the health care system and patient factors interact with each other at multiple levels.

Future research needs

The literature review that led to our classification system and the proposed model of interaction have identified specific areas for future study. These include assessing patients’ perspectives, investigating prevalence and causality, and testing interventions designed to improve patient safety. The current medical literature based primarily on physician reports describes events that are meaningful to the physician half of the dyad between patient and physician. Patients’ opinions about what constitutes error and the role of patients as active participants in error and safety are unknown,²⁴ although preliminary studies are currently underway.²⁵

No published studies to date have explored the prevalence of preventable adverse events and errors in primary care. Physician self-report biases reporting toward remembered events and errors. In addition, medical error studies to date have not directly studied causal links between errors and adverse events.^26,27 Observational and epidemiologic studies incorporating multiple methods may be necessary to ascertain and compare all components of the medical error equation: the amount of harm done, the preventable adverse events and near misses, the process errors, and the error-free functioning of the health care system. Although observational studies have assessed adverse events in a hospital setting²⁸ and described primary care practices,²⁹ they have not been used to assess preventable adverse events in the primary care setting.

This literature review and synthesis may have missed some studies that merited inclusion. Only English-language studies were included. Studies pertaining to specific diseases, diagnoses, or treatments or from non-primary care settings may have shed light on the interaction of errors, adverse events, and harm but could not have helped in defining a classification system for primary care errors. The small number of studies available and their small sample sizes also limit the depth and breadth of derived classification components.

Decreasing medical errors and increasing patient safety are important parts of quality health care.³⁰ Currently, the research agenda aiming to identify effective error reduction strategies appears to be based more on ease of study subject or accessibility of patients than on the severity or importance of the problem.³¹ By categorizing process errors and preventable adverse events and studying their relations more thoroughly and by adding the patient’s perspective, interventions can be designed that address the most common and the most serious of preventable adverse events in primary care.

Every primary care clinician in the United States knows the frustration of lost charts, misplaced reports, and messages from patients that should have been answered yesterday. These are some of the common frustrations and failures in day-to-day clinical practice. Many clinicians also know the guilt, shame, and self-doubt that occur when patients suffer a serious complication or die due to a mistake made by the clinician, health care team, or health care system. Between the common frustrations of practice and the rare patient death due to an error lies a large chasm, a rarely explored territory of relationships, causes and effects, and mitigating factors. Looking backward from a catastrophic patient outcome rarely goes beyond blaming the immediate person “at fault.” ¹ Looking forward from common charting errors rarely goes beyond a conclusion to be “more careful.”

Hospital-based research has categorized preventable adverse outcomes and some process errors associated with them,^2-4 but this has not been done in primary care.⁵ There are difficulties in studying errors in the primary care setting: care takes place in many locations; involves multiple visits; is provided in person, by phone, by mail, and even by computer; and involves interactions with many health care workers. However, it is important to study errors in primary care⁶ because it is the location of most health care visits in the United States.⁷

A classification or taxonomy of errors and preventable adverse events is an important first step in improving patient care. Prevalence and epidemiology studies, clinical and system interventions, and even individual practice group databases of errors and adverse events⁸ can more easily be developed if there is a beginning classification system. Just as clinicians use a differential diagnostic list for analyzing symptoms or a list of risk factors for assessing disease, so, too, can clinicians use a classification and listing of process errors and preventable adverse events to “diagnose” and “prevent” patient harm from medical care. Many taxonomies of medical error do exist and have been used in hospital accreditation or malpractice contexts for some time.⁹ These taxonomies have not been generally available for purposes other than their intended use, ie, to help their developers understand the data they were dealing with, and because these data do not originate from primary care practice, it remains unknown how well the taxonomies might meet the needs of family physicians and other primary care researchers.

 

The purpose of this study was to use published data from original research to understand and classify process errors and preventable adverse events associated with primary medical care. Through a systematic review and synthesis of the medical literature, we developed a classification of medical errors relevant to primary care.

Methods

To identify eligible published English-language original research articles, we searched MEDLINE and the Cochrane Library from 1965 through March 2001 with the MESH search term medical errors, modified by adding family practice, primary health care, physicians/family, or ambulatory care to the primary term. Published bibliographies from the National Patient Safety Foundation (NPSF) and the Institute for Healthcare Improvement (IHI) were also reviewed. The Web sites of the American Academy of Family Physicians, the American College of Physicians–American Society of Internal Medicine, the Institute of Medicine, the NPSF, and the IHI were also reviewed for unpublished reports, presentations, and leads to other sites, journals, or investigators with relevant work. Additional papers were identified from the references of the papers reviewed, from seminal papers in the field, and from discussion with others working in the field of patient safety or quality improvement in primary care.

We reviewed titles of 379 articles identified by electronic searches for inclusion. We excluded papers if they related to comparisons of different approaches to diagnosis or treatment of specific diseases, the evaluation of teaching or research tools, or exclusively to hospitalized patients. If there was uncertainty as to the appropriateness of an article, we read the abstract. We reviewed complete papers if they appeared from the title and abstract to report original research involving a broad assessment of medical errors and preventable adverse events in primary care. Data relating to topic, study quality, and research results were abstracted from identified papers. Both authors performed independent MEDLINE searches and reviewed citations in the papers. To broaden the search for potential studies, one author searched Web sites and NPSF and IHI bibliographies. Both authors agreed on the inclusion of the chosen studies, appraised them independently, and abstracted key classification components. One author (N.C.E.) initially prepared the classification system presented here; it was then reviewed by both authors and revised after their discussions.

Results

Four original research studies directly studied and described medical errors and preventable adverse events in primary care.^10-13 Three other studies peripherally addressed primary care medical errors as part of an investigation with another central focus^14-16 (Table 1).

TABLE 1
Primary care studies describing medical error

Study	Research purpose	Definition of error	Method	Pertinent results
Primary care studies directly describing medical error
Bhasale et al10	Describe incidents occurring in general practice	An unintended event, no matter how seemingly trivial or commonplace, that could have harmed or did harm a patient	Self-report by 324 Australian sentinel research network FPs using reporting cards	805 incidents reported, 76% preventable; categories were drug management, non-drug management, diagnosis, and equipment; causes included communication, actions of others, and clinical judgment errors
Ely et al12	Describe the causes to which family physicians attribute errors	Act or omission for which the physician felt responsible and which had serious consequences for the patient	30-min interviews with 53 randomly chosen Iowa FPs	53 errors reported: delayed diagnoses, surgical and medical treatment mishaps; causes included physical stressors, process of care factors, patient related factors, and physician characteristics
Dovey et al11	Describe medical errors reported by FPs	Something in one’s practice that should not have happened, that was not anticipated, and that makes one say, “I don’t want it to happen again”	Self-report by 42 American research network FPs using electronic and reply card reporting	330 reported errors, 83% from health care system and 13% from knowledge and skills; subcategories were office administration, investigations, treatments, communication, execution of clinical tasks, misdiagnosis, and wrong treatment decision
Fischer et al13	Describe the prevalence of adverse events in a risk management database	Incidents resulting in, or having the potential for, physical, emotional, or financial liability for the patient	Review of incident reports entered by 8 primary care clinics into risk management database	Prevalence of adverse events was 3.7/100,000 clinic visits, 83% were preventable; categories included diagnostic, treatment, and preventive and other errors
Primary care studies peripherally describing medical error
Holden et al15	Determine patterns of death and potential preventive factors		Formal review of all patient deaths in a group of general practices	5.1% of deaths due to preventable FP factors; 2 main categories were delay of diagnosis and treatment and lack of prevention with aspirin therapy
Gandhi et al14communication	Evaluate primary care and specialist inter physician		Surveys in academic medical center	Main issues for doctors were lack of timeliness and inadequate content
Britten et al16	Describe misunderstandings between patients and FPs		Qualitative study using 5 data sources	14 categories of misunderstandings were identified
FP, family physician.

 

Outcome measures

Bhasale and colleagues¹⁰ and Fischer and coworkers¹³ collected patient outcome data; they specifically examined incidents that had “harmed” patients or had “potential for harm.” Ely and associates¹² also studied incidents causing patients harm by investigating possible causes of these incidents. Dovey and colleagues¹¹ reported physician-observed errors regardless of whether they were associated with an adverse event. Britten and coworkers¹⁶ analyzed misunderstandings between patients and physicians that had adverse consequences for taking medicines. Gandhi and associates¹⁴ described communication between primary care physicians and specialists. Holden and colleagues¹⁵ investigated deaths in general practices. All these studies attempted some categorization of medical errors. Bhasale and associates¹⁰ and Fischer and colleagues¹³ defined 4 incident cate gories and then assessed preventability. Dovey and coworkers¹¹ and Ely and associates¹² placed medical errors into categories, and Bhasale and colleagues¹⁰ listed a number of contributing factors. Britten and coworkers¹⁶ and Gandhi and associates¹⁴ categorized clinician communication problems. Holden and colleagues¹⁵ classified clinician actions that led to preventable deaths.

Due to the multiple methods used in the 7 studies and the descriptive nature of the studies, a standard assessment of quality and quantitative synthesis of data were not possible. Six studies used practicing community-based primary care physicians as their main study group. The study by Gandhi and coworkers, of communication between primary care physicians and specialists,¹⁴ was performed in an academic institution.

Classification system

We derived the following classification system (outlined in Tables 2 and 3) from the errors and preventable adverse events reported in these 7 studies.^10-16Table 2 defines the three main categories of preventable adverse events related by primary care physicians: diagnosis, treatment, and preventive services. These offer descriptors of what went wrong in the care of the patient but not of the level of harm. For example, a patient who was prescribed and took an incorrect drug has experienced a preventable adverse event. As a consequence, that patient may suffer no ill effects (a near miss), may die from anaphylaxis, or may experience some intermediate outcome (such as a rash).

Table 3 outlines “process errors” that clarify why something went wrong. For example, Why was the patient prescribed an incorrect drug? The answer may lie with a clinician factor (the doctor took an inadequate history), a communication factor (not dealing with a language or cultural barrier), an administrative factor (the medical chart was missing), or a blunt end factor (Medicare regulations). Often, multiple factors may be involved.

TABLE 2
Classification of preventable adverse events in primary care

Diagnosis
Related to symptoms
Misdiagnosis
Missed diagnosis
Delayed diagnosis
Related to prevention
Misdiagnosis
Missed diagnosis
Delayed diagnosis
Treatment
Drug
Incorrect drug
Incorrect dose
Delayed administration
Omitted administration
Non-drug
Inappropriate
Delayed
Omitted
Procedural complication
Preventive services
Inappropriate
Delayed
Omitted
Procedural complication

TABLE 3
Classification of process errors in primary care

Clinician factors
Clinical judgment
Procedural skills error
Communication factors
Clinician–patient
Clinician–clinician or health care system personnel
Administration factors
Clinician
Pharmacy
Ancillary providers (physical therapy, occupational therapy, etc)
Office setting
Blunt end factors
Personal and family issues of clinicians and staff
Insurance company regulations
Government regulations
Funding and employers
Physical size and location of practice
General health care system

Discussion

The results of this literature synthesis are important for 3 main reasons. First, they offer a summary of the current state of published research. Second, by synthesizing the results of this small body of literature, we were able to develop a working classification system of preventable adverse events (what went wrong) and process errors (why did it go wrong). Third, this classification may clarify the relations between patient safety, process errors, and preventable adverse events in primary care.

Other published classification systems of medical errors and preventable adverse events range from sparse (3 categories with 19 root causes)¹⁷ to dense (80 categories with more than 12,000 branching trees).¹⁸They generally derive from studies of safety in non-medical industries¹⁷ or from studies emphasizing hospital care.^2,18 In a recent review of the medical literature, Wilson and Sheikh noted the lack of a typology of medical errors in primary care and reasoned that the key safety issues in primary care are in the arenas of diagnosis, prescribing, communication, and organizational change.⁵ Their conclusions are congruent with ours, and our more structured classification system contains these arenas.

The classification in Table 3 was generated from research in primary care settings by using data from practicing family physicians and general practitioners. (A more complete version of Table 3 may be found at http://www.jfponline.com.) If the classification is valid and useful, it should assist clinicians and researchers in understanding how process errors and preventable adverse events happen during the practice of primary care. Models assist us in understanding these relations. Among previously proposed models are the “Swiss Cheese”¹⁹ and the “Toxic Cascades.”²⁰ The Swiss Cheese model postulates that barriers exist to prevent adverse events, but they are like slices of Swiss cheese with many holes (or errors) in them. Adverse events happen when the holes in many layers temporarily line up. The Toxic Cascades model conceptualizes 4 levels of threats to patient safety: trickles, which leave little trace of their existence; creeks, which have potential seriousness; rivers, which are the actual errors that harm patients; and torrents, which are errors that lead to a patient’s death or serious injury. From our classification, we can define some of the holes in the Swiss Cheese and name many trickles and creeks in primary care Toxic Cascades.

 

However, we found a striking gap in the literature of an absence of discussion of the contribution of patient factors to medical errors, despite a logic suggesting these are important issues.^21,22 A new model of patient safety dynamics should incorporate features of these models and add patient issues. Our proposed “Hourglass” model, derived from the classification system, incorporates 4 potential components of preventable adverse events in the primary care setting: 2 relating mainly to the primary health care system (process errors and patient safety factors) and 2 relating mainly to patients (patient risk factors for adverse events and patient-controlled patient safety factors; Figure). At the top of the hourglass, patient encounters enter like pieces of sand that flow through a health care system full of process errors that happen regularly. But, as in the Swiss Cheese model, there are barriers (patient safety factors) stopping these process errors from becoming preventable adverse events. Unfortunately, these barriers sometimes allow errors to slip through and a bad outcome results. Luckily, only a small number of patient encounters likely exits the primary health care system with a preventable adverse event, as demonstrated by the narrow part of the hourglass.

Outside the doctor’s office, factors in the patient’s milieu influence the probability of a preventable adverse event occurring. We postulate an experience analogous to that within the health care system. There are more factors increasing a patient’s likelihood of suffering a preventable adverse event,²³ but there are also patient-controlled factors serving as barriers against errors and their consequences. These are not well researched²⁴ but occur, for example, when a patient receives a blue pill from the pharmacy that had been pink in the past. The patient may prevent an adverse event by not taking the pill and double-checking with the clinician and pharmacist.

The order in which various process errors and safety factors interact with each other likely varies with each encounter and episode. Interactions within the classification suggest that, for any episode of disease or preventive care, the hourglass gets shaken and turned over numerous times as the health care system and patient factors interact with each other at multiple levels.

Future research needs

The literature review that led to our classification system and the proposed model of interaction have identified specific areas for future study. These include assessing patients’ perspectives, investigating prevalence and causality, and testing interventions designed to improve patient safety. The current medical literature based primarily on physician reports describes events that are meaningful to the physician half of the dyad between patient and physician. Patients’ opinions about what constitutes error and the role of patients as active participants in error and safety are unknown,²⁴ although preliminary studies are currently underway.²⁵

No published studies to date have explored the prevalence of preventable adverse events and errors in primary care. Physician self-report biases reporting toward remembered events and errors. In addition, medical error studies to date have not directly studied causal links between errors and adverse events.^26,27 Observational and epidemiologic studies incorporating multiple methods may be necessary to ascertain and compare all components of the medical error equation: the amount of harm done, the preventable adverse events and near misses, the process errors, and the error-free functioning of the health care system. Although observational studies have assessed adverse events in a hospital setting²⁸ and described primary care practices,²⁹ they have not been used to assess preventable adverse events in the primary care setting.

This literature review and synthesis may have missed some studies that merited inclusion. Only English-language studies were included. Studies pertaining to specific diseases, diagnoses, or treatments or from non-primary care settings may have shed light on the interaction of errors, adverse events, and harm but could not have helped in defining a classification system for primary care errors. The small number of studies available and their small sample sizes also limit the depth and breadth of derived classification components.

Decreasing medical errors and increasing patient safety are important parts of quality health care.³⁰ Currently, the research agenda aiming to identify effective error reduction strategies appears to be based more on ease of study subject or accessibility of patients than on the severity or importance of the problem.³¹ By categorizing process errors and preventable adverse events and studying their relations more thoroughly and by adding the patient’s perspective, interventions can be designed that address the most common and the most serious of preventable adverse events in primary care.

Every primary care clinician in the United States knows the frustration of lost charts, misplaced reports, and messages from patients that should have been answered yesterday. These are some of the common frustrations and failures in day-to-day clinical practice. Many clinicians also know the guilt, shame, and self-doubt that occur when patients suffer a serious complication or die due to a mistake made by the clinician, health care team, or health care system. Between the common frustrations of practice and the rare patient death due to an error lies a large chasm, a rarely explored territory of relationships, causes and effects, and mitigating factors. Looking backward from a catastrophic patient outcome rarely goes beyond blaming the immediate person “at fault.” ¹ Looking forward from common charting errors rarely goes beyond a conclusion to be “more careful.”

Hospital-based research has categorized preventable adverse outcomes and some process errors associated with them,^2-4 but this has not been done in primary care.⁵ There are difficulties in studying errors in the primary care setting: care takes place in many locations; involves multiple visits; is provided in person, by phone, by mail, and even by computer; and involves interactions with many health care workers. However, it is important to study errors in primary care⁶ because it is the location of most health care visits in the United States.⁷

A classification or taxonomy of errors and preventable adverse events is an important first step in improving patient care. Prevalence and epidemiology studies, clinical and system interventions, and even individual practice group databases of errors and adverse events⁸ can more easily be developed if there is a beginning classification system. Just as clinicians use a differential diagnostic list for analyzing symptoms or a list of risk factors for assessing disease, so, too, can clinicians use a classification and listing of process errors and preventable adverse events to “diagnose” and “prevent” patient harm from medical care. Many taxonomies of medical error do exist and have been used in hospital accreditation or malpractice contexts for some time.⁹ These taxonomies have not been generally available for purposes other than their intended use, ie, to help their developers understand the data they were dealing with, and because these data do not originate from primary care practice, it remains unknown how well the taxonomies might meet the needs of family physicians and other primary care researchers.

 

The purpose of this study was to use published data from original research to understand and classify process errors and preventable adverse events associated with primary medical care. Through a systematic review and synthesis of the medical literature, we developed a classification of medical errors relevant to primary care.

Methods

To identify eligible published English-language original research articles, we searched MEDLINE and the Cochrane Library from 1965 through March 2001 with the MESH search term medical errors, modified by adding family practice, primary health care, physicians/family, or ambulatory care to the primary term. Published bibliographies from the National Patient Safety Foundation (NPSF) and the Institute for Healthcare Improvement (IHI) were also reviewed. The Web sites of the American Academy of Family Physicians, the American College of Physicians–American Society of Internal Medicine, the Institute of Medicine, the NPSF, and the IHI were also reviewed for unpublished reports, presentations, and leads to other sites, journals, or investigators with relevant work. Additional papers were identified from the references of the papers reviewed, from seminal papers in the field, and from discussion with others working in the field of patient safety or quality improvement in primary care.

We reviewed titles of 379 articles identified by electronic searches for inclusion. We excluded papers if they related to comparisons of different approaches to diagnosis or treatment of specific diseases, the evaluation of teaching or research tools, or exclusively to hospitalized patients. If there was uncertainty as to the appropriateness of an article, we read the abstract. We reviewed complete papers if they appeared from the title and abstract to report original research involving a broad assessment of medical errors and preventable adverse events in primary care. Data relating to topic, study quality, and research results were abstracted from identified papers. Both authors performed independent MEDLINE searches and reviewed citations in the papers. To broaden the search for potential studies, one author searched Web sites and NPSF and IHI bibliographies. Both authors agreed on the inclusion of the chosen studies, appraised them independently, and abstracted key classification components. One author (N.C.E.) initially prepared the classification system presented here; it was then reviewed by both authors and revised after their discussions.

Results

Four original research studies directly studied and described medical errors and preventable adverse events in primary care.^10-13 Three other studies peripherally addressed primary care medical errors as part of an investigation with another central focus^14-16 (Table 1).

TABLE 1
Primary care studies describing medical error

Study	Research purpose	Definition of error	Method	Pertinent results
Primary care studies directly describing medical error
Bhasale et al10	Describe incidents occurring in general practice	An unintended event, no matter how seemingly trivial or commonplace, that could have harmed or did harm a patient	Self-report by 324 Australian sentinel research network FPs using reporting cards	805 incidents reported, 76% preventable; categories were drug management, non-drug management, diagnosis, and equipment; causes included communication, actions of others, and clinical judgment errors
Ely et al12	Describe the causes to which family physicians attribute errors	Act or omission for which the physician felt responsible and which had serious consequences for the patient	30-min interviews with 53 randomly chosen Iowa FPs	53 errors reported: delayed diagnoses, surgical and medical treatment mishaps; causes included physical stressors, process of care factors, patient related factors, and physician characteristics
Dovey et al11	Describe medical errors reported by FPs	Something in one’s practice that should not have happened, that was not anticipated, and that makes one say, “I don’t want it to happen again”	Self-report by 42 American research network FPs using electronic and reply card reporting	330 reported errors, 83% from health care system and 13% from knowledge and skills; subcategories were office administration, investigations, treatments, communication, execution of clinical tasks, misdiagnosis, and wrong treatment decision
Fischer et al13	Describe the prevalence of adverse events in a risk management database	Incidents resulting in, or having the potential for, physical, emotional, or financial liability for the patient	Review of incident reports entered by 8 primary care clinics into risk management database	Prevalence of adverse events was 3.7/100,000 clinic visits, 83% were preventable; categories included diagnostic, treatment, and preventive and other errors
Primary care studies peripherally describing medical error
Holden et al15	Determine patterns of death and potential preventive factors		Formal review of all patient deaths in a group of general practices	5.1% of deaths due to preventable FP factors; 2 main categories were delay of diagnosis and treatment and lack of prevention with aspirin therapy
Gandhi et al14communication	Evaluate primary care and specialist inter physician		Surveys in academic medical center	Main issues for doctors were lack of timeliness and inadequate content
Britten et al16	Describe misunderstandings between patients and FPs		Qualitative study using 5 data sources	14 categories of misunderstandings were identified
FP, family physician.

 

Outcome measures

Bhasale and colleagues¹⁰ and Fischer and coworkers¹³ collected patient outcome data; they specifically examined incidents that had “harmed” patients or had “potential for harm.” Ely and associates¹² also studied incidents causing patients harm by investigating possible causes of these incidents. Dovey and colleagues¹¹ reported physician-observed errors regardless of whether they were associated with an adverse event. Britten and coworkers¹⁶ analyzed misunderstandings between patients and physicians that had adverse consequences for taking medicines. Gandhi and associates¹⁴ described communication between primary care physicians and specialists. Holden and colleagues¹⁵ investigated deaths in general practices. All these studies attempted some categorization of medical errors. Bhasale and associates¹⁰ and Fischer and colleagues¹³ defined 4 incident cate gories and then assessed preventability. Dovey and coworkers¹¹ and Ely and associates¹² placed medical errors into categories, and Bhasale and colleagues¹⁰ listed a number of contributing factors. Britten and coworkers¹⁶ and Gandhi and associates¹⁴ categorized clinician communication problems. Holden and colleagues¹⁵ classified clinician actions that led to preventable deaths.

Due to the multiple methods used in the 7 studies and the descriptive nature of the studies, a standard assessment of quality and quantitative synthesis of data were not possible. Six studies used practicing community-based primary care physicians as their main study group. The study by Gandhi and coworkers, of communication between primary care physicians and specialists,¹⁴ was performed in an academic institution.

Classification system

We derived the following classification system (outlined in Tables 2 and 3) from the errors and preventable adverse events reported in these 7 studies.^10-16Table 2 defines the three main categories of preventable adverse events related by primary care physicians: diagnosis, treatment, and preventive services. These offer descriptors of what went wrong in the care of the patient but not of the level of harm. For example, a patient who was prescribed and took an incorrect drug has experienced a preventable adverse event. As a consequence, that patient may suffer no ill effects (a near miss), may die from anaphylaxis, or may experience some intermediate outcome (such as a rash).

Table 3 outlines “process errors” that clarify why something went wrong. For example, Why was the patient prescribed an incorrect drug? The answer may lie with a clinician factor (the doctor took an inadequate history), a communication factor (not dealing with a language or cultural barrier), an administrative factor (the medical chart was missing), or a blunt end factor (Medicare regulations). Often, multiple factors may be involved.

TABLE 2
Classification of preventable adverse events in primary care

Diagnosis
Related to symptoms
Misdiagnosis
Missed diagnosis
Delayed diagnosis
Related to prevention
Misdiagnosis
Missed diagnosis
Delayed diagnosis
Treatment
Drug
Incorrect drug
Incorrect dose
Delayed administration
Omitted administration
Non-drug
Inappropriate
Delayed
Omitted
Procedural complication
Preventive services
Inappropriate
Delayed
Omitted
Procedural complication

TABLE 3
Classification of process errors in primary care

Clinician factors
Clinical judgment
Procedural skills error
Communication factors
Clinician–patient
Clinician–clinician or health care system personnel
Administration factors
Clinician
Pharmacy
Ancillary providers (physical therapy, occupational therapy, etc)
Office setting
Blunt end factors
Personal and family issues of clinicians and staff
Insurance company regulations
Government regulations
Funding and employers
Physical size and location of practice
General health care system

Discussion

The results of this literature synthesis are important for 3 main reasons. First, they offer a summary of the current state of published research. Second, by synthesizing the results of this small body of literature, we were able to develop a working classification system of preventable adverse events (what went wrong) and process errors (why did it go wrong). Third, this classification may clarify the relations between patient safety, process errors, and preventable adverse events in primary care.

Other published classification systems of medical errors and preventable adverse events range from sparse (3 categories with 19 root causes)¹⁷ to dense (80 categories with more than 12,000 branching trees).¹⁸They generally derive from studies of safety in non-medical industries¹⁷ or from studies emphasizing hospital care.^2,18 In a recent review of the medical literature, Wilson and Sheikh noted the lack of a typology of medical errors in primary care and reasoned that the key safety issues in primary care are in the arenas of diagnosis, prescribing, communication, and organizational change.⁵ Their conclusions are congruent with ours, and our more structured classification system contains these arenas.

The classification in Table 3 was generated from research in primary care settings by using data from practicing family physicians and general practitioners. (A more complete version of Table 3 may be found at http://www.jfponline.com.) If the classification is valid and useful, it should assist clinicians and researchers in understanding how process errors and preventable adverse events happen during the practice of primary care. Models assist us in understanding these relations. Among previously proposed models are the “Swiss Cheese”¹⁹ and the “Toxic Cascades.”²⁰ The Swiss Cheese model postulates that barriers exist to prevent adverse events, but they are like slices of Swiss cheese with many holes (or errors) in them. Adverse events happen when the holes in many layers temporarily line up. The Toxic Cascades model conceptualizes 4 levels of threats to patient safety: trickles, which leave little trace of their existence; creeks, which have potential seriousness; rivers, which are the actual errors that harm patients; and torrents, which are errors that lead to a patient’s death or serious injury. From our classification, we can define some of the holes in the Swiss Cheese and name many trickles and creeks in primary care Toxic Cascades.

 

However, we found a striking gap in the literature of an absence of discussion of the contribution of patient factors to medical errors, despite a logic suggesting these are important issues.^21,22 A new model of patient safety dynamics should incorporate features of these models and add patient issues. Our proposed “Hourglass” model, derived from the classification system, incorporates 4 potential components of preventable adverse events in the primary care setting: 2 relating mainly to the primary health care system (process errors and patient safety factors) and 2 relating mainly to patients (patient risk factors for adverse events and patient-controlled patient safety factors; Figure). At the top of the hourglass, patient encounters enter like pieces of sand that flow through a health care system full of process errors that happen regularly. But, as in the Swiss Cheese model, there are barriers (patient safety factors) stopping these process errors from becoming preventable adverse events. Unfortunately, these barriers sometimes allow errors to slip through and a bad outcome results. Luckily, only a small number of patient encounters likely exits the primary health care system with a preventable adverse event, as demonstrated by the narrow part of the hourglass.

Outside the doctor’s office, factors in the patient’s milieu influence the probability of a preventable adverse event occurring. We postulate an experience analogous to that within the health care system. There are more factors increasing a patient’s likelihood of suffering a preventable adverse event,²³ but there are also patient-controlled factors serving as barriers against errors and their consequences. These are not well researched²⁴ but occur, for example, when a patient receives a blue pill from the pharmacy that had been pink in the past. The patient may prevent an adverse event by not taking the pill and double-checking with the clinician and pharmacist.

The order in which various process errors and safety factors interact with each other likely varies with each encounter and episode. Interactions within the classification suggest that, for any episode of disease or preventive care, the hourglass gets shaken and turned over numerous times as the health care system and patient factors interact with each other at multiple levels.

Future research needs

The literature review that led to our classification system and the proposed model of interaction have identified specific areas for future study. These include assessing patients’ perspectives, investigating prevalence and causality, and testing interventions designed to improve patient safety. The current medical literature based primarily on physician reports describes events that are meaningful to the physician half of the dyad between patient and physician. Patients’ opinions about what constitutes error and the role of patients as active participants in error and safety are unknown,²⁴ although preliminary studies are currently underway.²⁵

No published studies to date have explored the prevalence of preventable adverse events and errors in primary care. Physician self-report biases reporting toward remembered events and errors. In addition, medical error studies to date have not directly studied causal links between errors and adverse events.^26,27 Observational and epidemiologic studies incorporating multiple methods may be necessary to ascertain and compare all components of the medical error equation: the amount of harm done, the preventable adverse events and near misses, the process errors, and the error-free functioning of the health care system. Although observational studies have assessed adverse events in a hospital setting²⁸ and described primary care practices,²⁹ they have not been used to assess preventable adverse events in the primary care setting.

This literature review and synthesis may have missed some studies that merited inclusion. Only English-language studies were included. Studies pertaining to specific diseases, diagnoses, or treatments or from non-primary care settings may have shed light on the interaction of errors, adverse events, and harm but could not have helped in defining a classification system for primary care errors. The small number of studies available and their small sample sizes also limit the depth and breadth of derived classification components.

Decreasing medical errors and increasing patient safety are important parts of quality health care.³⁰ Currently, the research agenda aiming to identify effective error reduction strategies appears to be based more on ease of study subject or accessibility of patients than on the severity or importance of the problem.³¹ By categorizing process errors and preventable adverse events and studying their relations more thoroughly and by adding the patient’s perspective, interventions can be designed that address the most common and the most serious of preventable adverse events in primary care.

Clinicians are inundated with clinical practice guidelines. Many guidelines address the same problems, often with conflicting recommendations. The disagreement concerning recommendations often hinges on how aggressive clinicians are in promoting interventions, eg, screening for prostate cancer or prescribe antibiotics for the treatment of sore throat. There is also variation with regard to the methods that guideline developers use to make recommendations. The traditional “GOBSAT”-technique (“Good Old Boys Sat At Table”) has been criticized.¹ Several groups have proposed more systematic approaches for guideline development,^2-6 and some databases, such as the Guideline Advisory Committee’s Recommended Clinical Practice Guidelines, include only those guidelines that have been assessed for the rigor of the development process.⁷

There are strong logical arguments for developing guidelines systematically, eg, to ensure that they are based on current best evidence, to protect against bias, and to make the process transparent and open to criticism. However, guideline developers frequently do not adhere to such methods.^8-11 We explored the possible association between the methods used in guideline development and the recommendations given in those guidelines. We used guidelines for hypertension and hyperlipidemia in our study. Our hypothesis was that less rigorous methods would, on average, be associated with more aggressive recommendations in these guidelines.

Methods

Inclusion criteria

We defined clinical guidelines as recommendations intended to assist health professionals and patients in making decisions for specific clinical circumstances. To be included a guideline had to address at least 1 of the following issues: threshold for drug treatment of essential hypertension in primary prevention, threshold for drug treatment of hyperlipidemia in primary prevention, or identification of the target population for cholesterol screening. Guidelines were excluded if they did not clearly identify the panel responsible for developing the guideline, identify a sponsoring organization, or include a reference list. We excluded textbooks, editorials, and commentaries. Review articles that were prepared and published as background documents were included with the relevant clinical practice guidelines. We included guidelines published after 1992. When multiple versions were available from the same organization, we used the most recent version.

Search strategy

We searched MEDLINE from 1992 through February 2000 by using hypertension, blood pressure, hyperlipidemia, or cholesterol as the key term, limited to practice guidelines as publication type. In addition, we searched databases of guidelines maintained by several groups around the world. One author (A.F.) reviewed all the citations and reference lists of relevant guidelines, retrieved potentially relevant guidelines, and selected guidelines for inclusion.

Guideline development methods

We used 8 criteria to rate the methodologic quality of the guidelines (Table 1). The criteria were adapted from a guideline appraisal instrument that is being developed and tested by a group of European researchers (the Agree/Biomed collaboration).¹² It is based on a British Appraisal Instrument for Clinical Guidelines ¹³ that has been tested for its validity and reliability and has been characterized as “the most well developed to date” in a recent study.¹⁴

Two authors (A.F. and J.W.W.) evaluated each guideline independently. For analytical purposes, all criteria were dichotomized (Table 1). Because fewer than 50% of the guidelines reported sufficient information to determine stakeholder involvement, we supplemented information on authors through an Internet search.

TABLE 1

Criteria used to appraise guideline-development methods

Criterion	Standard for fulfillment
Main outcomes identified
Is there an explicit statement of the main outcomes considered when developing the guideline?	Explicit statement of the main outcomes considered in developing the guidelines
Key stakeholders involved
Are the essential stakeholders involved in the development group?	Inclusion of all “essential” stakeholders (generalist physicians, specialists, and methodologists)*
Systematic search and selection
Has a systematic search for evidence been carried out and are criteria for inclusion and exclusion specified?	Search specifying all relevant databases or described “electronic databases” or
	Inclusion–exclusion criteria defined, at least briefly
Recommendations linked to evidence
Is there an explicit link between the evidence and the recommendations given?	Grading of strength of recommendations or level of evidence
Benefits and risks considered
Have the health benefits, side effects, and risks been considered?	Any quantitative or qualitative weighing of benefits and harms that is incorporated into formulating recommendations
Resources/costs
Has the impact on resources been considered?	Economic analysis or qualitative consideration of cost issues that is linked explicitly to the formulation of the recommendations
No industry influence
Is the guideline developed without funding or influence from the pharmaceutical industry?	No financial support from a pharmaceutical company and no involvement on the panel
Conflicts of interest stated
Is there an explicit statement of conflict of interest?	Statement of potential conflicts of interest for panel members
*Essential stakeholders are generalist physicians, specialists, and methodologists. Optional stakeholders are patients, policy makers/health administrators, nurses, pharmacists, economists, and other physicians.

 

Aggressiveness of recommendations

To grade the aggressiveness of the treatment recommendations, we evaluated the threshold for initiating pharmacologic treatment of hypertension (low = aggressive), threshold for initiating pharmacologic treatment of hyperlipidemia (low = aggressive), first-line antihypertensive drug (all drugs = aggressive), and the number of persons eligible for cholesterol screening (high = aggressive).

The thresholds for treatment of hypertension and hyperlipidemia were categorized with 4 clinical scenarios. By applying the scenarios we could determine the recommended thresholds for drug treatment among the various guidelines. We chose these specific scenarios to illustrate the existing variation in recommendations among the guidelines:

1. A 50-year-old man without a high-risk profile of cardiovascular disease.
   
2. A 50-year-old man with a high-risk profile of cardiovascular disease.
   
3. A 70-year-old man without a high-risk profile of cardiovascular disease.
   
4. A 70-year-old man with a high-risk profile of cardiovascular disease.
   

High-risk profile was defined separately for treatment of hypertension and treatment of hyperlipidemia. For hypertension, the high-risk scenario was a patient who smoked and was hyperlipidemic (total cholesterol > 310 mg/dL). For hyperlipidemia, the high-risk scenario was a patient who smoked and was hypertensive (blood pressure > 160/100 mm Hg). We assumed that lifestyle interventions had been attempted. For simplicity we used only the systolic value for blood pressure. We dichotomized the aggressiveness of thresholds for lipid-lowering treatment rather than calculating averages because many guidelines gave recommendations other than actual cholesterol values; eg, no treatment or familial hypercholesterolemia. Guidelines were considered nonaggressive if, for 2 or more of the scenarios, the threshold was set at 310 mg/dL or higher or was specified as familial hypercholesterolemia.

Guidelines recommending all common antihypertensive drugs were considered aggressive, and the ones suggesting more restrictive recommendations were considered nonaggressive. In a few guidelines this recommendation depended on the patient’s age. For simplicity we examined the recommendations for 50-year-olds. We graded aggressiveness of recommendations on cholesterol screening by estimating the proportion of the general adult population (in Norway) who would be candidates for screening or by case finding per year, if the guidelines were fully implemented.

Analysis

We qualitatively and quantitatively examined the relation between fulfillment of a methodologic criterion and the aggressiveness of recommendations. The power of our statistical analyses was limited by the available sample size. For hypertension, we averaged the treatment threshold for the 4 clinical scenarios within each guideline and compared the overall mean between guidelines meeting and not meeting the criterion. For the threshold to treat hyperlipidemia and for first-line therapy for hypertension, the degree of aggressiveness was dichotomized. We used the Fisher exact test to calculate P values for the association between the proportion of guidelines fulfilling a methodologic criterion and whether the recommendation was classified as aggressive. For cholesterol screening we found the mean yearly proportion of the adult population eligible for screening among guidelines fulfilling a methodologic criterion, and compared this with the mean for guidelines that did not fulfill the criterion. In addition to examining the association between methods and recommendations, we examined whether the level of stakeholder involvement or sponsorship by specialty societies was associated with fulfillment of the methodologic criteria. We included generalist physicians, specialists, and methodologists as “essential stakeholders” and patients, policy makers/health administrators, nurses, pharmacists, economists, and other physicians as “optional stakeholders.”

Results

We found 12 clinical guidelines for managing hypertension, 12 for hyperlipidemia, 5 for cholesterol screening, and 4 general guidelines for the prevention of coronary heart disease that met our inclusion criteria (references are available from the authors). Because each guideline was appraised according to the 8 methodologic criteria, we ended up with 264 appraisals (8 criteria applied to each of the 33 guidelines). There were 28 disagreements (11%), all of which were easily resolved by discussion. As expected, there was variation among the guidelines regarding fulfillment of methodologic criteria and the aggressiveness of recommendations.

Most guidelines did not meet the majority of the methodologic criteria (Table 2). Only 6 of the 33 guidelines met 5 or more of the 8 criteria. The threshold to start antihypertensive treatment varied in systolic blood pressure from 140 to 180 mm Hg for each of the 4 clinical scenarios we applied to the guidelines (Table 3). For 3 of the scenarios, the threshold to treat hyperlipidemia ranged from a total cholesterol value of 190 mg/dL to more than 310 mg/dL (Table 3). Fifteen guidelines gave recommendations for first-line therapy for hypertension. Three recommended thiazides only; 6 recommended thiazides and β-blockers; 1 recommended thiazides, β-blockers, and angiotensinconverting enzyme inhibitors; and 5 recommended all commonly used drugs. Recommendations for cholesterol screening ranged from no screening to testing the entire adult population every 2 to 5 years.

 

TABLE 2

Variability in fulfillment of methodologic criteria

Criterion	Fulfilled, n (%)	Not fulfilled, n (%)	No information, n (%)
Main outcomes identified	10 (30)	23 (70)	0 (0)
Key stakeholders involved	21 (64)	9 (27)	3 (9)
Systematic search and selection	7 (21)	26 (79)	0 (0)
Recommendations linked to evidence	10 (30)	23 (70)	0 (0)
Benefits and risks considered	21 (64)	12 (36)	0 (0)
Resources/costs	14 (42)	19 (58)	0 (0)
No industry influence	23 (70)	2 (6)	8 (24)
Conflicts of interest stated	4 (12)	29 (88)	0 (0)

TABLE 3

Variability in guideline recommendations*

	BP threshold (mm Hg) to treat hypertension
Age, clinical scenario	140	150	160	170	180
50 y, low risk	4 (25)	1 (6)	5 (31)	4 (25)	2 (13)
50 y, high risk	9 (56)	2 (13)	3 (19)	1 (6)	1 (6)
70 y, high risk	6 (38)	1 (6)	5 (31)	2 (13)	2 (13)
70 y, low risk	8 (50)	1 (6)	5 (31)	1 (6)	1 (6)
	Cholesterol threshold (mg/dL) to treat hyperlipidemia
		≤230	≤270	≥310 †
50 y, low risk	3 (19)	3 (19)	1 (6)	9 (56)
50 y, high risk	6 (38)	8 (50)	0	2 (13)
70 y, high risk	5 (31)	3 (19)	2 (13)	3 (19)
70 y, low risk	7 (44)	5 (31)	1 (6)	3 (19)
*Data are presented as number (%) of patients.
† Includes the recommendations “no treatment” and “familial hypercholesterolemia.”
BP, blood pressure.

Associations between recommendations and methodologic criteria

The threshold to treat hypertension did not seem to be associated with fulfillment of methodologic criteria. Differences in recommendations for first-line drugs for hypertension were not strongly associated with any of the criteria. Although not statistically significant, there was a trend for guidelines to recommend all commonly available drugs when methodologic criteria were not met (Table W1, available on the JFP web site: http://www.jfponline.com).

For all but 1 quality criteria (main outcomes identified), fulfilling the criteria tended to be associated with a higher threshold to treat hyperlipidemia. Similarly, guidelines meeting quality criteria tended to give less aggressive recommendations for cholesterol screening than did guidelines not fulfilling the criteria. The criterion on stakeholder involvement was the exception, but this criterion was fulfilled by all but 1 of the guidelines (Table 4).

TABLE 4

Guidelines for hyperlipidemia: relation between adherence to methodologic criteria and recommendations given*

	Guidelines recommending a low treatment-threshold ‡
Criterion	Criterion fulfilled	Criterion not fulfilled	P
Main outcomes identified	3/4 (75)	7/12 (58)	1.00†
Key stakeholders involved	5/10 (50)	3/4 (75)	.58†
Systematic search and selection	0/3	10/13 (77)	.036†
Recommendations linked to evidence	1/3 (33)	9/13 (69)	.52†
Benefits and risks considered	4/10 (40)	6/6 (100)	.034†
Resources/costs	4/8 (50)	6/8 (75)	.61†
Conflicts of interest stated	0/2	10/14 (71)	.13†
	Population to screen annually §
Main outcomes identified	8 (0–21)	11 (5–17)	3 (−8.1 to 14)¶
Key stakeholders involved	11 (4.8–16)	1†	−10||,¶
Systematic search and selection	4 (0–9.8)	12 (6.2–18)	8 (−2.3 to 18)¶
Recommendations linked to evidence	6 (0–13)	11 (5.3–18)	5 (−5.5 to 16)¶
Benefits and risks considered	8 (1.2–15)	14 (6.0–21)	6 (−4.1 to 15)¶
Resources/costs	6 (0.9–12)	14 (5.8–22)	8 (−1.2 to 16)¶
Conflicts of interest stated	0	12 (6.6–16)	12 (−1.8 to 25)¶
*The criterion on industry influence is not included because all the guidelines either fulfilled the criterion or provided insufficient information to assess if the criterion was met
† P values assessed with the Fisher exact test.
‡ Guidelines in which the threshold to treat is less than 310 mg/dL for 3 or more of the clinical scenarios described in the text. Data are presented as proportion (%).
§ Data are presented as percentage (95% confidence interval).
|| One guideline did not fulfill this criterion, so confidence intervals could not be calculated.
¶ Difference in percentage (95% confidence interval).

Stakeholder involvement and sponsorship by specialty societies

Guidelines that involved major stakeholders in the development process tended to fulfill the methodologic criteria to a greater extent than did guidelines that did not (Table W2, available on the JFP web site: http://www.jfponline.com). Nine of the 33 guidelines were sponsored by specialty societies. These fulfilled the methodologic criteria less often than did other guidelines (Table 5).

TABLE 5

Relation between specialty society sponsorship and fulfillment of methodologic criteria*

	Guidelines fulfilling criterion
Criterion	specialty by specialty society	Sponsored by Not sponsored society	P †
Main outcomes identified	6/9 (67)	4/24 (17)	.010
Key stakeholders involved‡	2/9 (22)	19/21 (90)	.001
Systematic search and selection	1/9 (11)	6/24 (25)	.64
Recommendations linked to evidence	2/9 (22)	8/24 (33)	.69
Benefits and risks considered	3/9 (33)	18/24 (75)	.044
Resources/costs	3/9 (33)	11/24 (46)	.70
No industry influence‡	2/4 (50)	21/21 (100)	.02
Conflicts of interest stated	2/9 (22)	2/24 (8)	.30
*Data are presented as proportion (%).
† Assessed with the Fisher exact test.
‡ We did not take into account the guidelines for which we had insufficient information to assess whether the criterion was met.

Discussion

We found that nonadherence to rigorous methods when developing guidelines for hypertension and hyperlipidemia tends to be associated with more aggressive recommendations. We are not aware of other studies that have investigated the relation between methods and recommendations in clinical practice guidelines. The relatively small number of guidelines that met our inclusion criteria limited the power of our analyses, which rarely reached the conventional level of statistical significance (P

Many articles have assessed the methodologic quality of clinical practice guidelines with the use of similar criteria, all these studies found poor adherence to recommendations for guideline development.^8-11 Grilli and colleagues found that “the quality of reporting of practice guidelines produced by specialty societies fell short of acceptable methodology” for the 431 guidelines they assessed.^10(p104) Shaneyfelt and colleagues found no difference in methodologic rigor between guidelines published by specialty societies and those published by others but decided that methodologic criteria frequently were not met.¹¹ We also found that methodologic criteria frequently were not met, and that they were met less often for guidelines sponsored by specialty societies than for those sponsored by other groups.

 

Stakeholder involvement, as we have defined it, was closely related to panel composition, which has been examined by others. For example, a link was found between panel composition and ratings of the appropriateness of procedures. Those who used a given procedure were more likely to rate it as appropriate than were those who did not use it.^15,16 Murphy and coworkers found that “members of a specialty are more likely to advocate techniques that involve their special-ty.”^17(p37) Savoie and colleagues, in their critical appraisal of guidelines for cholesterol testing, found that “the greater the involvement of clinical experts in the development process of the clinical practice guidelines, the less the recommendations reflected the research evidence.”^9(p76) This is consistent with our finding that broader stakeholder involvement was associated with methodologic criteria being met more often.

In our study, guideline developers that did not use rigorous methods appeared more likely to promote aggressive intervention. This may be true for guidelines for conditions other than hypertension and hyperlipidemia. However, guideline developers also may introduce biases toward less aggressive recommendations, eg, purchasers of health services. The degree to which bias is likely and even the direction sometimes may be difficult to predict.

The quality among the guidelines we assessed was not associated with year of publication or the country where the guidelines were developed. The 6 guidelines fulfilling 5 or more of the quality criteria were not published more recently. The countries of origin for these 6 guidelines were Australia, Canada, France, the United Kingdom, and the United States.

There are strong logical reasons for users of guidelines to consider the methods used by guideline developers. Given the extent of disagreement among guidelines, it is necessary for users to understand the basis of those recommendations. This is only possible if guideline developers employ systematic methods and explicitly report the methods that were used. Our study provides empirical support of skepticism toward guidelines that have been developed without employing systematic methods.

· Acknowledgments ·

Wethank Signe Flottorp and Lena Nordheim for helping with the appraisal of non-English guidelines and Jonathan Lomas and Brian Hutchison who helped develop the idea for this study.

Clinicians are inundated with clinical practice guidelines. Many guidelines address the same problems, often with conflicting recommendations. The disagreement concerning recommendations often hinges on how aggressive clinicians are in promoting interventions, eg, screening for prostate cancer or prescribe antibiotics for the treatment of sore throat. There is also variation with regard to the methods that guideline developers use to make recommendations. The traditional “GOBSAT”-technique (“Good Old Boys Sat At Table”) has been criticized.¹ Several groups have proposed more systematic approaches for guideline development,^2-6 and some databases, such as the Guideline Advisory Committee’s Recommended Clinical Practice Guidelines, include only those guidelines that have been assessed for the rigor of the development process.⁷

There are strong logical arguments for developing guidelines systematically, eg, to ensure that they are based on current best evidence, to protect against bias, and to make the process transparent and open to criticism. However, guideline developers frequently do not adhere to such methods.^8-11 We explored the possible association between the methods used in guideline development and the recommendations given in those guidelines. We used guidelines for hypertension and hyperlipidemia in our study. Our hypothesis was that less rigorous methods would, on average, be associated with more aggressive recommendations in these guidelines.

Methods

Inclusion criteria

We defined clinical guidelines as recommendations intended to assist health professionals and patients in making decisions for specific clinical circumstances. To be included a guideline had to address at least 1 of the following issues: threshold for drug treatment of essential hypertension in primary prevention, threshold for drug treatment of hyperlipidemia in primary prevention, or identification of the target population for cholesterol screening. Guidelines were excluded if they did not clearly identify the panel responsible for developing the guideline, identify a sponsoring organization, or include a reference list. We excluded textbooks, editorials, and commentaries. Review articles that were prepared and published as background documents were included with the relevant clinical practice guidelines. We included guidelines published after 1992. When multiple versions were available from the same organization, we used the most recent version.

Search strategy

We searched MEDLINE from 1992 through February 2000 by using hypertension, blood pressure, hyperlipidemia, or cholesterol as the key term, limited to practice guidelines as publication type. In addition, we searched databases of guidelines maintained by several groups around the world. One author (A.F.) reviewed all the citations and reference lists of relevant guidelines, retrieved potentially relevant guidelines, and selected guidelines for inclusion.

Guideline development methods

We used 8 criteria to rate the methodologic quality of the guidelines (Table 1). The criteria were adapted from a guideline appraisal instrument that is being developed and tested by a group of European researchers (the Agree/Biomed collaboration).¹² It is based on a British Appraisal Instrument for Clinical Guidelines ¹³ that has been tested for its validity and reliability and has been characterized as “the most well developed to date” in a recent study.¹⁴

Two authors (A.F. and J.W.W.) evaluated each guideline independently. For analytical purposes, all criteria were dichotomized (Table 1). Because fewer than 50% of the guidelines reported sufficient information to determine stakeholder involvement, we supplemented information on authors through an Internet search.

TABLE 1

Criteria used to appraise guideline-development methods

Criterion	Standard for fulfillment
Main outcomes identified
Is there an explicit statement of the main outcomes considered when developing the guideline?	Explicit statement of the main outcomes considered in developing the guidelines
Key stakeholders involved
Are the essential stakeholders involved in the development group?	Inclusion of all “essential” stakeholders (generalist physicians, specialists, and methodologists)*
Systematic search and selection
Has a systematic search for evidence been carried out and are criteria for inclusion and exclusion specified?	Search specifying all relevant databases or described “electronic databases” or
	Inclusion–exclusion criteria defined, at least briefly
Recommendations linked to evidence
Is there an explicit link between the evidence and the recommendations given?	Grading of strength of recommendations or level of evidence
Benefits and risks considered
Have the health benefits, side effects, and risks been considered?	Any quantitative or qualitative weighing of benefits and harms that is incorporated into formulating recommendations
Resources/costs
Has the impact on resources been considered?	Economic analysis or qualitative consideration of cost issues that is linked explicitly to the formulation of the recommendations
No industry influence
Is the guideline developed without funding or influence from the pharmaceutical industry?	No financial support from a pharmaceutical company and no involvement on the panel
Conflicts of interest stated
Is there an explicit statement of conflict of interest?	Statement of potential conflicts of interest for panel members
*Essential stakeholders are generalist physicians, specialists, and methodologists. Optional stakeholders are patients, policy makers/health administrators, nurses, pharmacists, economists, and other physicians.

 

Aggressiveness of recommendations

To grade the aggressiveness of the treatment recommendations, we evaluated the threshold for initiating pharmacologic treatment of hypertension (low = aggressive), threshold for initiating pharmacologic treatment of hyperlipidemia (low = aggressive), first-line antihypertensive drug (all drugs = aggressive), and the number of persons eligible for cholesterol screening (high = aggressive).

The thresholds for treatment of hypertension and hyperlipidemia were categorized with 4 clinical scenarios. By applying the scenarios we could determine the recommended thresholds for drug treatment among the various guidelines. We chose these specific scenarios to illustrate the existing variation in recommendations among the guidelines:

1. A 50-year-old man without a high-risk profile of cardiovascular disease.
   
2. A 50-year-old man with a high-risk profile of cardiovascular disease.
   
3. A 70-year-old man without a high-risk profile of cardiovascular disease.
   
4. A 70-year-old man with a high-risk profile of cardiovascular disease.
   

High-risk profile was defined separately for treatment of hypertension and treatment of hyperlipidemia. For hypertension, the high-risk scenario was a patient who smoked and was hyperlipidemic (total cholesterol > 310 mg/dL). For hyperlipidemia, the high-risk scenario was a patient who smoked and was hypertensive (blood pressure > 160/100 mm Hg). We assumed that lifestyle interventions had been attempted. For simplicity we used only the systolic value for blood pressure. We dichotomized the aggressiveness of thresholds for lipid-lowering treatment rather than calculating averages because many guidelines gave recommendations other than actual cholesterol values; eg, no treatment or familial hypercholesterolemia. Guidelines were considered nonaggressive if, for 2 or more of the scenarios, the threshold was set at 310 mg/dL or higher or was specified as familial hypercholesterolemia.

Guidelines recommending all common antihypertensive drugs were considered aggressive, and the ones suggesting more restrictive recommendations were considered nonaggressive. In a few guidelines this recommendation depended on the patient’s age. For simplicity we examined the recommendations for 50-year-olds. We graded aggressiveness of recommendations on cholesterol screening by estimating the proportion of the general adult population (in Norway) who would be candidates for screening or by case finding per year, if the guidelines were fully implemented.

Analysis

We qualitatively and quantitatively examined the relation between fulfillment of a methodologic criterion and the aggressiveness of recommendations. The power of our statistical analyses was limited by the available sample size. For hypertension, we averaged the treatment threshold for the 4 clinical scenarios within each guideline and compared the overall mean between guidelines meeting and not meeting the criterion. For the threshold to treat hyperlipidemia and for first-line therapy for hypertension, the degree of aggressiveness was dichotomized. We used the Fisher exact test to calculate P values for the association between the proportion of guidelines fulfilling a methodologic criterion and whether the recommendation was classified as aggressive. For cholesterol screening we found the mean yearly proportion of the adult population eligible for screening among guidelines fulfilling a methodologic criterion, and compared this with the mean for guidelines that did not fulfill the criterion. In addition to examining the association between methods and recommendations, we examined whether the level of stakeholder involvement or sponsorship by specialty societies was associated with fulfillment of the methodologic criteria. We included generalist physicians, specialists, and methodologists as “essential stakeholders” and patients, policy makers/health administrators, nurses, pharmacists, economists, and other physicians as “optional stakeholders.”

Results

We found 12 clinical guidelines for managing hypertension, 12 for hyperlipidemia, 5 for cholesterol screening, and 4 general guidelines for the prevention of coronary heart disease that met our inclusion criteria (references are available from the authors). Because each guideline was appraised according to the 8 methodologic criteria, we ended up with 264 appraisals (8 criteria applied to each of the 33 guidelines). There were 28 disagreements (11%), all of which were easily resolved by discussion. As expected, there was variation among the guidelines regarding fulfillment of methodologic criteria and the aggressiveness of recommendations.

Most guidelines did not meet the majority of the methodologic criteria (Table 2). Only 6 of the 33 guidelines met 5 or more of the 8 criteria. The threshold to start antihypertensive treatment varied in systolic blood pressure from 140 to 180 mm Hg for each of the 4 clinical scenarios we applied to the guidelines (Table 3). For 3 of the scenarios, the threshold to treat hyperlipidemia ranged from a total cholesterol value of 190 mg/dL to more than 310 mg/dL (Table 3). Fifteen guidelines gave recommendations for first-line therapy for hypertension. Three recommended thiazides only; 6 recommended thiazides and β-blockers; 1 recommended thiazides, β-blockers, and angiotensinconverting enzyme inhibitors; and 5 recommended all commonly used drugs. Recommendations for cholesterol screening ranged from no screening to testing the entire adult population every 2 to 5 years.

 

TABLE 2

Variability in fulfillment of methodologic criteria

Criterion	Fulfilled, n (%)	Not fulfilled, n (%)	No information, n (%)
Main outcomes identified	10 (30)	23 (70)	0 (0)
Key stakeholders involved	21 (64)	9 (27)	3 (9)
Systematic search and selection	7 (21)	26 (79)	0 (0)
Recommendations linked to evidence	10 (30)	23 (70)	0 (0)
Benefits and risks considered	21 (64)	12 (36)	0 (0)
Resources/costs	14 (42)	19 (58)	0 (0)
No industry influence	23 (70)	2 (6)	8 (24)
Conflicts of interest stated	4 (12)	29 (88)	0 (0)

TABLE 3

Variability in guideline recommendations*

	BP threshold (mm Hg) to treat hypertension
Age, clinical scenario	140	150	160	170	180
50 y, low risk	4 (25)	1 (6)	5 (31)	4 (25)	2 (13)
50 y, high risk	9 (56)	2 (13)	3 (19)	1 (6)	1 (6)
70 y, high risk	6 (38)	1 (6)	5 (31)	2 (13)	2 (13)
70 y, low risk	8 (50)	1 (6)	5 (31)	1 (6)	1 (6)
	Cholesterol threshold (mg/dL) to treat hyperlipidemia
		≤230	≤270	≥310 †
50 y, low risk	3 (19)	3 (19)	1 (6)	9 (56)
50 y, high risk	6 (38)	8 (50)	0	2 (13)
70 y, high risk	5 (31)	3 (19)	2 (13)	3 (19)
70 y, low risk	7 (44)	5 (31)	1 (6)	3 (19)
*Data are presented as number (%) of patients.
† Includes the recommendations “no treatment” and “familial hypercholesterolemia.”
BP, blood pressure.

Associations between recommendations and methodologic criteria

The threshold to treat hypertension did not seem to be associated with fulfillment of methodologic criteria. Differences in recommendations for first-line drugs for hypertension were not strongly associated with any of the criteria. Although not statistically significant, there was a trend for guidelines to recommend all commonly available drugs when methodologic criteria were not met (Table W1, available on the JFP web site: http://www.jfponline.com).

For all but 1 quality criteria (main outcomes identified), fulfilling the criteria tended to be associated with a higher threshold to treat hyperlipidemia. Similarly, guidelines meeting quality criteria tended to give less aggressive recommendations for cholesterol screening than did guidelines not fulfilling the criteria. The criterion on stakeholder involvement was the exception, but this criterion was fulfilled by all but 1 of the guidelines (Table 4).

TABLE 4

Guidelines for hyperlipidemia: relation between adherence to methodologic criteria and recommendations given*

	Guidelines recommending a low treatment-threshold ‡
Criterion	Criterion fulfilled	Criterion not fulfilled	P
Main outcomes identified	3/4 (75)	7/12 (58)	1.00†
Key stakeholders involved	5/10 (50)	3/4 (75)	.58†
Systematic search and selection	0/3	10/13 (77)	.036†
Recommendations linked to evidence	1/3 (33)	9/13 (69)	.52†
Benefits and risks considered	4/10 (40)	6/6 (100)	.034†
Resources/costs	4/8 (50)	6/8 (75)	.61†
Conflicts of interest stated	0/2	10/14 (71)	.13†
	Population to screen annually §
Main outcomes identified	8 (0–21)	11 (5–17)	3 (−8.1 to 14)¶
Key stakeholders involved	11 (4.8–16)	1†	−10||,¶
Systematic search and selection	4 (0–9.8)	12 (6.2–18)	8 (−2.3 to 18)¶
Recommendations linked to evidence	6 (0–13)	11 (5.3–18)	5 (−5.5 to 16)¶
Benefits and risks considered	8 (1.2–15)	14 (6.0–21)	6 (−4.1 to 15)¶
Resources/costs	6 (0.9–12)	14 (5.8–22)	8 (−1.2 to 16)¶
Conflicts of interest stated	0	12 (6.6–16)	12 (−1.8 to 25)¶
*The criterion on industry influence is not included because all the guidelines either fulfilled the criterion or provided insufficient information to assess if the criterion was met
† P values assessed with the Fisher exact test.
‡ Guidelines in which the threshold to treat is less than 310 mg/dL for 3 or more of the clinical scenarios described in the text. Data are presented as proportion (%).
§ Data are presented as percentage (95% confidence interval).
|| One guideline did not fulfill this criterion, so confidence intervals could not be calculated.
¶ Difference in percentage (95% confidence interval).

Stakeholder involvement and sponsorship by specialty societies

Guidelines that involved major stakeholders in the development process tended to fulfill the methodologic criteria to a greater extent than did guidelines that did not (Table W2, available on the JFP web site: http://www.jfponline.com). Nine of the 33 guidelines were sponsored by specialty societies. These fulfilled the methodologic criteria less often than did other guidelines (Table 5).

TABLE 5

Relation between specialty society sponsorship and fulfillment of methodologic criteria*

	Guidelines fulfilling criterion
Criterion	specialty by specialty society	Sponsored by Not sponsored society	P †
Main outcomes identified	6/9 (67)	4/24 (17)	.010
Key stakeholders involved‡	2/9 (22)	19/21 (90)	.001
Systematic search and selection	1/9 (11)	6/24 (25)	.64
Recommendations linked to evidence	2/9 (22)	8/24 (33)	.69
Benefits and risks considered	3/9 (33)	18/24 (75)	.044
Resources/costs	3/9 (33)	11/24 (46)	.70
No industry influence‡	2/4 (50)	21/21 (100)	.02
Conflicts of interest stated	2/9 (22)	2/24 (8)	.30
*Data are presented as proportion (%).
† Assessed with the Fisher exact test.
‡ We did not take into account the guidelines for which we had insufficient information to assess whether the criterion was met.

Discussion

We found that nonadherence to rigorous methods when developing guidelines for hypertension and hyperlipidemia tends to be associated with more aggressive recommendations. We are not aware of other studies that have investigated the relation between methods and recommendations in clinical practice guidelines. The relatively small number of guidelines that met our inclusion criteria limited the power of our analyses, which rarely reached the conventional level of statistical significance (P

Many articles have assessed the methodologic quality of clinical practice guidelines with the use of similar criteria, all these studies found poor adherence to recommendations for guideline development.^8-11 Grilli and colleagues found that “the quality of reporting of practice guidelines produced by specialty societies fell short of acceptable methodology” for the 431 guidelines they assessed.^10(p104) Shaneyfelt and colleagues found no difference in methodologic rigor between guidelines published by specialty societies and those published by others but decided that methodologic criteria frequently were not met.¹¹ We also found that methodologic criteria frequently were not met, and that they were met less often for guidelines sponsored by specialty societies than for those sponsored by other groups.

 

Stakeholder involvement, as we have defined it, was closely related to panel composition, which has been examined by others. For example, a link was found between panel composition and ratings of the appropriateness of procedures. Those who used a given procedure were more likely to rate it as appropriate than were those who did not use it.^15,16 Murphy and coworkers found that “members of a specialty are more likely to advocate techniques that involve their special-ty.”^17(p37) Savoie and colleagues, in their critical appraisal of guidelines for cholesterol testing, found that “the greater the involvement of clinical experts in the development process of the clinical practice guidelines, the less the recommendations reflected the research evidence.”^9(p76) This is consistent with our finding that broader stakeholder involvement was associated with methodologic criteria being met more often.

In our study, guideline developers that did not use rigorous methods appeared more likely to promote aggressive intervention. This may be true for guidelines for conditions other than hypertension and hyperlipidemia. However, guideline developers also may introduce biases toward less aggressive recommendations, eg, purchasers of health services. The degree to which bias is likely and even the direction sometimes may be difficult to predict.

The quality among the guidelines we assessed was not associated with year of publication or the country where the guidelines were developed. The 6 guidelines fulfilling 5 or more of the quality criteria were not published more recently. The countries of origin for these 6 guidelines were Australia, Canada, France, the United Kingdom, and the United States.

There are strong logical reasons for users of guidelines to consider the methods used by guideline developers. Given the extent of disagreement among guidelines, it is necessary for users to understand the basis of those recommendations. This is only possible if guideline developers employ systematic methods and explicitly report the methods that were used. Our study provides empirical support of skepticism toward guidelines that have been developed without employing systematic methods.

· Acknowledgments ·

Wethank Signe Flottorp and Lena Nordheim for helping with the appraisal of non-English guidelines and Jonathan Lomas and Brian Hutchison who helped develop the idea for this study.

Clinicians are inundated with clinical practice guidelines. Many guidelines address the same problems, often with conflicting recommendations. The disagreement concerning recommendations often hinges on how aggressive clinicians are in promoting interventions, eg, screening for prostate cancer or prescribe antibiotics for the treatment of sore throat. There is also variation with regard to the methods that guideline developers use to make recommendations. The traditional “GOBSAT”-technique (“Good Old Boys Sat At Table”) has been criticized.¹ Several groups have proposed more systematic approaches for guideline development,^2-6 and some databases, such as the Guideline Advisory Committee’s Recommended Clinical Practice Guidelines, include only those guidelines that have been assessed for the rigor of the development process.⁷

There are strong logical arguments for developing guidelines systematically, eg, to ensure that they are based on current best evidence, to protect against bias, and to make the process transparent and open to criticism. However, guideline developers frequently do not adhere to such methods.^8-11 We explored the possible association between the methods used in guideline development and the recommendations given in those guidelines. We used guidelines for hypertension and hyperlipidemia in our study. Our hypothesis was that less rigorous methods would, on average, be associated with more aggressive recommendations in these guidelines.

Methods

Inclusion criteria

We defined clinical guidelines as recommendations intended to assist health professionals and patients in making decisions for specific clinical circumstances. To be included a guideline had to address at least 1 of the following issues: threshold for drug treatment of essential hypertension in primary prevention, threshold for drug treatment of hyperlipidemia in primary prevention, or identification of the target population for cholesterol screening. Guidelines were excluded if they did not clearly identify the panel responsible for developing the guideline, identify a sponsoring organization, or include a reference list. We excluded textbooks, editorials, and commentaries. Review articles that were prepared and published as background documents were included with the relevant clinical practice guidelines. We included guidelines published after 1992. When multiple versions were available from the same organization, we used the most recent version.

Search strategy

We searched MEDLINE from 1992 through February 2000 by using hypertension, blood pressure, hyperlipidemia, or cholesterol as the key term, limited to practice guidelines as publication type. In addition, we searched databases of guidelines maintained by several groups around the world. One author (A.F.) reviewed all the citations and reference lists of relevant guidelines, retrieved potentially relevant guidelines, and selected guidelines for inclusion.

Guideline development methods

We used 8 criteria to rate the methodologic quality of the guidelines (Table 1). The criteria were adapted from a guideline appraisal instrument that is being developed and tested by a group of European researchers (the Agree/Biomed collaboration).¹² It is based on a British Appraisal Instrument for Clinical Guidelines ¹³ that has been tested for its validity and reliability and has been characterized as “the most well developed to date” in a recent study.¹⁴

Two authors (A.F. and J.W.W.) evaluated each guideline independently. For analytical purposes, all criteria were dichotomized (Table 1). Because fewer than 50% of the guidelines reported sufficient information to determine stakeholder involvement, we supplemented information on authors through an Internet search.

TABLE 1

Criteria used to appraise guideline-development methods

Criterion	Standard for fulfillment
Main outcomes identified
Is there an explicit statement of the main outcomes considered when developing the guideline?	Explicit statement of the main outcomes considered in developing the guidelines
Key stakeholders involved
Are the essential stakeholders involved in the development group?	Inclusion of all “essential” stakeholders (generalist physicians, specialists, and methodologists)*
Systematic search and selection
Has a systematic search for evidence been carried out and are criteria for inclusion and exclusion specified?	Search specifying all relevant databases or described “electronic databases” or
	Inclusion–exclusion criteria defined, at least briefly
Recommendations linked to evidence
Is there an explicit link between the evidence and the recommendations given?	Grading of strength of recommendations or level of evidence
Benefits and risks considered
Have the health benefits, side effects, and risks been considered?	Any quantitative or qualitative weighing of benefits and harms that is incorporated into formulating recommendations
Resources/costs
Has the impact on resources been considered?	Economic analysis or qualitative consideration of cost issues that is linked explicitly to the formulation of the recommendations
No industry influence
Is the guideline developed without funding or influence from the pharmaceutical industry?	No financial support from a pharmaceutical company and no involvement on the panel
Conflicts of interest stated
Is there an explicit statement of conflict of interest?	Statement of potential conflicts of interest for panel members
*Essential stakeholders are generalist physicians, specialists, and methodologists. Optional stakeholders are patients, policy makers/health administrators, nurses, pharmacists, economists, and other physicians.

 

Aggressiveness of recommendations

To grade the aggressiveness of the treatment recommendations, we evaluated the threshold for initiating pharmacologic treatment of hypertension (low = aggressive), threshold for initiating pharmacologic treatment of hyperlipidemia (low = aggressive), first-line antihypertensive drug (all drugs = aggressive), and the number of persons eligible for cholesterol screening (high = aggressive).

The thresholds for treatment of hypertension and hyperlipidemia were categorized with 4 clinical scenarios. By applying the scenarios we could determine the recommended thresholds for drug treatment among the various guidelines. We chose these specific scenarios to illustrate the existing variation in recommendations among the guidelines:

1. A 50-year-old man without a high-risk profile of cardiovascular disease.
   
2. A 50-year-old man with a high-risk profile of cardiovascular disease.
   
3. A 70-year-old man without a high-risk profile of cardiovascular disease.
   
4. A 70-year-old man with a high-risk profile of cardiovascular disease.
   

High-risk profile was defined separately for treatment of hypertension and treatment of hyperlipidemia. For hypertension, the high-risk scenario was a patient who smoked and was hyperlipidemic (total cholesterol > 310 mg/dL). For hyperlipidemia, the high-risk scenario was a patient who smoked and was hypertensive (blood pressure > 160/100 mm Hg). We assumed that lifestyle interventions had been attempted. For simplicity we used only the systolic value for blood pressure. We dichotomized the aggressiveness of thresholds for lipid-lowering treatment rather than calculating averages because many guidelines gave recommendations other than actual cholesterol values; eg, no treatment or familial hypercholesterolemia. Guidelines were considered nonaggressive if, for 2 or more of the scenarios, the threshold was set at 310 mg/dL or higher or was specified as familial hypercholesterolemia.

Guidelines recommending all common antihypertensive drugs were considered aggressive, and the ones suggesting more restrictive recommendations were considered nonaggressive. In a few guidelines this recommendation depended on the patient’s age. For simplicity we examined the recommendations for 50-year-olds. We graded aggressiveness of recommendations on cholesterol screening by estimating the proportion of the general adult population (in Norway) who would be candidates for screening or by case finding per year, if the guidelines were fully implemented.

Analysis

We qualitatively and quantitatively examined the relation between fulfillment of a methodologic criterion and the aggressiveness of recommendations. The power of our statistical analyses was limited by the available sample size. For hypertension, we averaged the treatment threshold for the 4 clinical scenarios within each guideline and compared the overall mean between guidelines meeting and not meeting the criterion. For the threshold to treat hyperlipidemia and for first-line therapy for hypertension, the degree of aggressiveness was dichotomized. We used the Fisher exact test to calculate P values for the association between the proportion of guidelines fulfilling a methodologic criterion and whether the recommendation was classified as aggressive. For cholesterol screening we found the mean yearly proportion of the adult population eligible for screening among guidelines fulfilling a methodologic criterion, and compared this with the mean for guidelines that did not fulfill the criterion. In addition to examining the association between methods and recommendations, we examined whether the level of stakeholder involvement or sponsorship by specialty societies was associated with fulfillment of the methodologic criteria. We included generalist physicians, specialists, and methodologists as “essential stakeholders” and patients, policy makers/health administrators, nurses, pharmacists, economists, and other physicians as “optional stakeholders.”

Results

We found 12 clinical guidelines for managing hypertension, 12 for hyperlipidemia, 5 for cholesterol screening, and 4 general guidelines for the prevention of coronary heart disease that met our inclusion criteria (references are available from the authors). Because each guideline was appraised according to the 8 methodologic criteria, we ended up with 264 appraisals (8 criteria applied to each of the 33 guidelines). There were 28 disagreements (11%), all of which were easily resolved by discussion. As expected, there was variation among the guidelines regarding fulfillment of methodologic criteria and the aggressiveness of recommendations.

Most guidelines did not meet the majority of the methodologic criteria (Table 2). Only 6 of the 33 guidelines met 5 or more of the 8 criteria. The threshold to start antihypertensive treatment varied in systolic blood pressure from 140 to 180 mm Hg for each of the 4 clinical scenarios we applied to the guidelines (Table 3). For 3 of the scenarios, the threshold to treat hyperlipidemia ranged from a total cholesterol value of 190 mg/dL to more than 310 mg/dL (Table 3). Fifteen guidelines gave recommendations for first-line therapy for hypertension. Three recommended thiazides only; 6 recommended thiazides and β-blockers; 1 recommended thiazides, β-blockers, and angiotensinconverting enzyme inhibitors; and 5 recommended all commonly used drugs. Recommendations for cholesterol screening ranged from no screening to testing the entire adult population every 2 to 5 years.

 

TABLE 2

Variability in fulfillment of methodologic criteria

Criterion	Fulfilled, n (%)	Not fulfilled, n (%)	No information, n (%)
Main outcomes identified	10 (30)	23 (70)	0 (0)
Key stakeholders involved	21 (64)	9 (27)	3 (9)
Systematic search and selection	7 (21)	26 (79)	0 (0)
Recommendations linked to evidence	10 (30)	23 (70)	0 (0)
Benefits and risks considered	21 (64)	12 (36)	0 (0)
Resources/costs	14 (42)	19 (58)	0 (0)
No industry influence	23 (70)	2 (6)	8 (24)
Conflicts of interest stated	4 (12)	29 (88)	0 (0)

TABLE 3

Variability in guideline recommendations*

	BP threshold (mm Hg) to treat hypertension
Age, clinical scenario	140	150	160	170	180
50 y, low risk	4 (25)	1 (6)	5 (31)	4 (25)	2 (13)
50 y, high risk	9 (56)	2 (13)	3 (19)	1 (6)	1 (6)
70 y, high risk	6 (38)	1 (6)	5 (31)	2 (13)	2 (13)
70 y, low risk	8 (50)	1 (6)	5 (31)	1 (6)	1 (6)
	Cholesterol threshold (mg/dL) to treat hyperlipidemia
		≤230	≤270	≥310 †
50 y, low risk	3 (19)	3 (19)	1 (6)	9 (56)
50 y, high risk	6 (38)	8 (50)	0	2 (13)
70 y, high risk	5 (31)	3 (19)	2 (13)	3 (19)
70 y, low risk	7 (44)	5 (31)	1 (6)	3 (19)
*Data are presented as number (%) of patients.
† Includes the recommendations “no treatment” and “familial hypercholesterolemia.”
BP, blood pressure.

Associations between recommendations and methodologic criteria

The threshold to treat hypertension did not seem to be associated with fulfillment of methodologic criteria. Differences in recommendations for first-line drugs for hypertension were not strongly associated with any of the criteria. Although not statistically significant, there was a trend for guidelines to recommend all commonly available drugs when methodologic criteria were not met (Table W1, available on the JFP web site: http://www.jfponline.com).

For all but 1 quality criteria (main outcomes identified), fulfilling the criteria tended to be associated with a higher threshold to treat hyperlipidemia. Similarly, guidelines meeting quality criteria tended to give less aggressive recommendations for cholesterol screening than did guidelines not fulfilling the criteria. The criterion on stakeholder involvement was the exception, but this criterion was fulfilled by all but 1 of the guidelines (Table 4).

TABLE 4

Guidelines for hyperlipidemia: relation between adherence to methodologic criteria and recommendations given*

	Guidelines recommending a low treatment-threshold ‡
Criterion	Criterion fulfilled	Criterion not fulfilled	P
Main outcomes identified	3/4 (75)	7/12 (58)	1.00†
Key stakeholders involved	5/10 (50)	3/4 (75)	.58†
Systematic search and selection	0/3	10/13 (77)	.036†
Recommendations linked to evidence	1/3 (33)	9/13 (69)	.52†
Benefits and risks considered	4/10 (40)	6/6 (100)	.034†
Resources/costs	4/8 (50)	6/8 (75)	.61†
Conflicts of interest stated	0/2	10/14 (71)	.13†
	Population to screen annually §
Main outcomes identified	8 (0–21)	11 (5–17)	3 (−8.1 to 14)¶
Key stakeholders involved	11 (4.8–16)	1†	−10||,¶
Systematic search and selection	4 (0–9.8)	12 (6.2–18)	8 (−2.3 to 18)¶
Recommendations linked to evidence	6 (0–13)	11 (5.3–18)	5 (−5.5 to 16)¶
Benefits and risks considered	8 (1.2–15)	14 (6.0–21)	6 (−4.1 to 15)¶
Resources/costs	6 (0.9–12)	14 (5.8–22)	8 (−1.2 to 16)¶
Conflicts of interest stated	0	12 (6.6–16)	12 (−1.8 to 25)¶
*The criterion on industry influence is not included because all the guidelines either fulfilled the criterion or provided insufficient information to assess if the criterion was met
† P values assessed with the Fisher exact test.
‡ Guidelines in which the threshold to treat is less than 310 mg/dL for 3 or more of the clinical scenarios described in the text. Data are presented as proportion (%).
§ Data are presented as percentage (95% confidence interval).
|| One guideline did not fulfill this criterion, so confidence intervals could not be calculated.
¶ Difference in percentage (95% confidence interval).

Stakeholder involvement and sponsorship by specialty societies

Guidelines that involved major stakeholders in the development process tended to fulfill the methodologic criteria to a greater extent than did guidelines that did not (Table W2, available on the JFP web site: http://www.jfponline.com). Nine of the 33 guidelines were sponsored by specialty societies. These fulfilled the methodologic criteria less often than did other guidelines (Table 5).

TABLE 5

Relation between specialty society sponsorship and fulfillment of methodologic criteria*

	Guidelines fulfilling criterion
Criterion	specialty by specialty society	Sponsored by Not sponsored society	P †
Main outcomes identified	6/9 (67)	4/24 (17)	.010
Key stakeholders involved‡	2/9 (22)	19/21 (90)	.001
Systematic search and selection	1/9 (11)	6/24 (25)	.64
Recommendations linked to evidence	2/9 (22)	8/24 (33)	.69
Benefits and risks considered	3/9 (33)	18/24 (75)	.044
Resources/costs	3/9 (33)	11/24 (46)	.70
No industry influence‡	2/4 (50)	21/21 (100)	.02
Conflicts of interest stated	2/9 (22)	2/24 (8)	.30
*Data are presented as proportion (%).
† Assessed with the Fisher exact test.
‡ We did not take into account the guidelines for which we had insufficient information to assess whether the criterion was met.

Discussion

We found that nonadherence to rigorous methods when developing guidelines for hypertension and hyperlipidemia tends to be associated with more aggressive recommendations. We are not aware of other studies that have investigated the relation between methods and recommendations in clinical practice guidelines. The relatively small number of guidelines that met our inclusion criteria limited the power of our analyses, which rarely reached the conventional level of statistical significance (P

Many articles have assessed the methodologic quality of clinical practice guidelines with the use of similar criteria, all these studies found poor adherence to recommendations for guideline development.^8-11 Grilli and colleagues found that “the quality of reporting of practice guidelines produced by specialty societies fell short of acceptable methodology” for the 431 guidelines they assessed.^10(p104) Shaneyfelt and colleagues found no difference in methodologic rigor between guidelines published by specialty societies and those published by others but decided that methodologic criteria frequently were not met.¹¹ We also found that methodologic criteria frequently were not met, and that they were met less often for guidelines sponsored by specialty societies than for those sponsored by other groups.

 

Stakeholder involvement, as we have defined it, was closely related to panel composition, which has been examined by others. For example, a link was found between panel composition and ratings of the appropriateness of procedures. Those who used a given procedure were more likely to rate it as appropriate than were those who did not use it.^15,16 Murphy and coworkers found that “members of a specialty are more likely to advocate techniques that involve their special-ty.”^17(p37) Savoie and colleagues, in their critical appraisal of guidelines for cholesterol testing, found that “the greater the involvement of clinical experts in the development process of the clinical practice guidelines, the less the recommendations reflected the research evidence.”^9(p76) This is consistent with our finding that broader stakeholder involvement was associated with methodologic criteria being met more often.

In our study, guideline developers that did not use rigorous methods appeared more likely to promote aggressive intervention. This may be true for guidelines for conditions other than hypertension and hyperlipidemia. However, guideline developers also may introduce biases toward less aggressive recommendations, eg, purchasers of health services. The degree to which bias is likely and even the direction sometimes may be difficult to predict.

The quality among the guidelines we assessed was not associated with year of publication or the country where the guidelines were developed. The 6 guidelines fulfilling 5 or more of the quality criteria were not published more recently. The countries of origin for these 6 guidelines were Australia, Canada, France, the United Kingdom, and the United States.

There are strong logical reasons for users of guidelines to consider the methods used by guideline developers. Given the extent of disagreement among guidelines, it is necessary for users to understand the basis of those recommendations. This is only possible if guideline developers employ systematic methods and explicitly report the methods that were used. Our study provides empirical support of skepticism toward guidelines that have been developed without employing systematic methods.

· Acknowledgments ·

Wethank Signe Flottorp and Lena Nordheim for helping with the appraisal of non-English guidelines and Jonathan Lomas and Brian Hutchison who helped develop the idea for this study.