User login
- When consulting narrative review articles, carefully consider the possibility of citation bias and whether recommendations are based on patient-oriented and not disease-oriented outcomes.
- Consult credible evidence-based resources such as Cochrane before making changes in practice suggested by narrative reviews.
Background: Citations in scientific articles may tend to favor the views presented. We studied whether there is such reference bias in narrative review articles that discuss interventions against house dust mites for people with asthma.
Design: Systematic review of reviews identified in a Medline search that expressed an opinion about the clinical effects of physical or chemical intervention methods.
Main outcome measure: Positive bias was judged to have occurred if the reference list contained a higher proportion of trial references with significant results than among all trials available to the authors (published 2 years or more prior to the review).
Results: Seventy reviews were included, of which 63 (90%) recommended physical interventions. Forty-six reviews had trial references, 4 of these only to chemical interventions. In the remaining 42 reviews, reference bias was detected (P=2 x 10-8). The most quoted trial had only 7 patients per group, its claimed significant result was probably erroneous, and it did not report a clinical outcome. Intervention recommendations were often based on nonrandomized evidence, and the most quoted nonrandomized controlled study had included only 10 patients per group but claimed very positive results.
Conclusion: The narrative review articles were severely biased, and their positive intervention recommendations are at variance with the systematic Cochrane Review on this topic and a recent very large trial of physical intervention, both of which failed to find an effect.
Trial reports and narrative review articles sometimes favor references supporting the views of the authors, lending credence to a particular treatment1 or hypothesis.2,3 This reference bias may render the conclusions of an article less reliable. Such was the finding in our study of narrative review articles discussing interventions against house dust mites for people with asthma.
Systematic reviews are a more reliable source of information for busy clinicians than narrative reviews, but they can also be problematic. An assessment of papers on asthma, based on a validated tool, showed that 40 of 50 systematic reviews on prevention and treatment had serious or extensive flaws.4 Cochrane systematic reviews on asthma were more rigorous and better reported than systematic reviews published in peer-reviewed, paper-based journals,4 probably because Cochrane Reviews are conducted according to a set of standard methods5 aimed at minimizing bias in the reviews, and because they usually include only randomized trials.
As the index review for our study, we chose the Cochrane Review on house dust mites control measures for asthma,6 to allow us to judge whether the recommendations in narrative reviews reflected fairly the available, reliable scientific evidence.
Methods
Inclusion criteria. Narrative reviews—that is, reviews that did not have a methods section with a search strategy for relevant research papers—were eligible if they expressed an opinion about the clinical effects of chemical or physical interventions aimed at reducing exposure to house dust mite antigens in the homes of mite-sensitive patients with asthma.
Search strategy. A Medline search selected review articles from 1966 to July 2002 using the terms review* and mite* and asthma*, with no language restrictions. Additional review articles could have been found in other databases, but we chose Medline since it is freely available and likely preferred by physicians.
Data collection. Abstracts identified in the Medline search were read by one of us (LMS), who excluded articles that clearly were not reviews or that reviewed aspects other than treatments (eg, causes of asthma only). Both authors read the remaining articles in full, and conducted another round of exclusion.
Data extraction was performed independently and results were compared. Disagreements were resolved by discussion; they were usually due to simple oversight, such as a missing trial reference. Interventions were classified as physical (eg, vacuum cleaning, washing, bed covers, heating, freezing), chemical (eg, acaricides), or combinations. The review-authors’ opinions were classified into 2 groups: positive, recommending 1 or more interventions (in a few cases rather vaguely, such as allergen avoidance), or neutral/negative (eg, more research is needed or the interventions should not be used).
The index review. The Cochrane review on house dust-mite control measures for asthma was first published in 1998 in the Cochrane Library8 and in BMJ.8 The latest update from 2001 included 28 randomized trials, one of which was 3-armed.9 When referring to randomized trials in our study, the sample is the 28 trial reports included in the Cochrane review. The Cochrane review, which included 939 patients in the analysis, failed to find an effect of chemical or physical methods on any of the outcomes (number of patients improved, asthma symptoms, medication usage, peak flow, FEV1, and bronchial hyperreactivity).
Data analysis. For reviews with positive intervention recommendations, the supporting references to randomized trials were compared with all those randomized trials that might potentially have been known to the authors—that is, published 2 years or more prior to the publication date of the review.1
If, for example, a review recommended acaricides and quoted 3 trials that claimed a significant effect of this intervention and 1 trial that did not, and if 8 trials were available at the time but only 3 favored acaricides, this would be a positively biased selection of references, since the proportion of positive trials in the reference list, 3/4, was greater than among all the available trials, 3/8.1
We did not judge whether the significance testing was correctly done but noted whether the trial authors had reported 1 or more significant results favoring the intervention. We then used a sign test, for positive vs negative selection, to determine whether there was significant reference bias.
Results
The Medline search identified 302 abstracts, of which 151 were clearly of no relevance and 5 were abstracts of the index Cochrane review. Of the remaining 146, 6 were excluded because of the language; 140 were read in full, and a further 70 were excluded for other reasons. This left 70 reviews that all had intervention recommendations (positive, neutral, or negative); 63 of them (90%) positively recommended physical interventions and 30 (43%) recommended chemical interventions (TABLE).
The reviews were published between 1971 and 2002 (see the APPENDIX); they were all narrative in regard to interventions—ie, had no methods section or search strategy, whereas 1 review was systematic in regard to economy.10 Forty-six reviews (66%) contained references to a least 1 randomized trial (range 1–20, median 2, interquartile range 1–3), 4 of these to chemical interventions only. In total there were 162 trial references out of 777 possible (21%). The most quoted trial 9 had 22 citations.
Thirty-nine of the 46 reviews with trials (85%) also had references to non-randomized controlled studies that had been excluded from the Cochrane review. The authors rarely distinguished between the 2 designs but emphasized equally, or some times even more, the results obtained with nonrandomized studies. The most quoted nonrandomized study11 was referred to 25 times.
TABLE
Reviews and treatment recommendations
| INTERVENTIONS | POSITIVE RECOMMENDATIONS | NEUTRAL OR NEGATIVE RECOMMENDATIONS | NO RECOMMENDATIONS | |||
|---|---|---|---|---|---|---|
| Total | With trials | Total | With trials | Total | With trials | |
| Physical | 63 (90%) | 38 | 7 (10%) | 4 | 0 | 0 |
| Chemical | 30 (43%) | 23 | 25 (36%) | 16 | 15 (21%) | 7 |
| Distribution of the included reviews with their treatment recommendations. Of the 63 reviews with positive physical treatment recommendations, 38 had physical trial references and were included in the statistical analysis. | ||||||
Citations to trials in reviews
Since all reviews (TABLE) had an opinion on physical interventions, and since most trials were of physical interventions, we restricted the analysis to the 38 reviews with a positive recommendation for physical interventions and omitted 4 with a neutral or negative recommendation for physical interventions. We excluded 2 trial reports of physical interventions because they were published in Italian12 and Dutch13; none of the reviews quoted the Italian trial and 1 quoted the Dutch trial.
There was significant bias towards a positive selection of references in the reviews that recommended physical interventions; 10 reviews were neutral in this respect, whereas 27 reviews had a positive selection and 1 a negative selection (P=2 x 10–8). Conversely, the 4 reviews that did not recommend physical interventions all had a negative selection of references.
Discussion
Intervention recommendations in narrative reviews of house dust mites and asthma do not reflect the available, reliable evidence. Ninety percent of the reviews positively recommended physical interventions, although the Cochrane review of 28 trials on the topic6 failed to find an effect of physical and chemical interventions.
Cochrane reviews are also to some extent subjective, but trials published after the latest update of the Cochrane review on mites have provided support to its negative findings. Of note is a recently published trial of 732 patients who were allergic to mites.14 It was a double-blind, placebo-controlled trial conducted in Britain to investigate the effect of physical intervention measures (allergen-impermeable covers for mattress, pillow, and quilt). Although the interventions reduced mite allergen levels after 6 months, peak flow, medication use, and asthma symptoms were very similar in the 2 groups. During the next 6 months there was a planned reduction of inhaled corticosteroids, and also after this period, the outcomes were very similar—the morning peak flow was 431 L/min in both groups. The authors concluded that the intervention seemed clinically ineffective, in accordance with the findings of the Cochrane review.14
The narrative reviews were heavily biased by a selection of trial references that supported the opinions of the authors. When this was not the case, some authors—instead of ignoring negative results—used them as evidence that more rigorous intervention measures are needed for a positive intervention effect.
Physical interventions may need to be applied repeatedly before the reduction in allergen levels is sufficient to be effective. However, the lack of effect was also apparent in the subgroup of trials with long treatment duration or follow-up. And, if interventions were effective, one would expect to see at least some effect also in short-term trials.
Some of the authors of narrative reviews used long trial duration as a positive selection criterion for what they believed were relevant trials, but most often disregarded whether the trials were of acceptable quality in other respects. For example, the most popular randomized trial ran for a year9 and was quoted by 22 of the 35 reviews with trial references where it could have been quoted (63% citation rate). However, this trial had only 7 patients per group in the analysis, the outcome was not a clinical outcome but the result of a histamine tolerance test, and the part of the trial that yielded a significant result was not blinded. Furthermore, this result was probably not statistically significant. The authors claimed a significant result with nonparametric testing (P<.05), but when we repeated the test, using their raw data as depicted in a graph, we got P=.07; and in the comparison with the placebo group that was used in the Cochrane review of the trials,6 we got P=.15 for the geometric mean.
This trial9 is not only the most quoted in our sample of reviews, it is also the most quoted of our 28 included trials in the Science Citation Index as of September 17, 2003, where it had received 209 citations. We find it disturbing that 7 children in a nonblinded trial with no clinical outcomes have been so influential for the intervention recommendations over the past 10 years.
A survey of review articles on the relation between diet and heart disease3 showed that authors often used nonrandomized studies to create a false impression of consensus by referring to them together with the randomized trials if their results supported the authors’ opinion and ignoring them if they did not. We also found a high citation rate for nonrandomized studies. In fact, the most popular nonrandomized controlled study was only surpassed in citation rate by the small randomized trial just discussed. It was a physical intervention study,11 quoted by 25 of the 45 reviews with trial references where it could have been quoted (56% citation rate). This study was usually referred to without comment on its methodology. It had only 10 patients per group and its duration was only 1 month. The likely explanation for the high quotation rate for this study is that it had a very positive outcome for the intervention. It therefore seems that the reported results were more important than the study duration as a criterion for selection of studies for quotation.
The 24 reviews we excluded from statistical analysis because they had no references to randomized trials generally had strong intervention recommendations and argued that a reduced allergen level will lead to patient improvement. The authors often based this claim on uncontrolled studies in which the patients had been relocated to a dry, high-altitude location in the Alps with good clinical effect, but the authors failed to appreciate that the kind of allergen reduction achieved in these studies currently appears impossible to obtain in the patients’ homes.
Authors of review articles can be expected to stay abreast with new trials and to be familiar with systematic reviews within their field of expertise. However, the only systematic review6-8 in this area was quoted in just 6 of 12 reviews published at least 2 years later than the Cochrane review. Only 1 review15 did not recommend physical intervention because of the findings in the Cochrane review; another review16 disregarded the Cochrane review with the reasoning that few of the included trials obtained a reduction in allergen levels (although the Results section of the Cochrane review explained that the results were very similar in the subgroup of trials that reported successful mite reduction6).
Of the other 4 reviews that quoted the Cochrane review, the first17 mentioned it only in the introduction; the second18 claimed that the methods investigated were out of date; the third19 noted that an expert panel reviewing the same articles had come to a different conclusion than the Cochrane review; and the fourth20 stated that the complexity and heterogeneity of the interventions limited the utility of a simple statistical summary of the data as presented in the Cochrane review. Interestingly, all 4 reviews recommended physical intervention. It therefore seems that the assets of a systematic approach to the literature based on a predefined protocol, and applying rigorous methodology, was little understood.
We conclude that narrative review articles on house dust mites and asthma are severely biased in their selection of references and very unreliable in their intervention recommendations. This finding may be generalizable, since there is no reason to believe that physicians specializing in asthma should be more biased than others1-3 when they write reviews.
CORRESPONDING AUTHOR
Peter C. Gøtzsche, MD, DrMedSci, Nordic Cochrane Centre, Rigshospitalet, DK-2100 København Ø, Denmark. E-mail: [email protected].
1. Gøtzsche PC. Reference bias in reports of drug trials. Br Med J 1987;295:654-656.
2. Ravnskov U. Cholesterol lowering trials in coronary heart disease: frequency of citation and outcome. BMJ 1992;305:15-19.
3. Ravnskov U. Quotation bias in reviews of the diet-heart idea. J Clin Epidemiol 1995;48:713-719.
4. Jadad AR, Moher M, Browman GP, et al. Systematic reviews and meta-analyses on treatment of asthma: critical evaluation. BMJ 2000;320:537-540.
5. Clarke M, Oxman AD. eds Cochrane Reviewers’ Handbook 4.1.6 [updated January 2003]. Available at: www.cochrane.dk/cochrane/handbook/handbook.htm.
6. Gøtzsche PC, Johansen HK, Burr ML, Hammarquist C. House dust mite control measures for asthma (Cochrane Review). In:The Cochrane Library, issue 3. Oxford: Update Software; 2001.
7. Hammarquist C, Burr ML, Gøtzsche PC. House dust mite control measures for asthma (Cochrane review). In: The Cochrane Library, issue 3. Oxford: Update Software; 1998.
8. Gøtzsche PC, Hammarquist C, Burr M. House dust mite control measures in the management of asthma: meta-analysis. BMJ 1998;317:1105-1110.
9. Ehnert B, Lau-Schadendorf S, Weber A, Buettner P, Schou C, Wahn U. Reducing domestic exposure to dust mite allergen reduces bronchial hyperreactivity in sensitive children with asthma. J Allergy Clin Immunol 1992;90:135-138.
10. Mellis CM, Peat JK, Woolcock AJ. The cost of asthma: can it be reduced? Pharmacoeconomics 1993;3:205-219.
11. Murray AB, Ferguson AC. Dust-free bedrooms in the treatment of asthmatic children with house dust or house dust mite allergy: a controlled trial. Pediatrics 1983;71:418-422.
12. Cinti C, Canessa PA, Lavecchia MA, Capecchi V. The efficacy of ‘mite-proof’ mattress-covers and pillow-covers in the control of asthma in patients allergic to mites [in Italian]. Lotta Contro La Tuberculosi e Le Malattie Polmonari Sociali 1996;66:131-138.
13. Maesen FP, Sluysmans FG, Brombacher PJ, Smeets JJ. [Experience with the use of air filtration-apparatus in homes of atopic subjects allergic to house dust]. Acta Tuberc Pneumol Belg 1977;68:133-146.
14. Woodcock A, Forster L, et al. Control of exposure to mite allergen and allergen-impermeable bed covers for adults with asthma. N Engl J Med 2003;349:225-236.
15. Strachan DP. The role of environmental factors in asthma. Br Med Bull 2000;56:865-882.
16. Custovic A, Woodcock A. Clinical effects of allergen avoidance. Clin Rev Allergy Immunol 2000;18:397-419.
17. Vaughan JW, Platts-Mills TA. New approaches to environmental control. Clin Rev Allergy Immunol 2000;18:325-339.
18. Schnyder B, Schweri T, Thomann B, Pichler C. [Allergy to house dust mites]. Schweiz Med Wochenschr 2000;130:443-447.
19. de Blay F, Casel S, Colas F, Spirlet F, Pauli G. [Elimination of airborne allergens from the household environment]. Rev Mal Respir 2000;17:29-39.
20. Tovey E, Marks G. Methods and effectiveness of environmental control. J Allergy Clin Immunol 1999;103(2 Pt 1):179-191.
- When consulting narrative review articles, carefully consider the possibility of citation bias and whether recommendations are based on patient-oriented and not disease-oriented outcomes.
- Consult credible evidence-based resources such as Cochrane before making changes in practice suggested by narrative reviews.
Background: Citations in scientific articles may tend to favor the views presented. We studied whether there is such reference bias in narrative review articles that discuss interventions against house dust mites for people with asthma.
Design: Systematic review of reviews identified in a Medline search that expressed an opinion about the clinical effects of physical or chemical intervention methods.
Main outcome measure: Positive bias was judged to have occurred if the reference list contained a higher proportion of trial references with significant results than among all trials available to the authors (published 2 years or more prior to the review).
Results: Seventy reviews were included, of which 63 (90%) recommended physical interventions. Forty-six reviews had trial references, 4 of these only to chemical interventions. In the remaining 42 reviews, reference bias was detected (P=2 x 10-8). The most quoted trial had only 7 patients per group, its claimed significant result was probably erroneous, and it did not report a clinical outcome. Intervention recommendations were often based on nonrandomized evidence, and the most quoted nonrandomized controlled study had included only 10 patients per group but claimed very positive results.
Conclusion: The narrative review articles were severely biased, and their positive intervention recommendations are at variance with the systematic Cochrane Review on this topic and a recent very large trial of physical intervention, both of which failed to find an effect.
Trial reports and narrative review articles sometimes favor references supporting the views of the authors, lending credence to a particular treatment1 or hypothesis.2,3 This reference bias may render the conclusions of an article less reliable. Such was the finding in our study of narrative review articles discussing interventions against house dust mites for people with asthma.
Systematic reviews are a more reliable source of information for busy clinicians than narrative reviews, but they can also be problematic. An assessment of papers on asthma, based on a validated tool, showed that 40 of 50 systematic reviews on prevention and treatment had serious or extensive flaws.4 Cochrane systematic reviews on asthma were more rigorous and better reported than systematic reviews published in peer-reviewed, paper-based journals,4 probably because Cochrane Reviews are conducted according to a set of standard methods5 aimed at minimizing bias in the reviews, and because they usually include only randomized trials.
As the index review for our study, we chose the Cochrane Review on house dust mites control measures for asthma,6 to allow us to judge whether the recommendations in narrative reviews reflected fairly the available, reliable scientific evidence.
Methods
Inclusion criteria. Narrative reviews—that is, reviews that did not have a methods section with a search strategy for relevant research papers—were eligible if they expressed an opinion about the clinical effects of chemical or physical interventions aimed at reducing exposure to house dust mite antigens in the homes of mite-sensitive patients with asthma.
Search strategy. A Medline search selected review articles from 1966 to July 2002 using the terms review* and mite* and asthma*, with no language restrictions. Additional review articles could have been found in other databases, but we chose Medline since it is freely available and likely preferred by physicians.
Data collection. Abstracts identified in the Medline search were read by one of us (LMS), who excluded articles that clearly were not reviews or that reviewed aspects other than treatments (eg, causes of asthma only). Both authors read the remaining articles in full, and conducted another round of exclusion.
Data extraction was performed independently and results were compared. Disagreements were resolved by discussion; they were usually due to simple oversight, such as a missing trial reference. Interventions were classified as physical (eg, vacuum cleaning, washing, bed covers, heating, freezing), chemical (eg, acaricides), or combinations. The review-authors’ opinions were classified into 2 groups: positive, recommending 1 or more interventions (in a few cases rather vaguely, such as allergen avoidance), or neutral/negative (eg, more research is needed or the interventions should not be used).
The index review. The Cochrane review on house dust-mite control measures for asthma was first published in 1998 in the Cochrane Library8 and in BMJ.8 The latest update from 2001 included 28 randomized trials, one of which was 3-armed.9 When referring to randomized trials in our study, the sample is the 28 trial reports included in the Cochrane review. The Cochrane review, which included 939 patients in the analysis, failed to find an effect of chemical or physical methods on any of the outcomes (number of patients improved, asthma symptoms, medication usage, peak flow, FEV1, and bronchial hyperreactivity).
Data analysis. For reviews with positive intervention recommendations, the supporting references to randomized trials were compared with all those randomized trials that might potentially have been known to the authors—that is, published 2 years or more prior to the publication date of the review.1
If, for example, a review recommended acaricides and quoted 3 trials that claimed a significant effect of this intervention and 1 trial that did not, and if 8 trials were available at the time but only 3 favored acaricides, this would be a positively biased selection of references, since the proportion of positive trials in the reference list, 3/4, was greater than among all the available trials, 3/8.1
We did not judge whether the significance testing was correctly done but noted whether the trial authors had reported 1 or more significant results favoring the intervention. We then used a sign test, for positive vs negative selection, to determine whether there was significant reference bias.
Results
The Medline search identified 302 abstracts, of which 151 were clearly of no relevance and 5 were abstracts of the index Cochrane review. Of the remaining 146, 6 were excluded because of the language; 140 were read in full, and a further 70 were excluded for other reasons. This left 70 reviews that all had intervention recommendations (positive, neutral, or negative); 63 of them (90%) positively recommended physical interventions and 30 (43%) recommended chemical interventions (TABLE).
The reviews were published between 1971 and 2002 (see the APPENDIX); they were all narrative in regard to interventions—ie, had no methods section or search strategy, whereas 1 review was systematic in regard to economy.10 Forty-six reviews (66%) contained references to a least 1 randomized trial (range 1–20, median 2, interquartile range 1–3), 4 of these to chemical interventions only. In total there were 162 trial references out of 777 possible (21%). The most quoted trial 9 had 22 citations.
Thirty-nine of the 46 reviews with trials (85%) also had references to non-randomized controlled studies that had been excluded from the Cochrane review. The authors rarely distinguished between the 2 designs but emphasized equally, or some times even more, the results obtained with nonrandomized studies. The most quoted nonrandomized study11 was referred to 25 times.
TABLE
Reviews and treatment recommendations
| INTERVENTIONS | POSITIVE RECOMMENDATIONS | NEUTRAL OR NEGATIVE RECOMMENDATIONS | NO RECOMMENDATIONS | |||
|---|---|---|---|---|---|---|
| Total | With trials | Total | With trials | Total | With trials | |
| Physical | 63 (90%) | 38 | 7 (10%) | 4 | 0 | 0 |
| Chemical | 30 (43%) | 23 | 25 (36%) | 16 | 15 (21%) | 7 |
| Distribution of the included reviews with their treatment recommendations. Of the 63 reviews with positive physical treatment recommendations, 38 had physical trial references and were included in the statistical analysis. | ||||||
Citations to trials in reviews
Since all reviews (TABLE) had an opinion on physical interventions, and since most trials were of physical interventions, we restricted the analysis to the 38 reviews with a positive recommendation for physical interventions and omitted 4 with a neutral or negative recommendation for physical interventions. We excluded 2 trial reports of physical interventions because they were published in Italian12 and Dutch13; none of the reviews quoted the Italian trial and 1 quoted the Dutch trial.
There was significant bias towards a positive selection of references in the reviews that recommended physical interventions; 10 reviews were neutral in this respect, whereas 27 reviews had a positive selection and 1 a negative selection (P=2 x 10–8). Conversely, the 4 reviews that did not recommend physical interventions all had a negative selection of references.
Discussion
Intervention recommendations in narrative reviews of house dust mites and asthma do not reflect the available, reliable evidence. Ninety percent of the reviews positively recommended physical interventions, although the Cochrane review of 28 trials on the topic6 failed to find an effect of physical and chemical interventions.
Cochrane reviews are also to some extent subjective, but trials published after the latest update of the Cochrane review on mites have provided support to its negative findings. Of note is a recently published trial of 732 patients who were allergic to mites.14 It was a double-blind, placebo-controlled trial conducted in Britain to investigate the effect of physical intervention measures (allergen-impermeable covers for mattress, pillow, and quilt). Although the interventions reduced mite allergen levels after 6 months, peak flow, medication use, and asthma symptoms were very similar in the 2 groups. During the next 6 months there was a planned reduction of inhaled corticosteroids, and also after this period, the outcomes were very similar—the morning peak flow was 431 L/min in both groups. The authors concluded that the intervention seemed clinically ineffective, in accordance with the findings of the Cochrane review.14
The narrative reviews were heavily biased by a selection of trial references that supported the opinions of the authors. When this was not the case, some authors—instead of ignoring negative results—used them as evidence that more rigorous intervention measures are needed for a positive intervention effect.
Physical interventions may need to be applied repeatedly before the reduction in allergen levels is sufficient to be effective. However, the lack of effect was also apparent in the subgroup of trials with long treatment duration or follow-up. And, if interventions were effective, one would expect to see at least some effect also in short-term trials.
Some of the authors of narrative reviews used long trial duration as a positive selection criterion for what they believed were relevant trials, but most often disregarded whether the trials were of acceptable quality in other respects. For example, the most popular randomized trial ran for a year9 and was quoted by 22 of the 35 reviews with trial references where it could have been quoted (63% citation rate). However, this trial had only 7 patients per group in the analysis, the outcome was not a clinical outcome but the result of a histamine tolerance test, and the part of the trial that yielded a significant result was not blinded. Furthermore, this result was probably not statistically significant. The authors claimed a significant result with nonparametric testing (P<.05), but when we repeated the test, using their raw data as depicted in a graph, we got P=.07; and in the comparison with the placebo group that was used in the Cochrane review of the trials,6 we got P=.15 for the geometric mean.
This trial9 is not only the most quoted in our sample of reviews, it is also the most quoted of our 28 included trials in the Science Citation Index as of September 17, 2003, where it had received 209 citations. We find it disturbing that 7 children in a nonblinded trial with no clinical outcomes have been so influential for the intervention recommendations over the past 10 years.
A survey of review articles on the relation between diet and heart disease3 showed that authors often used nonrandomized studies to create a false impression of consensus by referring to them together with the randomized trials if their results supported the authors’ opinion and ignoring them if they did not. We also found a high citation rate for nonrandomized studies. In fact, the most popular nonrandomized controlled study was only surpassed in citation rate by the small randomized trial just discussed. It was a physical intervention study,11 quoted by 25 of the 45 reviews with trial references where it could have been quoted (56% citation rate). This study was usually referred to without comment on its methodology. It had only 10 patients per group and its duration was only 1 month. The likely explanation for the high quotation rate for this study is that it had a very positive outcome for the intervention. It therefore seems that the reported results were more important than the study duration as a criterion for selection of studies for quotation.
The 24 reviews we excluded from statistical analysis because they had no references to randomized trials generally had strong intervention recommendations and argued that a reduced allergen level will lead to patient improvement. The authors often based this claim on uncontrolled studies in which the patients had been relocated to a dry, high-altitude location in the Alps with good clinical effect, but the authors failed to appreciate that the kind of allergen reduction achieved in these studies currently appears impossible to obtain in the patients’ homes.
Authors of review articles can be expected to stay abreast with new trials and to be familiar with systematic reviews within their field of expertise. However, the only systematic review6-8 in this area was quoted in just 6 of 12 reviews published at least 2 years later than the Cochrane review. Only 1 review15 did not recommend physical intervention because of the findings in the Cochrane review; another review16 disregarded the Cochrane review with the reasoning that few of the included trials obtained a reduction in allergen levels (although the Results section of the Cochrane review explained that the results were very similar in the subgroup of trials that reported successful mite reduction6).
Of the other 4 reviews that quoted the Cochrane review, the first17 mentioned it only in the introduction; the second18 claimed that the methods investigated were out of date; the third19 noted that an expert panel reviewing the same articles had come to a different conclusion than the Cochrane review; and the fourth20 stated that the complexity and heterogeneity of the interventions limited the utility of a simple statistical summary of the data as presented in the Cochrane review. Interestingly, all 4 reviews recommended physical intervention. It therefore seems that the assets of a systematic approach to the literature based on a predefined protocol, and applying rigorous methodology, was little understood.
We conclude that narrative review articles on house dust mites and asthma are severely biased in their selection of references and very unreliable in their intervention recommendations. This finding may be generalizable, since there is no reason to believe that physicians specializing in asthma should be more biased than others1-3 when they write reviews.
CORRESPONDING AUTHOR
Peter C. Gøtzsche, MD, DrMedSci, Nordic Cochrane Centre, Rigshospitalet, DK-2100 København Ø, Denmark. E-mail: [email protected].
- When consulting narrative review articles, carefully consider the possibility of citation bias and whether recommendations are based on patient-oriented and not disease-oriented outcomes.
- Consult credible evidence-based resources such as Cochrane before making changes in practice suggested by narrative reviews.
Background: Citations in scientific articles may tend to favor the views presented. We studied whether there is such reference bias in narrative review articles that discuss interventions against house dust mites for people with asthma.
Design: Systematic review of reviews identified in a Medline search that expressed an opinion about the clinical effects of physical or chemical intervention methods.
Main outcome measure: Positive bias was judged to have occurred if the reference list contained a higher proportion of trial references with significant results than among all trials available to the authors (published 2 years or more prior to the review).
Results: Seventy reviews were included, of which 63 (90%) recommended physical interventions. Forty-six reviews had trial references, 4 of these only to chemical interventions. In the remaining 42 reviews, reference bias was detected (P=2 x 10-8). The most quoted trial had only 7 patients per group, its claimed significant result was probably erroneous, and it did not report a clinical outcome. Intervention recommendations were often based on nonrandomized evidence, and the most quoted nonrandomized controlled study had included only 10 patients per group but claimed very positive results.
Conclusion: The narrative review articles were severely biased, and their positive intervention recommendations are at variance with the systematic Cochrane Review on this topic and a recent very large trial of physical intervention, both of which failed to find an effect.
Trial reports and narrative review articles sometimes favor references supporting the views of the authors, lending credence to a particular treatment1 or hypothesis.2,3 This reference bias may render the conclusions of an article less reliable. Such was the finding in our study of narrative review articles discussing interventions against house dust mites for people with asthma.
Systematic reviews are a more reliable source of information for busy clinicians than narrative reviews, but they can also be problematic. An assessment of papers on asthma, based on a validated tool, showed that 40 of 50 systematic reviews on prevention and treatment had serious or extensive flaws.4 Cochrane systematic reviews on asthma were more rigorous and better reported than systematic reviews published in peer-reviewed, paper-based journals,4 probably because Cochrane Reviews are conducted according to a set of standard methods5 aimed at minimizing bias in the reviews, and because they usually include only randomized trials.
As the index review for our study, we chose the Cochrane Review on house dust mites control measures for asthma,6 to allow us to judge whether the recommendations in narrative reviews reflected fairly the available, reliable scientific evidence.
Methods
Inclusion criteria. Narrative reviews—that is, reviews that did not have a methods section with a search strategy for relevant research papers—were eligible if they expressed an opinion about the clinical effects of chemical or physical interventions aimed at reducing exposure to house dust mite antigens in the homes of mite-sensitive patients with asthma.
Search strategy. A Medline search selected review articles from 1966 to July 2002 using the terms review* and mite* and asthma*, with no language restrictions. Additional review articles could have been found in other databases, but we chose Medline since it is freely available and likely preferred by physicians.
Data collection. Abstracts identified in the Medline search were read by one of us (LMS), who excluded articles that clearly were not reviews or that reviewed aspects other than treatments (eg, causes of asthma only). Both authors read the remaining articles in full, and conducted another round of exclusion.
Data extraction was performed independently and results were compared. Disagreements were resolved by discussion; they were usually due to simple oversight, such as a missing trial reference. Interventions were classified as physical (eg, vacuum cleaning, washing, bed covers, heating, freezing), chemical (eg, acaricides), or combinations. The review-authors’ opinions were classified into 2 groups: positive, recommending 1 or more interventions (in a few cases rather vaguely, such as allergen avoidance), or neutral/negative (eg, more research is needed or the interventions should not be used).
The index review. The Cochrane review on house dust-mite control measures for asthma was first published in 1998 in the Cochrane Library8 and in BMJ.8 The latest update from 2001 included 28 randomized trials, one of which was 3-armed.9 When referring to randomized trials in our study, the sample is the 28 trial reports included in the Cochrane review. The Cochrane review, which included 939 patients in the analysis, failed to find an effect of chemical or physical methods on any of the outcomes (number of patients improved, asthma symptoms, medication usage, peak flow, FEV1, and bronchial hyperreactivity).
Data analysis. For reviews with positive intervention recommendations, the supporting references to randomized trials were compared with all those randomized trials that might potentially have been known to the authors—that is, published 2 years or more prior to the publication date of the review.1
If, for example, a review recommended acaricides and quoted 3 trials that claimed a significant effect of this intervention and 1 trial that did not, and if 8 trials were available at the time but only 3 favored acaricides, this would be a positively biased selection of references, since the proportion of positive trials in the reference list, 3/4, was greater than among all the available trials, 3/8.1
We did not judge whether the significance testing was correctly done but noted whether the trial authors had reported 1 or more significant results favoring the intervention. We then used a sign test, for positive vs negative selection, to determine whether there was significant reference bias.
Results
The Medline search identified 302 abstracts, of which 151 were clearly of no relevance and 5 were abstracts of the index Cochrane review. Of the remaining 146, 6 were excluded because of the language; 140 were read in full, and a further 70 were excluded for other reasons. This left 70 reviews that all had intervention recommendations (positive, neutral, or negative); 63 of them (90%) positively recommended physical interventions and 30 (43%) recommended chemical interventions (TABLE).
The reviews were published between 1971 and 2002 (see the APPENDIX); they were all narrative in regard to interventions—ie, had no methods section or search strategy, whereas 1 review was systematic in regard to economy.10 Forty-six reviews (66%) contained references to a least 1 randomized trial (range 1–20, median 2, interquartile range 1–3), 4 of these to chemical interventions only. In total there were 162 trial references out of 777 possible (21%). The most quoted trial 9 had 22 citations.
Thirty-nine of the 46 reviews with trials (85%) also had references to non-randomized controlled studies that had been excluded from the Cochrane review. The authors rarely distinguished between the 2 designs but emphasized equally, or some times even more, the results obtained with nonrandomized studies. The most quoted nonrandomized study11 was referred to 25 times.
TABLE
Reviews and treatment recommendations
| INTERVENTIONS | POSITIVE RECOMMENDATIONS | NEUTRAL OR NEGATIVE RECOMMENDATIONS | NO RECOMMENDATIONS | |||
|---|---|---|---|---|---|---|
| Total | With trials | Total | With trials | Total | With trials | |
| Physical | 63 (90%) | 38 | 7 (10%) | 4 | 0 | 0 |
| Chemical | 30 (43%) | 23 | 25 (36%) | 16 | 15 (21%) | 7 |
| Distribution of the included reviews with their treatment recommendations. Of the 63 reviews with positive physical treatment recommendations, 38 had physical trial references and were included in the statistical analysis. | ||||||
Citations to trials in reviews
Since all reviews (TABLE) had an opinion on physical interventions, and since most trials were of physical interventions, we restricted the analysis to the 38 reviews with a positive recommendation for physical interventions and omitted 4 with a neutral or negative recommendation for physical interventions. We excluded 2 trial reports of physical interventions because they were published in Italian12 and Dutch13; none of the reviews quoted the Italian trial and 1 quoted the Dutch trial.
There was significant bias towards a positive selection of references in the reviews that recommended physical interventions; 10 reviews were neutral in this respect, whereas 27 reviews had a positive selection and 1 a negative selection (P=2 x 10–8). Conversely, the 4 reviews that did not recommend physical interventions all had a negative selection of references.
Discussion
Intervention recommendations in narrative reviews of house dust mites and asthma do not reflect the available, reliable evidence. Ninety percent of the reviews positively recommended physical interventions, although the Cochrane review of 28 trials on the topic6 failed to find an effect of physical and chemical interventions.
Cochrane reviews are also to some extent subjective, but trials published after the latest update of the Cochrane review on mites have provided support to its negative findings. Of note is a recently published trial of 732 patients who were allergic to mites.14 It was a double-blind, placebo-controlled trial conducted in Britain to investigate the effect of physical intervention measures (allergen-impermeable covers for mattress, pillow, and quilt). Although the interventions reduced mite allergen levels after 6 months, peak flow, medication use, and asthma symptoms were very similar in the 2 groups. During the next 6 months there was a planned reduction of inhaled corticosteroids, and also after this period, the outcomes were very similar—the morning peak flow was 431 L/min in both groups. The authors concluded that the intervention seemed clinically ineffective, in accordance with the findings of the Cochrane review.14
The narrative reviews were heavily biased by a selection of trial references that supported the opinions of the authors. When this was not the case, some authors—instead of ignoring negative results—used them as evidence that more rigorous intervention measures are needed for a positive intervention effect.
Physical interventions may need to be applied repeatedly before the reduction in allergen levels is sufficient to be effective. However, the lack of effect was also apparent in the subgroup of trials with long treatment duration or follow-up. And, if interventions were effective, one would expect to see at least some effect also in short-term trials.
Some of the authors of narrative reviews used long trial duration as a positive selection criterion for what they believed were relevant trials, but most often disregarded whether the trials were of acceptable quality in other respects. For example, the most popular randomized trial ran for a year9 and was quoted by 22 of the 35 reviews with trial references where it could have been quoted (63% citation rate). However, this trial had only 7 patients per group in the analysis, the outcome was not a clinical outcome but the result of a histamine tolerance test, and the part of the trial that yielded a significant result was not blinded. Furthermore, this result was probably not statistically significant. The authors claimed a significant result with nonparametric testing (P<.05), but when we repeated the test, using their raw data as depicted in a graph, we got P=.07; and in the comparison with the placebo group that was used in the Cochrane review of the trials,6 we got P=.15 for the geometric mean.
This trial9 is not only the most quoted in our sample of reviews, it is also the most quoted of our 28 included trials in the Science Citation Index as of September 17, 2003, where it had received 209 citations. We find it disturbing that 7 children in a nonblinded trial with no clinical outcomes have been so influential for the intervention recommendations over the past 10 years.
A survey of review articles on the relation between diet and heart disease3 showed that authors often used nonrandomized studies to create a false impression of consensus by referring to them together with the randomized trials if their results supported the authors’ opinion and ignoring them if they did not. We also found a high citation rate for nonrandomized studies. In fact, the most popular nonrandomized controlled study was only surpassed in citation rate by the small randomized trial just discussed. It was a physical intervention study,11 quoted by 25 of the 45 reviews with trial references where it could have been quoted (56% citation rate). This study was usually referred to without comment on its methodology. It had only 10 patients per group and its duration was only 1 month. The likely explanation for the high quotation rate for this study is that it had a very positive outcome for the intervention. It therefore seems that the reported results were more important than the study duration as a criterion for selection of studies for quotation.
The 24 reviews we excluded from statistical analysis because they had no references to randomized trials generally had strong intervention recommendations and argued that a reduced allergen level will lead to patient improvement. The authors often based this claim on uncontrolled studies in which the patients had been relocated to a dry, high-altitude location in the Alps with good clinical effect, but the authors failed to appreciate that the kind of allergen reduction achieved in these studies currently appears impossible to obtain in the patients’ homes.
Authors of review articles can be expected to stay abreast with new trials and to be familiar with systematic reviews within their field of expertise. However, the only systematic review6-8 in this area was quoted in just 6 of 12 reviews published at least 2 years later than the Cochrane review. Only 1 review15 did not recommend physical intervention because of the findings in the Cochrane review; another review16 disregarded the Cochrane review with the reasoning that few of the included trials obtained a reduction in allergen levels (although the Results section of the Cochrane review explained that the results were very similar in the subgroup of trials that reported successful mite reduction6).
Of the other 4 reviews that quoted the Cochrane review, the first17 mentioned it only in the introduction; the second18 claimed that the methods investigated were out of date; the third19 noted that an expert panel reviewing the same articles had come to a different conclusion than the Cochrane review; and the fourth20 stated that the complexity and heterogeneity of the interventions limited the utility of a simple statistical summary of the data as presented in the Cochrane review. Interestingly, all 4 reviews recommended physical intervention. It therefore seems that the assets of a systematic approach to the literature based on a predefined protocol, and applying rigorous methodology, was little understood.
We conclude that narrative review articles on house dust mites and asthma are severely biased in their selection of references and very unreliable in their intervention recommendations. This finding may be generalizable, since there is no reason to believe that physicians specializing in asthma should be more biased than others1-3 when they write reviews.
CORRESPONDING AUTHOR
Peter C. Gøtzsche, MD, DrMedSci, Nordic Cochrane Centre, Rigshospitalet, DK-2100 København Ø, Denmark. E-mail: [email protected].
1. Gøtzsche PC. Reference bias in reports of drug trials. Br Med J 1987;295:654-656.
2. Ravnskov U. Cholesterol lowering trials in coronary heart disease: frequency of citation and outcome. BMJ 1992;305:15-19.
3. Ravnskov U. Quotation bias in reviews of the diet-heart idea. J Clin Epidemiol 1995;48:713-719.
4. Jadad AR, Moher M, Browman GP, et al. Systematic reviews and meta-analyses on treatment of asthma: critical evaluation. BMJ 2000;320:537-540.
5. Clarke M, Oxman AD. eds Cochrane Reviewers’ Handbook 4.1.6 [updated January 2003]. Available at: www.cochrane.dk/cochrane/handbook/handbook.htm.
6. Gøtzsche PC, Johansen HK, Burr ML, Hammarquist C. House dust mite control measures for asthma (Cochrane Review). In:The Cochrane Library, issue 3. Oxford: Update Software; 2001.
7. Hammarquist C, Burr ML, Gøtzsche PC. House dust mite control measures for asthma (Cochrane review). In: The Cochrane Library, issue 3. Oxford: Update Software; 1998.
8. Gøtzsche PC, Hammarquist C, Burr M. House dust mite control measures in the management of asthma: meta-analysis. BMJ 1998;317:1105-1110.
9. Ehnert B, Lau-Schadendorf S, Weber A, Buettner P, Schou C, Wahn U. Reducing domestic exposure to dust mite allergen reduces bronchial hyperreactivity in sensitive children with asthma. J Allergy Clin Immunol 1992;90:135-138.
10. Mellis CM, Peat JK, Woolcock AJ. The cost of asthma: can it be reduced? Pharmacoeconomics 1993;3:205-219.
11. Murray AB, Ferguson AC. Dust-free bedrooms in the treatment of asthmatic children with house dust or house dust mite allergy: a controlled trial. Pediatrics 1983;71:418-422.
12. Cinti C, Canessa PA, Lavecchia MA, Capecchi V. The efficacy of ‘mite-proof’ mattress-covers and pillow-covers in the control of asthma in patients allergic to mites [in Italian]. Lotta Contro La Tuberculosi e Le Malattie Polmonari Sociali 1996;66:131-138.
13. Maesen FP, Sluysmans FG, Brombacher PJ, Smeets JJ. [Experience with the use of air filtration-apparatus in homes of atopic subjects allergic to house dust]. Acta Tuberc Pneumol Belg 1977;68:133-146.
14. Woodcock A, Forster L, et al. Control of exposure to mite allergen and allergen-impermeable bed covers for adults with asthma. N Engl J Med 2003;349:225-236.
15. Strachan DP. The role of environmental factors in asthma. Br Med Bull 2000;56:865-882.
16. Custovic A, Woodcock A. Clinical effects of allergen avoidance. Clin Rev Allergy Immunol 2000;18:397-419.
17. Vaughan JW, Platts-Mills TA. New approaches to environmental control. Clin Rev Allergy Immunol 2000;18:325-339.
18. Schnyder B, Schweri T, Thomann B, Pichler C. [Allergy to house dust mites]. Schweiz Med Wochenschr 2000;130:443-447.
19. de Blay F, Casel S, Colas F, Spirlet F, Pauli G. [Elimination of airborne allergens from the household environment]. Rev Mal Respir 2000;17:29-39.
20. Tovey E, Marks G. Methods and effectiveness of environmental control. J Allergy Clin Immunol 1999;103(2 Pt 1):179-191.
1. Gøtzsche PC. Reference bias in reports of drug trials. Br Med J 1987;295:654-656.
2. Ravnskov U. Cholesterol lowering trials in coronary heart disease: frequency of citation and outcome. BMJ 1992;305:15-19.
3. Ravnskov U. Quotation bias in reviews of the diet-heart idea. J Clin Epidemiol 1995;48:713-719.
4. Jadad AR, Moher M, Browman GP, et al. Systematic reviews and meta-analyses on treatment of asthma: critical evaluation. BMJ 2000;320:537-540.
5. Clarke M, Oxman AD. eds Cochrane Reviewers’ Handbook 4.1.6 [updated January 2003]. Available at: www.cochrane.dk/cochrane/handbook/handbook.htm.
6. Gøtzsche PC, Johansen HK, Burr ML, Hammarquist C. House dust mite control measures for asthma (Cochrane Review). In:The Cochrane Library, issue 3. Oxford: Update Software; 2001.
7. Hammarquist C, Burr ML, Gøtzsche PC. House dust mite control measures for asthma (Cochrane review). In: The Cochrane Library, issue 3. Oxford: Update Software; 1998.
8. Gøtzsche PC, Hammarquist C, Burr M. House dust mite control measures in the management of asthma: meta-analysis. BMJ 1998;317:1105-1110.
9. Ehnert B, Lau-Schadendorf S, Weber A, Buettner P, Schou C, Wahn U. Reducing domestic exposure to dust mite allergen reduces bronchial hyperreactivity in sensitive children with asthma. J Allergy Clin Immunol 1992;90:135-138.
10. Mellis CM, Peat JK, Woolcock AJ. The cost of asthma: can it be reduced? Pharmacoeconomics 1993;3:205-219.
11. Murray AB, Ferguson AC. Dust-free bedrooms in the treatment of asthmatic children with house dust or house dust mite allergy: a controlled trial. Pediatrics 1983;71:418-422.
12. Cinti C, Canessa PA, Lavecchia MA, Capecchi V. The efficacy of ‘mite-proof’ mattress-covers and pillow-covers in the control of asthma in patients allergic to mites [in Italian]. Lotta Contro La Tuberculosi e Le Malattie Polmonari Sociali 1996;66:131-138.
13. Maesen FP, Sluysmans FG, Brombacher PJ, Smeets JJ. [Experience with the use of air filtration-apparatus in homes of atopic subjects allergic to house dust]. Acta Tuberc Pneumol Belg 1977;68:133-146.
14. Woodcock A, Forster L, et al. Control of exposure to mite allergen and allergen-impermeable bed covers for adults with asthma. N Engl J Med 2003;349:225-236.
15. Strachan DP. The role of environmental factors in asthma. Br Med Bull 2000;56:865-882.
16. Custovic A, Woodcock A. Clinical effects of allergen avoidance. Clin Rev Allergy Immunol 2000;18:397-419.
17. Vaughan JW, Platts-Mills TA. New approaches to environmental control. Clin Rev Allergy Immunol 2000;18:325-339.
18. Schnyder B, Schweri T, Thomann B, Pichler C. [Allergy to house dust mites]. Schweiz Med Wochenschr 2000;130:443-447.
19. de Blay F, Casel S, Colas F, Spirlet F, Pauli G. [Elimination of airborne allergens from the household environment]. Rev Mal Respir 2000;17:29-39.
20. Tovey E, Marks G. Methods and effectiveness of environmental control. J Allergy Clin Immunol 1999;103(2 Pt 1):179-191.