Do drug treatment POEMs report data in clinically useful ways?

Medical professionals are inundated by new information, which some have described as an “information jungle.”¹ Thousands of articles are published each year in hundreds of journals,² adding to an ever-expanding knowledge base. One study suggests that the experienced primary care physician uses up to 2 million pieces of information each year to manage patients.³ To provide appropriate patient care, physicians must stay abreast of current medical knowledge.⁴ However, busy clinicians have little time to navigate the information jungle and sift through all of the data to determine what is relevant and clinically useful.

More than a decade has passed since Allen Shaughnessy and David Slawson developed the concept of Patient-Oriented Evidence that Matters (POEMs), “a summary of a valid piece of research that carries information that is important to patients and so to their doctors.”⁵ They developed a formula to classify research as a POEM: U=R×V/W, where U=usefulness of the information to doctors, R=relevance of the information to doctors, V=validity of the information, and W=work to access the information.¹ The most useful information is both relevant and valid and takes little work to access. In 2002, the British Medical Journal (BMJ) proposed publishing one POEM a week based on the following criteria:

• It addresses a question that doctors encounter.
• It measures outcomes that doctors and their patients care about: symptoms, morbidity, quality of life, and mortality.
• It has the potential to change the way doctors practice.⁵

Even with the advent of POEMs, the true benefit of research and its application to clinical practice has yet to be determined. Physicians still have to decide which studies are valid, interpret the outcomes, and determine how they affect individual practice.

Research shows that clinicians, patients, and policy makers are more impressed by larger percentage differences than smaller ones.⁶ This fact is evident in the way trial results are presented in the news, by pharmaceutical representatives, and in journal articles. The relative risk reduction (RRR) is touted as suggesting either benefit or reduced harm, and the absolute numbers are largely under-reported. One study found that treatment effectiveness was perceived to be lower when the absolute risk reduction (ARR) rather than the RRR was reported.⁷ Perception of effectiveness decreased further when the number needed to treat (NNT) was presented.

The authors of the study concluded that ARR and NNT provide more concrete information than RRR about an intervention because they express efficacy “in a way which incorporates both the magnitude of the reduction of risk and the baseline risk without treatment.” They note that “because the exclusive reporting of relative risk may overstate the effectiveness of a treatment, actual event rates and absolute changes in risk should be reported.”⁷

These numbers are rarely found in journal articles and, when present, rarely appear in abstracts, tables, or graphs, where the busy clinician looks to find information quickly. Our study sought to estimate the percentage of drug treatment articles published in major medical journals that provide a calculated ARR, NNT, or number needed to harm (NNH), as demonstrated in TABLE 1.

TABLE 1
How to calculate RRR, ARR, and NNT

Methods

Pilot study
We first performed a pilot study that retrospectively reviewed all drug therapy articles published in the Journal of the American Medical Association (JAMA) from April 1, 2008, through April 1, 2009. Its purpose was to ensure concurrence in data gathering, rule out any measurement bias, and refine the analysis tool.

We applied an algorithmic approach to the review and used an Excel spreadsheet as a record-keeping tool, giving each article an abbreviated name and recording the issue, year, volume, and page numbers. We excluded case reports, review articles that were not systematic reviews or meta-analyses, letters, and editorials. We also excluded articles on cancer chemotherapy because, although family physicians need to have a working knowledge of antineoplastic drugs, they do not routinely prescribe them. Moreover, family physicians rarely write the first prescription for such a drug.

We reviewed the drug treatment articles to determine relevance—that is, whether they met POEMs criteria: addressed a question that most family medicine doctors encounter in a typical 6-month period, measured an outcome that family physicians and patients care about—such as morbidity, mortality, quality of life, or effect on clinical events—and had the potential to change clinical practice.⁸ Articles that met all 3 criteria were included in our analysis; articles that did not were recorded but excluded from further examination.

We analyzed articles that met relevance criteria for validity and clinical usefulness. We assessed validity based on whether the article was a randomized, controlled, double-blinded trial and whether allocation was concealed, follow-up was complete, information was analyzed on an intention-to-treat basis, and the results were statistically significant. We assessed clinical usefulness by recording whether the study reported RRR, ARR, NNT, or NNH, and if so, whether it recorded the information in free text, including the abstract, or in a graph or chart.

Review of articles in 7 journals
After the pilot study, we reviewed articles in JAMA and 6 other journals during a 6-month period from April 1, 2008 through September 30, 2008. We applied the same algorithmic analysis as in the pilot study to drug therapy articles in Archives of Internal Medicine (Arch Intern Med), BMJ, New England Journal of Medicine (N Engl J Med), Lancet, Obstetrics and Gynecology (Obstet Gynecol), and Pediatrics. We met regularly and settled disagreements about relevance, validity, or clinical usefulness by re-reviewing the article.

We recorded the total number of drug therapy articles for each journal, then tallied the number of POEMs and the total number of calculations of RRR, ARR, NNT, and NNH in articles that met POEMs criteria. We used these numbers to determine the percentage of POEMs and POEMs with clinically useful information for each journal.

Results

We identified a total of 995 articles in the 7 journals during the 6-month study period. Of these, 24 (2.4%) were classified as drug therapy POEMs and 6 (0.6%) were POEMs with clinically useful information.

The journals that published the most POEMs were Arch Intern Med, N Engl J Med, BMJ, and JAMA. Arch Intern Med had the highest percentage (33%), followed by JAMA (20%), and BMJ (17%) (TABLE 2).

When we analyzed the POEMs for clinical usefulness based on whether they provided calculated ARR, NNT, or NNH, only 3 journals published POEMs with clinically useful information: JAMA published the most (15%), followed by BMJ (9%) and Obstet Gynecol (4%) (FIGURE).

FIGURE
Percentage of POEMs with clinically useful information*

TABLE 2
Few drug therapy articles are POEMs

Discussion

Our study findings are consistent with the relevance data from a previous study published in 1999.⁸ After more than a decade, medical journals still are not publishing drug therapy POEMs.

A disturbing scarcity of useful information. The paucity of drug therapy POEMs with clinically useful information is alarming. Based on our data we estimate that a physician would have to read on average 36 drug therapy articles to find one clinically helpful drug therapy POEM. This finding suggests that the medical literature is not helping clinicians provide the best patient care or, when it does, the busy clinician is forced to spend what little time is available in calculations to determine what can actually affect practice in positive ways.

Study limitations. Our study has a number of limitations. The study settings in the articles we reviewed ultimately determined what information was important and could potentially change clinical practice. Some studies, for example, were performed in developing countries, where the therapy being tested was not commonly used and would alter practice. In the United States, however, the same treatment would not affect clinical practice because it was either common practice or standard of care.

We reviewed only 7 major medical journals. Our results cannot necessarily be extrapolated to other major journals, although they do suggest that the findings are not limited to a few publications. Moreover, we reviewed only 2 specialty journals. It is possible that other such journals are publishing more POEMs than we observed and providing more concrete numbers that specialists can use to quickly and easily adjust their practice patterns than general journals.

We did not analyze any family medicine journals for the following reasons: American Family Physician publishes only review articles; The Journal of Family Practice does not routinely publish original research; and Annals of Family Medicine (the research journal of the discipline) is less well established than the other journals we selected for this review, having been launched in 2003.

In addition, 6 months may not have been long enough to accurately calculate the percentage of POEMs or clinically useful information in the journals we reviewed. During the pilot study, in which we analyzed a full year of JAMA, only one POEM was published in the second 6 months, and it did not contain clinically useful information. Moreover, we reviewed most journals over 6 consecutive months rather than 6 randomly chosen months.

Finally, we reviewed only drug therapy articles. Future studies could examine surgical, diagnostic, or prognostic studies.

Toward more, and more useful, POEMs
Despite the scarcity of POEMs that provide clinically useful information in major medical journals, it is important that physicians continue to practice evidence-based medicine, sifting through the available information and even calculating ARR, NNT, and NNH themselves, which most busy clinicians do not have the time or inclination to do.

How can we improve the clinical usefulness of published data? One way is for journal editors to require that authors provide ARRs and NNTs or NNHs. Another is for authors to include these calculations on their own initiative. Either way, the goal is better clinical practice and optimal patient care.

CORRESPONDENCE Adrienne Z. Ables, PharmD, Edward Via College of Osteopathic Medicine, Carolinas Campus, 350 Howard Street, Spartanburg, SC 29303; [email protected]

Medical professionals are inundated by new information, which some have described as an “information jungle.”¹ Thousands of articles are published each year in hundreds of journals,² adding to an ever-expanding knowledge base. One study suggests that the experienced primary care physician uses up to 2 million pieces of information each year to manage patients.³ To provide appropriate patient care, physicians must stay abreast of current medical knowledge.⁴ However, busy clinicians have little time to navigate the information jungle and sift through all of the data to determine what is relevant and clinically useful.

More than a decade has passed since Allen Shaughnessy and David Slawson developed the concept of Patient-Oriented Evidence that Matters (POEMs), “a summary of a valid piece of research that carries information that is important to patients and so to their doctors.”⁵ They developed a formula to classify research as a POEM: U=R×V/W, where U=usefulness of the information to doctors, R=relevance of the information to doctors, V=validity of the information, and W=work to access the information.¹ The most useful information is both relevant and valid and takes little work to access. In 2002, the British Medical Journal (BMJ) proposed publishing one POEM a week based on the following criteria:

• It addresses a question that doctors encounter.
• It measures outcomes that doctors and their patients care about: symptoms, morbidity, quality of life, and mortality.
• It has the potential to change the way doctors practice.⁵

Even with the advent of POEMs, the true benefit of research and its application to clinical practice has yet to be determined. Physicians still have to decide which studies are valid, interpret the outcomes, and determine how they affect individual practice.

Research shows that clinicians, patients, and policy makers are more impressed by larger percentage differences than smaller ones.⁶ This fact is evident in the way trial results are presented in the news, by pharmaceutical representatives, and in journal articles. The relative risk reduction (RRR) is touted as suggesting either benefit or reduced harm, and the absolute numbers are largely under-reported. One study found that treatment effectiveness was perceived to be lower when the absolute risk reduction (ARR) rather than the RRR was reported.⁷ Perception of effectiveness decreased further when the number needed to treat (NNT) was presented.

The authors of the study concluded that ARR and NNT provide more concrete information than RRR about an intervention because they express efficacy “in a way which incorporates both the magnitude of the reduction of risk and the baseline risk without treatment.” They note that “because the exclusive reporting of relative risk may overstate the effectiveness of a treatment, actual event rates and absolute changes in risk should be reported.”⁷

These numbers are rarely found in journal articles and, when present, rarely appear in abstracts, tables, or graphs, where the busy clinician looks to find information quickly. Our study sought to estimate the percentage of drug treatment articles published in major medical journals that provide a calculated ARR, NNT, or number needed to harm (NNH), as demonstrated in TABLE 1.

TABLE 1
How to calculate RRR, ARR, and NNT

Methods

Pilot study
We first performed a pilot study that retrospectively reviewed all drug therapy articles published in the Journal of the American Medical Association (JAMA) from April 1, 2008, through April 1, 2009. Its purpose was to ensure concurrence in data gathering, rule out any measurement bias, and refine the analysis tool.

We applied an algorithmic approach to the review and used an Excel spreadsheet as a record-keeping tool, giving each article an abbreviated name and recording the issue, year, volume, and page numbers. We excluded case reports, review articles that were not systematic reviews or meta-analyses, letters, and editorials. We also excluded articles on cancer chemotherapy because, although family physicians need to have a working knowledge of antineoplastic drugs, they do not routinely prescribe them. Moreover, family physicians rarely write the first prescription for such a drug.

We reviewed the drug treatment articles to determine relevance—that is, whether they met POEMs criteria: addressed a question that most family medicine doctors encounter in a typical 6-month period, measured an outcome that family physicians and patients care about—such as morbidity, mortality, quality of life, or effect on clinical events—and had the potential to change clinical practice.⁸ Articles that met all 3 criteria were included in our analysis; articles that did not were recorded but excluded from further examination.

We analyzed articles that met relevance criteria for validity and clinical usefulness. We assessed validity based on whether the article was a randomized, controlled, double-blinded trial and whether allocation was concealed, follow-up was complete, information was analyzed on an intention-to-treat basis, and the results were statistically significant. We assessed clinical usefulness by recording whether the study reported RRR, ARR, NNT, or NNH, and if so, whether it recorded the information in free text, including the abstract, or in a graph or chart.

Review of articles in 7 journals
After the pilot study, we reviewed articles in JAMA and 6 other journals during a 6-month period from April 1, 2008 through September 30, 2008. We applied the same algorithmic analysis as in the pilot study to drug therapy articles in Archives of Internal Medicine (Arch Intern Med), BMJ, New England Journal of Medicine (N Engl J Med), Lancet, Obstetrics and Gynecology (Obstet Gynecol), and Pediatrics. We met regularly and settled disagreements about relevance, validity, or clinical usefulness by re-reviewing the article.

We recorded the total number of drug therapy articles for each journal, then tallied the number of POEMs and the total number of calculations of RRR, ARR, NNT, and NNH in articles that met POEMs criteria. We used these numbers to determine the percentage of POEMs and POEMs with clinically useful information for each journal.

Results

We identified a total of 995 articles in the 7 journals during the 6-month study period. Of these, 24 (2.4%) were classified as drug therapy POEMs and 6 (0.6%) were POEMs with clinically useful information.

The journals that published the most POEMs were Arch Intern Med, N Engl J Med, BMJ, and JAMA. Arch Intern Med had the highest percentage (33%), followed by JAMA (20%), and BMJ (17%) (TABLE 2).

When we analyzed the POEMs for clinical usefulness based on whether they provided calculated ARR, NNT, or NNH, only 3 journals published POEMs with clinically useful information: JAMA published the most (15%), followed by BMJ (9%) and Obstet Gynecol (4%) (FIGURE).

FIGURE
Percentage of POEMs with clinically useful information*

TABLE 2
Few drug therapy articles are POEMs

Discussion

Our study findings are consistent with the relevance data from a previous study published in 1999.⁸ After more than a decade, medical journals still are not publishing drug therapy POEMs.

A disturbing scarcity of useful information. The paucity of drug therapy POEMs with clinically useful information is alarming. Based on our data we estimate that a physician would have to read on average 36 drug therapy articles to find one clinically helpful drug therapy POEM. This finding suggests that the medical literature is not helping clinicians provide the best patient care or, when it does, the busy clinician is forced to spend what little time is available in calculations to determine what can actually affect practice in positive ways.

Study limitations. Our study has a number of limitations. The study settings in the articles we reviewed ultimately determined what information was important and could potentially change clinical practice. Some studies, for example, were performed in developing countries, where the therapy being tested was not commonly used and would alter practice. In the United States, however, the same treatment would not affect clinical practice because it was either common practice or standard of care.

We reviewed only 7 major medical journals. Our results cannot necessarily be extrapolated to other major journals, although they do suggest that the findings are not limited to a few publications. Moreover, we reviewed only 2 specialty journals. It is possible that other such journals are publishing more POEMs than we observed and providing more concrete numbers that specialists can use to quickly and easily adjust their practice patterns than general journals.

We did not analyze any family medicine journals for the following reasons: American Family Physician publishes only review articles; The Journal of Family Practice does not routinely publish original research; and Annals of Family Medicine (the research journal of the discipline) is less well established than the other journals we selected for this review, having been launched in 2003.

In addition, 6 months may not have been long enough to accurately calculate the percentage of POEMs or clinically useful information in the journals we reviewed. During the pilot study, in which we analyzed a full year of JAMA, only one POEM was published in the second 6 months, and it did not contain clinically useful information. Moreover, we reviewed most journals over 6 consecutive months rather than 6 randomly chosen months.

Finally, we reviewed only drug therapy articles. Future studies could examine surgical, diagnostic, or prognostic studies.

Toward more, and more useful, POEMs
Despite the scarcity of POEMs that provide clinically useful information in major medical journals, it is important that physicians continue to practice evidence-based medicine, sifting through the available information and even calculating ARR, NNT, and NNH themselves, which most busy clinicians do not have the time or inclination to do.

How can we improve the clinical usefulness of published data? One way is for journal editors to require that authors provide ARRs and NNTs or NNHs. Another is for authors to include these calculations on their own initiative. Either way, the goal is better clinical practice and optimal patient care.

CORRESPONDENCE Adrienne Z. Ables, PharmD, Edward Via College of Osteopathic Medicine, Carolinas Campus, 350 Howard Street, Spartanburg, SC 29303; [email protected]

Medical professionals are inundated by new information, which some have described as an “information jungle.”¹ Thousands of articles are published each year in hundreds of journals,² adding to an ever-expanding knowledge base. One study suggests that the experienced primary care physician uses up to 2 million pieces of information each year to manage patients.³ To provide appropriate patient care, physicians must stay abreast of current medical knowledge.⁴ However, busy clinicians have little time to navigate the information jungle and sift through all of the data to determine what is relevant and clinically useful.

More than a decade has passed since Allen Shaughnessy and David Slawson developed the concept of Patient-Oriented Evidence that Matters (POEMs), “a summary of a valid piece of research that carries information that is important to patients and so to their doctors.”⁵ They developed a formula to classify research as a POEM: U=R×V/W, where U=usefulness of the information to doctors, R=relevance of the information to doctors, V=validity of the information, and W=work to access the information.¹ The most useful information is both relevant and valid and takes little work to access. In 2002, the British Medical Journal (BMJ) proposed publishing one POEM a week based on the following criteria:

• It addresses a question that doctors encounter.
• It measures outcomes that doctors and their patients care about: symptoms, morbidity, quality of life, and mortality.
• It has the potential to change the way doctors practice.⁵

Even with the advent of POEMs, the true benefit of research and its application to clinical practice has yet to be determined. Physicians still have to decide which studies are valid, interpret the outcomes, and determine how they affect individual practice.

Research shows that clinicians, patients, and policy makers are more impressed by larger percentage differences than smaller ones.⁶ This fact is evident in the way trial results are presented in the news, by pharmaceutical representatives, and in journal articles. The relative risk reduction (RRR) is touted as suggesting either benefit or reduced harm, and the absolute numbers are largely under-reported. One study found that treatment effectiveness was perceived to be lower when the absolute risk reduction (ARR) rather than the RRR was reported.⁷ Perception of effectiveness decreased further when the number needed to treat (NNT) was presented.

The authors of the study concluded that ARR and NNT provide more concrete information than RRR about an intervention because they express efficacy “in a way which incorporates both the magnitude of the reduction of risk and the baseline risk without treatment.” They note that “because the exclusive reporting of relative risk may overstate the effectiveness of a treatment, actual event rates and absolute changes in risk should be reported.”⁷

These numbers are rarely found in journal articles and, when present, rarely appear in abstracts, tables, or graphs, where the busy clinician looks to find information quickly. Our study sought to estimate the percentage of drug treatment articles published in major medical journals that provide a calculated ARR, NNT, or number needed to harm (NNH), as demonstrated in TABLE 1.

TABLE 1
How to calculate RRR, ARR, and NNT

Methods

Pilot study
We first performed a pilot study that retrospectively reviewed all drug therapy articles published in the Journal of the American Medical Association (JAMA) from April 1, 2008, through April 1, 2009. Its purpose was to ensure concurrence in data gathering, rule out any measurement bias, and refine the analysis tool.

We applied an algorithmic approach to the review and used an Excel spreadsheet as a record-keeping tool, giving each article an abbreviated name and recording the issue, year, volume, and page numbers. We excluded case reports, review articles that were not systematic reviews or meta-analyses, letters, and editorials. We also excluded articles on cancer chemotherapy because, although family physicians need to have a working knowledge of antineoplastic drugs, they do not routinely prescribe them. Moreover, family physicians rarely write the first prescription for such a drug.

We reviewed the drug treatment articles to determine relevance—that is, whether they met POEMs criteria: addressed a question that most family medicine doctors encounter in a typical 6-month period, measured an outcome that family physicians and patients care about—such as morbidity, mortality, quality of life, or effect on clinical events—and had the potential to change clinical practice.⁸ Articles that met all 3 criteria were included in our analysis; articles that did not were recorded but excluded from further examination.

We analyzed articles that met relevance criteria for validity and clinical usefulness. We assessed validity based on whether the article was a randomized, controlled, double-blinded trial and whether allocation was concealed, follow-up was complete, information was analyzed on an intention-to-treat basis, and the results were statistically significant. We assessed clinical usefulness by recording whether the study reported RRR, ARR, NNT, or NNH, and if so, whether it recorded the information in free text, including the abstract, or in a graph or chart.

Review of articles in 7 journals
After the pilot study, we reviewed articles in JAMA and 6 other journals during a 6-month period from April 1, 2008 through September 30, 2008. We applied the same algorithmic analysis as in the pilot study to drug therapy articles in Archives of Internal Medicine (Arch Intern Med), BMJ, New England Journal of Medicine (N Engl J Med), Lancet, Obstetrics and Gynecology (Obstet Gynecol), and Pediatrics. We met regularly and settled disagreements about relevance, validity, or clinical usefulness by re-reviewing the article.

We recorded the total number of drug therapy articles for each journal, then tallied the number of POEMs and the total number of calculations of RRR, ARR, NNT, and NNH in articles that met POEMs criteria. We used these numbers to determine the percentage of POEMs and POEMs with clinically useful information for each journal.

Results

We identified a total of 995 articles in the 7 journals during the 6-month study period. Of these, 24 (2.4%) were classified as drug therapy POEMs and 6 (0.6%) were POEMs with clinically useful information.

The journals that published the most POEMs were Arch Intern Med, N Engl J Med, BMJ, and JAMA. Arch Intern Med had the highest percentage (33%), followed by JAMA (20%), and BMJ (17%) (TABLE 2).

When we analyzed the POEMs for clinical usefulness based on whether they provided calculated ARR, NNT, or NNH, only 3 journals published POEMs with clinically useful information: JAMA published the most (15%), followed by BMJ (9%) and Obstet Gynecol (4%) (FIGURE).

FIGURE
Percentage of POEMs with clinically useful information*

TABLE 2
Few drug therapy articles are POEMs

Discussion

Our study findings are consistent with the relevance data from a previous study published in 1999.⁸ After more than a decade, medical journals still are not publishing drug therapy POEMs.

A disturbing scarcity of useful information. The paucity of drug therapy POEMs with clinically useful information is alarming. Based on our data we estimate that a physician would have to read on average 36 drug therapy articles to find one clinically helpful drug therapy POEM. This finding suggests that the medical literature is not helping clinicians provide the best patient care or, when it does, the busy clinician is forced to spend what little time is available in calculations to determine what can actually affect practice in positive ways.

Study limitations. Our study has a number of limitations. The study settings in the articles we reviewed ultimately determined what information was important and could potentially change clinical practice. Some studies, for example, were performed in developing countries, where the therapy being tested was not commonly used and would alter practice. In the United States, however, the same treatment would not affect clinical practice because it was either common practice or standard of care.

We reviewed only 7 major medical journals. Our results cannot necessarily be extrapolated to other major journals, although they do suggest that the findings are not limited to a few publications. Moreover, we reviewed only 2 specialty journals. It is possible that other such journals are publishing more POEMs than we observed and providing more concrete numbers that specialists can use to quickly and easily adjust their practice patterns than general journals.

We did not analyze any family medicine journals for the following reasons: American Family Physician publishes only review articles; The Journal of Family Practice does not routinely publish original research; and Annals of Family Medicine (the research journal of the discipline) is less well established than the other journals we selected for this review, having been launched in 2003.

In addition, 6 months may not have been long enough to accurately calculate the percentage of POEMs or clinically useful information in the journals we reviewed. During the pilot study, in which we analyzed a full year of JAMA, only one POEM was published in the second 6 months, and it did not contain clinically useful information. Moreover, we reviewed most journals over 6 consecutive months rather than 6 randomly chosen months.

Finally, we reviewed only drug therapy articles. Future studies could examine surgical, diagnostic, or prognostic studies.

Toward more, and more useful, POEMs
Despite the scarcity of POEMs that provide clinically useful information in major medical journals, it is important that physicians continue to practice evidence-based medicine, sifting through the available information and even calculating ARR, NNT, and NNH themselves, which most busy clinicians do not have the time or inclination to do.

How can we improve the clinical usefulness of published data? One way is for journal editors to require that authors provide ARRs and NNTs or NNHs. Another is for authors to include these calculations on their own initiative. Either way, the goal is better clinical practice and optimal patient care.

CORRESPONDENCE Adrienne Z. Ables, PharmD, Edward Via College of Osteopathic Medicine, Carolinas Campus, 350 Howard Street, Spartanburg, SC 29303; [email protected]