Few Strong Studies Examine P4P Programs' Effect on Quality

WASHINGTON — The few studies that have examined the effectiveness of incentivized pay-for-performance programs have found a mix of moderate to no improvement in quality measures, Dr. Daniel B. Mark said at the annual meeting of the Heart Failure Society of America.

There are more than 100 incentive programs in the private U.S. health care sector under the control of employer groups or managed care organizations, but congressionally authorized programs by the Centers for Medicare and Medicaid Services get the most attention, said Dr. Mark, director of the Outcomes Research and Assessment Group at the Duke (University) Clinical Research Institute, Durham, N.C.

During the last 20 years, incentivized performance programs have shown that “what you measure generally improves and what gets measured is generally what's easiest to measure. But the ease of measurement does not necessarily define the importance of the measurement.” Furthermore, little is known about whether these initiatives are cost effective for the health care system at large, Dr. Mark said, though he conceded that may be an oversimplification.

A systematic overview of 17 studies published between 1980–2005 on pay-for-performance programs found that 1 of 2 studies on system-level incentives had a positive result in which all performance measures improved. In nine studies of incentive programs aimed at the provider group level, seven had partially positive or fully positive results but had “quite small” effect sizes. Positive or partially-positive results were seen in five of six programs at the physician level (Ann. Int. Med. 2006;145:265–72).

Nine of the studies were randomized and controlled, but eight had a sample size of fewer than 100 physicians or groups; the other had fewer than 200 groups. “If these had been clinical trials, they would have all been considered extremely underpowered and preliminary,” Dr. Mark said.

Programs in four studies created unintended consequences, including “gaming the baseline level of illness,” avoiding sicker patients, and an improvement in documentation in immunization studies without any actual change in the number of immunizations given or effect on care. The studies did not include any information on the optimal duration of these programs or whether or not their effect persisted after the program was terminated. Only one study had a preliminary examination of the cost-effectiveness of a program.

Another study compared patients with acute non-ST-elevation myocardial infarction in 57 hospitals that participated in CMs' Hospital Quality Incentive Demonstration and 113 control hospitals to determine if a pay-for-performance strategy produced better quality of care. There was “very little evidence that there was any intervention effect,” said Dr. Mark. Measures not incentivized by CMS also did not appear to change (JAMA 2007;297:2373–80).

In the United Kingdom, family practice physicians participated in a pay-for-performance program in 2004 that focused on 146 quality indicators for 10 chronic diseases as well as measures related to the organization of care and the patient's experience. The National Health Service substantially increased its deficit that year because the approximately $3.2 billion that was allocated for the project was eaten by greater than predicted success in achieving the quality indicators (83% achieved vs. an expected 75%). This led to an average increase in the physicians' pay of about $40,000 that year (N. Engl. J. Med. 2006;355:375–84).

Other investigators noted that in the 1998–2003 period prior to the NHS project all of the quality indicators had already been improving, “so it's not clear how much the program's achievements can actually be attributed to the program itself,” he said (N. Engl. J. Med. 2007;357:181–90). And it is not clear what effect the program had on other conditions that were not a part of the incentive program. In any case, the U.K. government has significantly tightened up its requirements for earning extra money in the program in 2008, according to Dr. Mark.

Another study showed public reporting of quality measures alone could improve a set of quality indicators on heart failure and acute myocardial infarction by the same magnitude as a pay-for performance program that included public reporting (N. Engl. J. Med. 2007;356:486–96).

WASHINGTON — The few studies that have examined the effectiveness of incentivized pay-for-performance programs have found a mix of moderate to no improvement in quality measures, Dr. Daniel B. Mark said at the annual meeting of the Heart Failure Society of America.

There are more than 100 incentive programs in the private U.S. health care sector under the control of employer groups or managed care organizations, but congressionally authorized programs by the Centers for Medicare and Medicaid Services get the most attention, said Dr. Mark, director of the Outcomes Research and Assessment Group at the Duke (University) Clinical Research Institute, Durham, N.C.

During the last 20 years, incentivized performance programs have shown that “what you measure generally improves and what gets measured is generally what's easiest to measure. But the ease of measurement does not necessarily define the importance of the measurement.” Furthermore, little is known about whether these initiatives are cost effective for the health care system at large, Dr. Mark said, though he conceded that may be an oversimplification.

A systematic overview of 17 studies published between 1980–2005 on pay-for-performance programs found that 1 of 2 studies on system-level incentives had a positive result in which all performance measures improved. In nine studies of incentive programs aimed at the provider group level, seven had partially positive or fully positive results but had “quite small” effect sizes. Positive or partially-positive results were seen in five of six programs at the physician level (Ann. Int. Med. 2006;145:265–72).

Nine of the studies were randomized and controlled, but eight had a sample size of fewer than 100 physicians or groups; the other had fewer than 200 groups. “If these had been clinical trials, they would have all been considered extremely underpowered and preliminary,” Dr. Mark said.

Programs in four studies created unintended consequences, including “gaming the baseline level of illness,” avoiding sicker patients, and an improvement in documentation in immunization studies without any actual change in the number of immunizations given or effect on care. The studies did not include any information on the optimal duration of these programs or whether or not their effect persisted after the program was terminated. Only one study had a preliminary examination of the cost-effectiveness of a program.

Another study compared patients with acute non-ST-elevation myocardial infarction in 57 hospitals that participated in CMs' Hospital Quality Incentive Demonstration and 113 control hospitals to determine if a pay-for-performance strategy produced better quality of care. There was “very little evidence that there was any intervention effect,” said Dr. Mark. Measures not incentivized by CMS also did not appear to change (JAMA 2007;297:2373–80).

In the United Kingdom, family practice physicians participated in a pay-for-performance program in 2004 that focused on 146 quality indicators for 10 chronic diseases as well as measures related to the organization of care and the patient's experience. The National Health Service substantially increased its deficit that year because the approximately $3.2 billion that was allocated for the project was eaten by greater than predicted success in achieving the quality indicators (83% achieved vs. an expected 75%). This led to an average increase in the physicians' pay of about $40,000 that year (N. Engl. J. Med. 2006;355:375–84).

Other investigators noted that in the 1998–2003 period prior to the NHS project all of the quality indicators had already been improving, “so it's not clear how much the program's achievements can actually be attributed to the program itself,” he said (N. Engl. J. Med. 2007;357:181–90). And it is not clear what effect the program had on other conditions that were not a part of the incentive program. In any case, the U.K. government has significantly tightened up its requirements for earning extra money in the program in 2008, according to Dr. Mark.

Another study showed public reporting of quality measures alone could improve a set of quality indicators on heart failure and acute myocardial infarction by the same magnitude as a pay-for performance program that included public reporting (N. Engl. J. Med. 2007;356:486–96).

WASHINGTON — The few studies that have examined the effectiveness of incentivized pay-for-performance programs have found a mix of moderate to no improvement in quality measures, Dr. Daniel B. Mark said at the annual meeting of the Heart Failure Society of America.

There are more than 100 incentive programs in the private U.S. health care sector under the control of employer groups or managed care organizations, but congressionally authorized programs by the Centers for Medicare and Medicaid Services get the most attention, said Dr. Mark, director of the Outcomes Research and Assessment Group at the Duke (University) Clinical Research Institute, Durham, N.C.

During the last 20 years, incentivized performance programs have shown that “what you measure generally improves and what gets measured is generally what's easiest to measure. But the ease of measurement does not necessarily define the importance of the measurement.” Furthermore, little is known about whether these initiatives are cost effective for the health care system at large, Dr. Mark said, though he conceded that may be an oversimplification.

A systematic overview of 17 studies published between 1980–2005 on pay-for-performance programs found that 1 of 2 studies on system-level incentives had a positive result in which all performance measures improved. In nine studies of incentive programs aimed at the provider group level, seven had partially positive or fully positive results but had “quite small” effect sizes. Positive or partially-positive results were seen in five of six programs at the physician level (Ann. Int. Med. 2006;145:265–72).

Nine of the studies were randomized and controlled, but eight had a sample size of fewer than 100 physicians or groups; the other had fewer than 200 groups. “If these had been clinical trials, they would have all been considered extremely underpowered and preliminary,” Dr. Mark said.

Programs in four studies created unintended consequences, including “gaming the baseline level of illness,” avoiding sicker patients, and an improvement in documentation in immunization studies without any actual change in the number of immunizations given or effect on care. The studies did not include any information on the optimal duration of these programs or whether or not their effect persisted after the program was terminated. Only one study had a preliminary examination of the cost-effectiveness of a program.

Another study compared patients with acute non-ST-elevation myocardial infarction in 57 hospitals that participated in CMs' Hospital Quality Incentive Demonstration and 113 control hospitals to determine if a pay-for-performance strategy produced better quality of care. There was “very little evidence that there was any intervention effect,” said Dr. Mark. Measures not incentivized by CMS also did not appear to change (JAMA 2007;297:2373–80).

In the United Kingdom, family practice physicians participated in a pay-for-performance program in 2004 that focused on 146 quality indicators for 10 chronic diseases as well as measures related to the organization of care and the patient's experience. The National Health Service substantially increased its deficit that year because the approximately $3.2 billion that was allocated for the project was eaten by greater than predicted success in achieving the quality indicators (83% achieved vs. an expected 75%). This led to an average increase in the physicians' pay of about $40,000 that year (N. Engl. J. Med. 2006;355:375–84).

Other investigators noted that in the 1998–2003 period prior to the NHS project all of the quality indicators had already been improving, “so it's not clear how much the program's achievements can actually be attributed to the program itself,” he said (N. Engl. J. Med. 2007;357:181–90). And it is not clear what effect the program had on other conditions that were not a part of the incentive program. In any case, the U.K. government has significantly tightened up its requirements for earning extra money in the program in 2008, according to Dr. Mark.

Another study showed public reporting of quality measures alone could improve a set of quality indicators on heart failure and acute myocardial infarction by the same magnitude as a pay-for performance program that included public reporting (N. Engl. J. Med. 2007;356:486–96).