Hospital Procedure Volume May Not Capture Facility Quality

SAN FRANCISCO – Hospital procedure volume, which is commonly used as a proxy measure for hospital quality, is not significantly associated with in-hospital mortality for four common surgical procedures, based on a rigorous statistical analysis of data from the Nationwide Inpatient Sample.

Furthermore, "no identifiable threshold values exist for hospital procedure volume at which mortality risk significantly increased. Mortality risk was primarily attributable to patient-level risk factors," said Dr. Damian J. LaPar of the University of Virginia in Charlottesville.

Dr. LaPar and his colleagues examined the relative strength of association between hospital volume and mortality vs. other modeled variables by comparing model covariate likelihood ratios for four high-risk procedures: pancreatic resection, abdominal aortic aneurysm (AAA) repair, esophageal resection, and coronary artery bypass graft (CABG).

Using data from the Nationwide Inpatient Sample in 2008, they obtained weighted discharge records for 261,142 patients: 19,194 patients who had pancreatic resection, 15,266 who had AAA repair, 4,764 who had esophageal resection, and 222,122 who had CABG. The primary outcome of interest was the estimated risk-adjusted effect of hospital procedure volume on mortality (in-hospital death). Comorbid disease was assessed based on Agency for Healthcare Research and Quality (AHRQ) comorbidity categories.

"In all four models, hospital volume was associated with the lowest statistical strength of association with mortality," compared with all other factors, Dr. LaPar said at the annual meeting of the American Surgical Association. Alternatively, other operation and patient-related risk factors – including elective vs. nonelective status, age, sex, hypertension, weight loss, heart failure, chronic obstructive pulmonary disease, liver disease, and renal failure – had higher strengths of association with mortality.

Dr. LaPar noted that procedure volume is an attractive metric for regulatory bodies to use as a predictor of surgical outcomes; it is easy to measure and intuitive in nature. In addition, higher-volume hospitals are more likely to have established system-based processes and the infrastructure in place to improve patient outcomes.

The Leapfrog Group and the AHRQ both have adopted procedure volume as a quality indicator for the four high-risk surgical procedures. Arbitrarily defined volume thresholds have been adopted as a metric of quality for these procedures. However, many previous statistical methods that are used to define these thresholds have drawn criticism in the recent surgical literature. In many former series, volume is represented as arbitrarily defined categories, rather than as a continuous variable. Furthermore, there has been little attempt to rigorously assess and compare statistical model performance; to assess the relative strength of the association of procedure volume with other outcome predictors; and to utilize hierarchical, multilevel, statistical modeling techniques for complex, multicenter patient samples.

Dr. LaPar and his colleagues used hierarchical general linear modeling and created separate models for each procedure, which were adjusted for patient and operative factors as potential confounders. Patient factors included age, sex, and comorbid disease. Operative factors included procedure volume and elective/nonelective status. All model covariates were selected a priori.

The researchers used hospital volume as a continuous variable with restricted cubic spline regression, which uses all data points to estimate the shape of the association between hospital volume and mortality, and is considered to be the best way to visually identify threshold values. They also assessed the relative strength of association between hospital volume and mortality, compared with other factors (likelihood ratio). Model performance was assessed by looking at discrimination, calibration, and predictive capacity.

AAA repair was associated with the greatest in-hospital death. Patients undergoing AAA repair had the greatest burden of comorbid disease, including peripheral vascular disease, chronic obstructive pulmonary disease, and renal failure.

Patient age was 60 years or greater. Women were most represented in pancreatic resections. Most procedures were elective.

Dr. LaPar noted that the study did not investigate the impact of surgeon volume, nor did it adjust for surgical risk factors such as tumor type/stage, pulmonary function, performance status, surgical technique, preoperative medications, and neoadjuvant therapy. The researchers were also unable to assess the effects of hospital volume on long-term survival, resource utilization, and hospital readmission.

The findings have several implications. Previous reports using conventional modeling techniques may have overestimated the significance of hospital volume as a predictor of mortality. "However, these data do not intend to declare that hospital volume is irrelevant, but rather that hospital procedure volume may be a surrogate for other unidentified institutional factors that influence quality," said Dr. LaPar. "Most importantly, these data do not support the current policy of using hospital procedure volume as a proxy measure for quality."

Invited discussant Dr. Edward Livingston praised the group’s rigorous statistical analysis of the association between hospital procedure volume and quality of care (mortality). He noted that earlier papers showed a statistical association between procedure volume and mortality. "Where the volume outcome research efforts took a left turn is that, instead of trying to understand what it was about volume that’s associated with outcomes, there have been 2 decades of papers published looking at and reconfirming a statistical association between procedure volume and outcomes. Procedure volume itself does not translate into better outcomes. It is the things associated with procedure volume, such as surgeon experience, better functioning [operating room teams, and the like]. We really haven’t looked into those causative factors."

If the causative factors could be identified, "then we could take the experience of high-volume centers and translate that to everybody else, so everybody could have good outcomes," he said.

According to Dr. Livingston, the Dr. Lee Hudson–Robert R. Penn Chair in Surgery at the University of Texas, Dallas, previous studies relied on statistical modeling of the mortality relationship. "Those models are only as good as the model can represent the data," he said, and very few have been rigorously assessed to see how well they describe the phenomenon that they’re trying to describe.

Dr. LaPar’s rigorous work shows that the models don’t actually work that well, said Dr. Livingstone. This paper "should serve as the template for what everyone should do when they’re performing volume outcome studies or any kind of regression analysis."

Dr. Livingston asked what metric should be used in place of volume. Dr. LaPar replied, "I think that’s the billion dollar question. ... This is a complex issue; this is a multifactorial issue that likely includes many different qualitative and quantitative measures that we’re going to have to take a look at."

The authors reported that they have no financial disclosures.

The complete manuscript of the presentation is anticipated to be published in the Annals of Surgery pending editorial review.

SAN FRANCISCO – Hospital procedure volume, which is commonly used as a proxy measure for hospital quality, is not significantly associated with in-hospital mortality for four common surgical procedures, based on a rigorous statistical analysis of data from the Nationwide Inpatient Sample.

Furthermore, "no identifiable threshold values exist for hospital procedure volume at which mortality risk significantly increased. Mortality risk was primarily attributable to patient-level risk factors," said Dr. Damian J. LaPar of the University of Virginia in Charlottesville.

Dr. LaPar and his colleagues examined the relative strength of association between hospital volume and mortality vs. other modeled variables by comparing model covariate likelihood ratios for four high-risk procedures: pancreatic resection, abdominal aortic aneurysm (AAA) repair, esophageal resection, and coronary artery bypass graft (CABG).

Using data from the Nationwide Inpatient Sample in 2008, they obtained weighted discharge records for 261,142 patients: 19,194 patients who had pancreatic resection, 15,266 who had AAA repair, 4,764 who had esophageal resection, and 222,122 who had CABG. The primary outcome of interest was the estimated risk-adjusted effect of hospital procedure volume on mortality (in-hospital death). Comorbid disease was assessed based on Agency for Healthcare Research and Quality (AHRQ) comorbidity categories.

"In all four models, hospital volume was associated with the lowest statistical strength of association with mortality," compared with all other factors, Dr. LaPar said at the annual meeting of the American Surgical Association. Alternatively, other operation and patient-related risk factors – including elective vs. nonelective status, age, sex, hypertension, weight loss, heart failure, chronic obstructive pulmonary disease, liver disease, and renal failure – had higher strengths of association with mortality.

Dr. LaPar noted that procedure volume is an attractive metric for regulatory bodies to use as a predictor of surgical outcomes; it is easy to measure and intuitive in nature. In addition, higher-volume hospitals are more likely to have established system-based processes and the infrastructure in place to improve patient outcomes.

The Leapfrog Group and the AHRQ both have adopted procedure volume as a quality indicator for the four high-risk surgical procedures. Arbitrarily defined volume thresholds have been adopted as a metric of quality for these procedures. However, many previous statistical methods that are used to define these thresholds have drawn criticism in the recent surgical literature. In many former series, volume is represented as arbitrarily defined categories, rather than as a continuous variable. Furthermore, there has been little attempt to rigorously assess and compare statistical model performance; to assess the relative strength of the association of procedure volume with other outcome predictors; and to utilize hierarchical, multilevel, statistical modeling techniques for complex, multicenter patient samples.

Dr. LaPar and his colleagues used hierarchical general linear modeling and created separate models for each procedure, which were adjusted for patient and operative factors as potential confounders. Patient factors included age, sex, and comorbid disease. Operative factors included procedure volume and elective/nonelective status. All model covariates were selected a priori.

The researchers used hospital volume as a continuous variable with restricted cubic spline regression, which uses all data points to estimate the shape of the association between hospital volume and mortality, and is considered to be the best way to visually identify threshold values. They also assessed the relative strength of association between hospital volume and mortality, compared with other factors (likelihood ratio). Model performance was assessed by looking at discrimination, calibration, and predictive capacity.

AAA repair was associated with the greatest in-hospital death. Patients undergoing AAA repair had the greatest burden of comorbid disease, including peripheral vascular disease, chronic obstructive pulmonary disease, and renal failure.

Patient age was 60 years or greater. Women were most represented in pancreatic resections. Most procedures were elective.

Dr. LaPar noted that the study did not investigate the impact of surgeon volume, nor did it adjust for surgical risk factors such as tumor type/stage, pulmonary function, performance status, surgical technique, preoperative medications, and neoadjuvant therapy. The researchers were also unable to assess the effects of hospital volume on long-term survival, resource utilization, and hospital readmission.

The findings have several implications. Previous reports using conventional modeling techniques may have overestimated the significance of hospital volume as a predictor of mortality. "However, these data do not intend to declare that hospital volume is irrelevant, but rather that hospital procedure volume may be a surrogate for other unidentified institutional factors that influence quality," said Dr. LaPar. "Most importantly, these data do not support the current policy of using hospital procedure volume as a proxy measure for quality."

Invited discussant Dr. Edward Livingston praised the group’s rigorous statistical analysis of the association between hospital procedure volume and quality of care (mortality). He noted that earlier papers showed a statistical association between procedure volume and mortality. "Where the volume outcome research efforts took a left turn is that, instead of trying to understand what it was about volume that’s associated with outcomes, there have been 2 decades of papers published looking at and reconfirming a statistical association between procedure volume and outcomes. Procedure volume itself does not translate into better outcomes. It is the things associated with procedure volume, such as surgeon experience, better functioning [operating room teams, and the like]. We really haven’t looked into those causative factors."

If the causative factors could be identified, "then we could take the experience of high-volume centers and translate that to everybody else, so everybody could have good outcomes," he said.

According to Dr. Livingston, the Dr. Lee Hudson–Robert R. Penn Chair in Surgery at the University of Texas, Dallas, previous studies relied on statistical modeling of the mortality relationship. "Those models are only as good as the model can represent the data," he said, and very few have been rigorously assessed to see how well they describe the phenomenon that they’re trying to describe.

Dr. LaPar’s rigorous work shows that the models don’t actually work that well, said Dr. Livingstone. This paper "should serve as the template for what everyone should do when they’re performing volume outcome studies or any kind of regression analysis."

Dr. Livingston asked what metric should be used in place of volume. Dr. LaPar replied, "I think that’s the billion dollar question. ... This is a complex issue; this is a multifactorial issue that likely includes many different qualitative and quantitative measures that we’re going to have to take a look at."

The authors reported that they have no financial disclosures.

The complete manuscript of the presentation is anticipated to be published in the Annals of Surgery pending editorial review.

SAN FRANCISCO – Hospital procedure volume, which is commonly used as a proxy measure for hospital quality, is not significantly associated with in-hospital mortality for four common surgical procedures, based on a rigorous statistical analysis of data from the Nationwide Inpatient Sample.

Furthermore, "no identifiable threshold values exist for hospital procedure volume at which mortality risk significantly increased. Mortality risk was primarily attributable to patient-level risk factors," said Dr. Damian J. LaPar of the University of Virginia in Charlottesville.

Dr. LaPar and his colleagues examined the relative strength of association between hospital volume and mortality vs. other modeled variables by comparing model covariate likelihood ratios for four high-risk procedures: pancreatic resection, abdominal aortic aneurysm (AAA) repair, esophageal resection, and coronary artery bypass graft (CABG).

Using data from the Nationwide Inpatient Sample in 2008, they obtained weighted discharge records for 261,142 patients: 19,194 patients who had pancreatic resection, 15,266 who had AAA repair, 4,764 who had esophageal resection, and 222,122 who had CABG. The primary outcome of interest was the estimated risk-adjusted effect of hospital procedure volume on mortality (in-hospital death). Comorbid disease was assessed based on Agency for Healthcare Research and Quality (AHRQ) comorbidity categories.

"In all four models, hospital volume was associated with the lowest statistical strength of association with mortality," compared with all other factors, Dr. LaPar said at the annual meeting of the American Surgical Association. Alternatively, other operation and patient-related risk factors – including elective vs. nonelective status, age, sex, hypertension, weight loss, heart failure, chronic obstructive pulmonary disease, liver disease, and renal failure – had higher strengths of association with mortality.

Dr. LaPar noted that procedure volume is an attractive metric for regulatory bodies to use as a predictor of surgical outcomes; it is easy to measure and intuitive in nature. In addition, higher-volume hospitals are more likely to have established system-based processes and the infrastructure in place to improve patient outcomes.

The Leapfrog Group and the AHRQ both have adopted procedure volume as a quality indicator for the four high-risk surgical procedures. Arbitrarily defined volume thresholds have been adopted as a metric of quality for these procedures. However, many previous statistical methods that are used to define these thresholds have drawn criticism in the recent surgical literature. In many former series, volume is represented as arbitrarily defined categories, rather than as a continuous variable. Furthermore, there has been little attempt to rigorously assess and compare statistical model performance; to assess the relative strength of the association of procedure volume with other outcome predictors; and to utilize hierarchical, multilevel, statistical modeling techniques for complex, multicenter patient samples.

Dr. LaPar and his colleagues used hierarchical general linear modeling and created separate models for each procedure, which were adjusted for patient and operative factors as potential confounders. Patient factors included age, sex, and comorbid disease. Operative factors included procedure volume and elective/nonelective status. All model covariates were selected a priori.

The researchers used hospital volume as a continuous variable with restricted cubic spline regression, which uses all data points to estimate the shape of the association between hospital volume and mortality, and is considered to be the best way to visually identify threshold values. They also assessed the relative strength of association between hospital volume and mortality, compared with other factors (likelihood ratio). Model performance was assessed by looking at discrimination, calibration, and predictive capacity.

AAA repair was associated with the greatest in-hospital death. Patients undergoing AAA repair had the greatest burden of comorbid disease, including peripheral vascular disease, chronic obstructive pulmonary disease, and renal failure.

Patient age was 60 years or greater. Women were most represented in pancreatic resections. Most procedures were elective.

Dr. LaPar noted that the study did not investigate the impact of surgeon volume, nor did it adjust for surgical risk factors such as tumor type/stage, pulmonary function, performance status, surgical technique, preoperative medications, and neoadjuvant therapy. The researchers were also unable to assess the effects of hospital volume on long-term survival, resource utilization, and hospital readmission.

The findings have several implications. Previous reports using conventional modeling techniques may have overestimated the significance of hospital volume as a predictor of mortality. "However, these data do not intend to declare that hospital volume is irrelevant, but rather that hospital procedure volume may be a surrogate for other unidentified institutional factors that influence quality," said Dr. LaPar. "Most importantly, these data do not support the current policy of using hospital procedure volume as a proxy measure for quality."

Invited discussant Dr. Edward Livingston praised the group’s rigorous statistical analysis of the association between hospital procedure volume and quality of care (mortality). He noted that earlier papers showed a statistical association between procedure volume and mortality. "Where the volume outcome research efforts took a left turn is that, instead of trying to understand what it was about volume that’s associated with outcomes, there have been 2 decades of papers published looking at and reconfirming a statistical association between procedure volume and outcomes. Procedure volume itself does not translate into better outcomes. It is the things associated with procedure volume, such as surgeon experience, better functioning [operating room teams, and the like]. We really haven’t looked into those causative factors."

If the causative factors could be identified, "then we could take the experience of high-volume centers and translate that to everybody else, so everybody could have good outcomes," he said.

According to Dr. Livingston, the Dr. Lee Hudson–Robert R. Penn Chair in Surgery at the University of Texas, Dallas, previous studies relied on statistical modeling of the mortality relationship. "Those models are only as good as the model can represent the data," he said, and very few have been rigorously assessed to see how well they describe the phenomenon that they’re trying to describe.

Dr. LaPar’s rigorous work shows that the models don’t actually work that well, said Dr. Livingstone. This paper "should serve as the template for what everyone should do when they’re performing volume outcome studies or any kind of regression analysis."

Dr. Livingston asked what metric should be used in place of volume. Dr. LaPar replied, "I think that’s the billion dollar question. ... This is a complex issue; this is a multifactorial issue that likely includes many different qualitative and quantitative measures that we’re going to have to take a look at."

The authors reported that they have no financial disclosures.

The complete manuscript of the presentation is anticipated to be published in the Annals of Surgery pending editorial review.