Vineet Arora, MD, understands the unique value of being named one of this year’s three Masters in Hospital Medicine. It’s an honor bestowed for hospitalists, by hospitalists.

“I take a lot of pride in an honor determined by peers,” said Dr. Arora, an academic hospitalist at University of Chicago Medicine. “While peers are often the biggest support you receive in your professional career, because they are in the trenches with you, they can also be your best critics. That is especially true of the type of work that I do, which relies on the buy-in of frontline clinicians – including hospitalists and trainees – to achieve better patient care and education.”

Vineet Arora, MD, understands the unique value of being named one of this year’s three Masters in Hospital Medicine. It’s an honor bestowed for hospitalists, by hospitalists.

“I take a lot of pride in an honor determined by peers,” said Dr. Arora, an academic hospitalist at University of Chicago Medicine. “While peers are often the biggest support you receive in your professional career, because they are in the trenches with you, they can also be your best critics. That is especially true of the type of work that I do, which relies on the buy-in of frontline clinicians – including hospitalists and trainees – to achieve better patient care and education.”

Vineet Arora, MD, understands the unique value of being named one of this year’s three Masters in Hospital Medicine. It’s an honor bestowed for hospitalists, by hospitalists.

“I take a lot of pride in an honor determined by peers,” said Dr. Arora, an academic hospitalist at University of Chicago Medicine. “While peers are often the biggest support you receive in your professional career, because they are in the trenches with you, they can also be your best critics. That is especially true of the type of work that I do, which relies on the buy-in of frontline clinicians – including hospitalists and trainees – to achieve better patient care and education.”

Occult cancers accounted for one in about every 12 major gastrointestinal bleeding events among patients taking warfarin or dabigatran for atrial fibrillation, according to a retrospective analysis of data from a randomized prospective trial reported in the May issue of Clinical Gastroenterology and Hepatology (2017. doi: org/10.1016/j.cgh.2016.10.011).

These bleeding events caused similarly significant morbidity among patients taking either drug, Kathryn F. Flack, MD, of Icahn School of Medicine at Mount Sinai in New York and her associates wrote. “Patients bleeding from cancer required a mean of approximately 10 nights in the hospital, and approximately one-fourth required intensive care, but 0 of 44 died as a direct result of the bleeding,” the researchers reported. They hoped the specific dabigatran reversal agent, idarucizumab (Praxbind), will improve bleeding outcomes in patients receiving dabigatran.

Source: American Gastroenterological Association

Major gastrointestinal bleeding (MGIB) is the first sign of occult malignancy in certain patients receiving anticoagulation therapy. Starting an anticoagulant is a type of “stress test” that can reveal an occult cancer, the researchers said. Although dabigatran etexilate (Pradaxa) is generally safe and effective, a twice-daily, 150-mg dose of this direct oral anticoagulant slightly increased MGIB, compared with a lower dose in the international, multicenter RE-LY (Randomized Evaluation of Long Term Anticoagulant Therapy) trial (N Engl J Med. 2009;361:1139-51). Furthermore, unlike warfarin, dabigatran therapy places active anticoagulant within the luminal gastrointestinal tract, which “might promote bleeding from friable gastrointestinal cancers,” the investigators noted. To explore this possibility, they evaluated 546 unique MGIB events among RE-LY patients.

Medical chart reviews identified 44 (8.1%) MGIB events resulting from occult gastrointestinal cancers. Cancer accounted for similar proportions of MGIB among warfarin and dabigatran recipients (8.5% and 6.8%; P = .6). Nearly all cancers were colorectal or gastric, except for one case each of ampullary cancer, renal cell carcinoma, and melanoma that had metastasized to the luminal gastrointestinal tract. Colorectal cancer accounted for 80% of cancer-related MGIB overall, including 88% in the dabigatran group and 50% in the warfarin group (P = .02). Conversely, warfarin recipients had more MGIB associated with gastric cancer (50%) than did dabigatran recipients (2.9%; P = .001).

Short-term outcomes of MGIB associated with cancer did not vary by anticoagulant, the investigators said. There were no deaths, but two (4.5%) MGIB events required emergency endoscopic treatment, one (2.3%) required emergency surgery, and 33 (75%) required at least one red blood cell transfusion. Compared with patients whose MGIB was unrelated to cancer, those with cancer were more likely to bleed for more than 7 days (27.3% vs. 63.6%; P less than .001). Patients with occult cancer also developed MGIB sooner after starting anticoagulation (223 vs. 343 days; P = .003), but time to bleeding did not significantly vary by type of anticoagulant.

“Most prior studies on cancer bleeding have been case reports and case series in patients receiving warfarin,” the investigators wrote. “Our study is relevant because of the increasing prevalence of atrial fibrillation and anticoagulation in the aging global population, the increasing prescription of direct oral anticoagulants, and the morbidity, mortality, and complex decision making associated with MGIB and especially cancer-related MGIB in patients receiving anticoagulation therapy.”

The RE-LY trial was sponsored by Boehringer Ingelheim . Dr. Flack reported no conflicts of interest. Senior author James Aisenberg, MD, disclosed advisory board and consulting relationships with Boehringer Ingelheim and Portola Pharmaceuticals. Five other coinvestigators disclosed ties to several pharmaceutical companies, and two coinvestigators reported employment with Boehringer Ingelheim. The other coinvestigators had no conflicts.

Occult cancers accounted for one in about every 12 major gastrointestinal bleeding events among patients taking warfarin or dabigatran for atrial fibrillation, according to a retrospective analysis of data from a randomized prospective trial reported in the May issue of Clinical Gastroenterology and Hepatology (2017. doi: org/10.1016/j.cgh.2016.10.011).

These bleeding events caused similarly significant morbidity among patients taking either drug, Kathryn F. Flack, MD, of Icahn School of Medicine at Mount Sinai in New York and her associates wrote. “Patients bleeding from cancer required a mean of approximately 10 nights in the hospital, and approximately one-fourth required intensive care, but 0 of 44 died as a direct result of the bleeding,” the researchers reported. They hoped the specific dabigatran reversal agent, idarucizumab (Praxbind), will improve bleeding outcomes in patients receiving dabigatran.

Source: American Gastroenterological Association

Major gastrointestinal bleeding (MGIB) is the first sign of occult malignancy in certain patients receiving anticoagulation therapy. Starting an anticoagulant is a type of “stress test” that can reveal an occult cancer, the researchers said. Although dabigatran etexilate (Pradaxa) is generally safe and effective, a twice-daily, 150-mg dose of this direct oral anticoagulant slightly increased MGIB, compared with a lower dose in the international, multicenter RE-LY (Randomized Evaluation of Long Term Anticoagulant Therapy) trial (N Engl J Med. 2009;361:1139-51). Furthermore, unlike warfarin, dabigatran therapy places active anticoagulant within the luminal gastrointestinal tract, which “might promote bleeding from friable gastrointestinal cancers,” the investigators noted. To explore this possibility, they evaluated 546 unique MGIB events among RE-LY patients.

Medical chart reviews identified 44 (8.1%) MGIB events resulting from occult gastrointestinal cancers. Cancer accounted for similar proportions of MGIB among warfarin and dabigatran recipients (8.5% and 6.8%; P = .6). Nearly all cancers were colorectal or gastric, except for one case each of ampullary cancer, renal cell carcinoma, and melanoma that had metastasized to the luminal gastrointestinal tract. Colorectal cancer accounted for 80% of cancer-related MGIB overall, including 88% in the dabigatran group and 50% in the warfarin group (P = .02). Conversely, warfarin recipients had more MGIB associated with gastric cancer (50%) than did dabigatran recipients (2.9%; P = .001).

Short-term outcomes of MGIB associated with cancer did not vary by anticoagulant, the investigators said. There were no deaths, but two (4.5%) MGIB events required emergency endoscopic treatment, one (2.3%) required emergency surgery, and 33 (75%) required at least one red blood cell transfusion. Compared with patients whose MGIB was unrelated to cancer, those with cancer were more likely to bleed for more than 7 days (27.3% vs. 63.6%; P less than .001). Patients with occult cancer also developed MGIB sooner after starting anticoagulation (223 vs. 343 days; P = .003), but time to bleeding did not significantly vary by type of anticoagulant.

“Most prior studies on cancer bleeding have been case reports and case series in patients receiving warfarin,” the investigators wrote. “Our study is relevant because of the increasing prevalence of atrial fibrillation and anticoagulation in the aging global population, the increasing prescription of direct oral anticoagulants, and the morbidity, mortality, and complex decision making associated with MGIB and especially cancer-related MGIB in patients receiving anticoagulation therapy.”

The RE-LY trial was sponsored by Boehringer Ingelheim . Dr. Flack reported no conflicts of interest. Senior author James Aisenberg, MD, disclosed advisory board and consulting relationships with Boehringer Ingelheim and Portola Pharmaceuticals. Five other coinvestigators disclosed ties to several pharmaceutical companies, and two coinvestigators reported employment with Boehringer Ingelheim. The other coinvestigators had no conflicts.

Occult cancers accounted for one in about every 12 major gastrointestinal bleeding events among patients taking warfarin or dabigatran for atrial fibrillation, according to a retrospective analysis of data from a randomized prospective trial reported in the May issue of Clinical Gastroenterology and Hepatology (2017. doi: org/10.1016/j.cgh.2016.10.011).

These bleeding events caused similarly significant morbidity among patients taking either drug, Kathryn F. Flack, MD, of Icahn School of Medicine at Mount Sinai in New York and her associates wrote. “Patients bleeding from cancer required a mean of approximately 10 nights in the hospital, and approximately one-fourth required intensive care, but 0 of 44 died as a direct result of the bleeding,” the researchers reported. They hoped the specific dabigatran reversal agent, idarucizumab (Praxbind), will improve bleeding outcomes in patients receiving dabigatran.

Source: American Gastroenterological Association

Major gastrointestinal bleeding (MGIB) is the first sign of occult malignancy in certain patients receiving anticoagulation therapy. Starting an anticoagulant is a type of “stress test” that can reveal an occult cancer, the researchers said. Although dabigatran etexilate (Pradaxa) is generally safe and effective, a twice-daily, 150-mg dose of this direct oral anticoagulant slightly increased MGIB, compared with a lower dose in the international, multicenter RE-LY (Randomized Evaluation of Long Term Anticoagulant Therapy) trial (N Engl J Med. 2009;361:1139-51). Furthermore, unlike warfarin, dabigatran therapy places active anticoagulant within the luminal gastrointestinal tract, which “might promote bleeding from friable gastrointestinal cancers,” the investigators noted. To explore this possibility, they evaluated 546 unique MGIB events among RE-LY patients.

Medical chart reviews identified 44 (8.1%) MGIB events resulting from occult gastrointestinal cancers. Cancer accounted for similar proportions of MGIB among warfarin and dabigatran recipients (8.5% and 6.8%; P = .6). Nearly all cancers were colorectal or gastric, except for one case each of ampullary cancer, renal cell carcinoma, and melanoma that had metastasized to the luminal gastrointestinal tract. Colorectal cancer accounted for 80% of cancer-related MGIB overall, including 88% in the dabigatran group and 50% in the warfarin group (P = .02). Conversely, warfarin recipients had more MGIB associated with gastric cancer (50%) than did dabigatran recipients (2.9%; P = .001).

Short-term outcomes of MGIB associated with cancer did not vary by anticoagulant, the investigators said. There were no deaths, but two (4.5%) MGIB events required emergency endoscopic treatment, one (2.3%) required emergency surgery, and 33 (75%) required at least one red blood cell transfusion. Compared with patients whose MGIB was unrelated to cancer, those with cancer were more likely to bleed for more than 7 days (27.3% vs. 63.6%; P less than .001). Patients with occult cancer also developed MGIB sooner after starting anticoagulation (223 vs. 343 days; P = .003), but time to bleeding did not significantly vary by type of anticoagulant.

“Most prior studies on cancer bleeding have been case reports and case series in patients receiving warfarin,” the investigators wrote. “Our study is relevant because of the increasing prevalence of atrial fibrillation and anticoagulation in the aging global population, the increasing prescription of direct oral anticoagulants, and the morbidity, mortality, and complex decision making associated with MGIB and especially cancer-related MGIB in patients receiving anticoagulation therapy.”

The RE-LY trial was sponsored by Boehringer Ingelheim . Dr. Flack reported no conflicts of interest. Senior author James Aisenberg, MD, disclosed advisory board and consulting relationships with Boehringer Ingelheim and Portola Pharmaceuticals. Five other coinvestigators disclosed ties to several pharmaceutical companies, and two coinvestigators reported employment with Boehringer Ingelheim. The other coinvestigators had no conflicts.

Robert Wachter, MD, MHM, has given the final plenary address at every SHM annual meeting since 2007. His talks are peppered with his one-of-a-kind take on the confluence of medicine, politics, and policy – and at least once he broke into an Elton John parody.

Where does that point of view come from? As the “dean” of hospital medicine says in his ever-popular Twitter bio, he is “what happens when a poli sci major becomes an academic physician.”

That’s a needed perspective this year, as the level of political upheaval in the United States ups the ante on the tumult the health care field has experienced over the past few years. Questions surrounding the implementation of the Medicare Access and CHIP Reauthorization Act of 2015 (MACRA) and the continued struggles experienced by clinicians using electronic health records (EHR) are among the topics to be addressed.

Robert Wachter, MD, MHM, has given the final plenary address at every SHM annual meeting since 2007. His talks are peppered with his one-of-a-kind take on the confluence of medicine, politics, and policy – and at least once he broke into an Elton John parody.

Where does that point of view come from? As the “dean” of hospital medicine says in his ever-popular Twitter bio, he is “what happens when a poli sci major becomes an academic physician.”

That’s a needed perspective this year, as the level of political upheaval in the United States ups the ante on the tumult the health care field has experienced over the past few years. Questions surrounding the implementation of the Medicare Access and CHIP Reauthorization Act of 2015 (MACRA) and the continued struggles experienced by clinicians using electronic health records (EHR) are among the topics to be addressed.

Robert Wachter, MD, MHM, has given the final plenary address at every SHM annual meeting since 2007. His talks are peppered with his one-of-a-kind take on the confluence of medicine, politics, and policy – and at least once he broke into an Elton John parody.

Where does that point of view come from? As the “dean” of hospital medicine says in his ever-popular Twitter bio, he is “what happens when a poli sci major becomes an academic physician.”

That’s a needed perspective this year, as the level of political upheaval in the United States ups the ante on the tumult the health care field has experienced over the past few years. Questions surrounding the implementation of the Medicare Access and CHIP Reauthorization Act of 2015 (MACRA) and the continued struggles experienced by clinicians using electronic health records (EHR) are among the topics to be addressed.

Ivan Misner once spent one week on Necker Island – the tony 74-acre island in the British Virgin Islands that is entirely owned by billionaire Sir Richard Branson – because he met a guy at a convention.

And Misner is really good at networking.

“I stayed in touch with the person, and when there was an opportunity, I got invited to this incredible ethics program on Necker where I had a chance to meet Sir Richard. It all comes from building relationships with people,” said Misner, founder and chairman of BNI (Business Network International), a 32-year-old global business networking platform based in Charlotte, N.C., that has led CNN to call him “the father of modern networking.”

Ivan Misner once spent one week on Necker Island – the tony 74-acre island in the British Virgin Islands that is entirely owned by billionaire Sir Richard Branson – because he met a guy at a convention.

And Misner is really good at networking.

“I stayed in touch with the person, and when there was an opportunity, I got invited to this incredible ethics program on Necker where I had a chance to meet Sir Richard. It all comes from building relationships with people,” said Misner, founder and chairman of BNI (Business Network International), a 32-year-old global business networking platform based in Charlotte, N.C., that has led CNN to call him “the father of modern networking.”

Ivan Misner once spent one week on Necker Island – the tony 74-acre island in the British Virgin Islands that is entirely owned by billionaire Sir Richard Branson – because he met a guy at a convention.

And Misner is really good at networking.

“I stayed in touch with the person, and when there was an opportunity, I got invited to this incredible ethics program on Necker where I had a chance to meet Sir Richard. It all comes from building relationships with people,” said Misner, founder and chairman of BNI (Business Network International), a 32-year-old global business networking platform based in Charlotte, N.C., that has led CNN to call him “the father of modern networking.”

Hospital-acquired venous thromboembolism (VTE) continues to be a critical quality challenge for U.S. hospitals,¹ and high-risk patients are often not adequately prophylaxed. Use of VTE prophylaxis (VTEP) varies as widely as 26% to 85% of patients in various studies, as does patient outcomes and care expenditures.^2-6 The 9th edition of the American College of Chest Physicians (CHEST) guidelines⁷ recommend the Padua Prediction Score (PPS) to select individual patients who may be at high risk for venous thromboembolism (VTE) and could benefit from thromboprophylaxis. Use of the manually calculated PPS to select patients for thromboprophylaxis has been shown to help decrease 30-day and 90-day mortality associated with VTE events after hospitalization to medical services.⁸ However, the PPS requires time-consuming manual calculation by a provider, who may be focused on more immediate aspects of patient care and several other risk scores competing for his attention, potentially decreasing its use.

Other risk scores that use only discrete scalar data, such as vital signs and lab results to predict early recognition of sepsis, have been successfully automated and implemented within electronic health records (EHRs).^9-11 Successful automation of scores requiring input of diagnoses, recent medical events, and current clinical status such as the PPS remains difficult.¹² Data representing these characteristics are more prone to error, and harder to translate clearly into a single data field than discrete elements like heart rate, potentially impacting validity of the calculated result.¹³ To improve usage of guideline based VTE risk assessment and decrease physician burden, we developed an algorithm called Automated Padua Prediction Score (APPS) that automatically calculates the PPS using only EHR data available within prior encounters and the first 4 hours of admission, a similar timeframe to when admitting providers would be entering orders. Our goal was to assess if an automatically calculated version of the PPS, a score that depends on criteria more complex than vital signs and labs, would accurately assess risk for hospital-acquired VTE when compared to traditional manual calculation of the Padua Prediction Score by a provider.

Site Description and Ethics

The study was conducted at University of California, San Francisco Medical Center, a 790-bed academic hospital; its Institutional Review Board approved the study and collection of data via chart review. Handling of patient information complied with the Health Insurance Portability and Accountability Act of 1996.

Patient Inclusion

Adult patients admitted to a medical or surgical service between July 1, 2012 and April 1, 2014 were included in the study if they were candidates for VTEP, defined as: length of stay (LOS) greater than 2 days, not on hospice care, not pregnant at admission, no present on admission VTE diagnosis, no known contraindications to prophylaxis (eg, gastrointestinal bleed), and were not receiving therapeutic doses of warfarin, low molecular weight heparins, heparin, or novel anticoagulants prior to admission.

Data Sources

Clinical variables were extracted from the EHR’s enterprise data warehouse (EDW) by SQL Server query (Microsoft, Redmond, Washington) and deposited in a secure database. Chart review was conducted by a trained researcher (Mr. Jacolbia) using the EHR and a standardized protocol. Findings were recorded using REDCap (REDCap Consortium, Vanderbilt University, Nashville, Tennessee). The specific ICD-9, procedure, and lab codes used to determine each criterion of APPS are available in the Appendix.

Creation of the Automated Padua Prediction Score (APPS)

We developed APPS from the original 11 criteria that comprise the Padua Prediction Score: active cancer, previous VTE (excluding superficial vein thrombosis), reduced mobility, known thrombophilic condition, recent (1 month or less) trauma and/or surgery, age 70 years or older, heart and/or respiratory failure, acute myocardial infarction and/or ischemic stroke, acute infection and/or rheumatologic disorder, body mass index (BMI) 30 or higher, and ongoing hormonal treatment.¹³ APPS has the same scoring methodology as PPS: criteria are weighted from 1 to 3 points and summed with a maximum score of 20, representing highest risk of VTE. To automate the score calculation from data routinely available in the EHR, APPS checks pre-selected structured data fields for specific values within laboratory results, orders, nursing flowsheets and claims. Claims data included all ICD-9 and procedure codes used for billing purposes. If any of the predetermined data elements are found, then the specific criterion is considered positive; otherwise, it is scored as negative. The creators of the PPS were consulted in the generation of these data queries to replicate the original standards for deeming a criterion positive. The automated calculation required no use of natural language processing.

Characterization of Study Population

We recorded patient demographics (age, race, gender, BMI), LOS, and rate of hospital-acquired VTE. These patients were separated into 2 cohorts determined by the VTE prophylaxis they received. The risk profile of patients who received pharmacologic prophylaxis was hypothesized to be inherently different from those who had not. To evaluate APPS within this heterogeneous cohort, patients were divided into 2 major categories: pharmacologic vs. no pharmacologic prophylaxis. If they had a completed order or medication administration record on the institution’s approved formulary for pharmacologic VTEP, they were considered to have received pharmacologic prophylaxis. If they had only a completed order for usage of mechanical prophylaxis (sequential compression devices) or no evidence of any form of VTEP, they were considered to have received no pharmacologic prophylaxis. Patients with evidence of both pharmacologic and mechanical were placed in the pharmacologic prophylaxis group. To ensure that automated designation of prophylaxis group was accurate, we reviewed 40 randomly chosen charts because prior researchers were able to achieve sensitivity and specificity greater than 90% with that sample size.¹⁴

The primary outcome of hospital-acquired VTE was defined as an ICD-9 code for VTE (specific codes are found in the Appendix) paired with a “present on admission = no” flag on that encounter’s hospital billing data, abstracted from the EDW. A previous study at this institution used the same methodology and found 212/226 (94%) of patients with a VTE ICD-9 code on claim had evidence of a hospital-acquired VTE event upon chart review.¹⁴ Chart review was also completed to ensure that the primary outcome of newly discovered hospital-acquired VTE was differentiated from chronic VTE or history of VTE. Theoretically, ICD-9 codes and other data elements treat chronic VTE, history of VTE, and hospital-acquired VTE as distinct diagnoses, but it was unclear if this was true in our dataset. For 75 randomly selected cases of presumed hospital-acquired VTE, charts were reviewed for evidence that confirmed newly found VTE during that encounter.

Validation of APPS through Comparison to Manual Calculation of the Original PPS

To compare our automated calculation to standard clinical practice, we manually calculated the PPS through chart review within the first 2 days of admission on 300 random patients, a subsample of the entire study cohort. The largest study we could find had manually calculated the PPS of 1,080 hospitalized patients with a mean PPS of 4.86 (standard deviation [SD], 2.26).¹⁵ One researcher (Mr. Jacolbia) accessed the EHR with all patient information available to physicians, including admission notes, orders, labs, flowsheets, past medical history, and all prior encounters to calculate and record the PPS. To limit potential score bias, 2 authors (Drs. Elias and Davies) assessed 30 randomly selected charts from the cohort of 300. The standardized chart review protocol mimicked a physician’s approach to determine if a patient met a criterion, such as concluding if he/she had active cancer by examining medication lists for chemotherapy, procedure notes for radiation, and recent diagnoses on problem lists. After the original PPS was manually calculated, APPS was automatically calculated for the same 300 patients. We intended to characterize similarities and differences between APPS and manual calculation prior to investigating APPS’ predictive capacity for the entire study population, because it would not be feasible to manually calculate the PPS for all 30,726 patients.

Statistical Analysis

For the 75 randomly selected cases of presumed hospital-acquired VTE, the number of cases was chosen by powering our analysis to find a difference in proportion of 20% with 90% power, α = 0.05 (two-sided). We conducted χ² tests on the entire study cohort to determine if there were significant differences in demographics, LOS, and incidence of hospital-acquired VTE by prophylaxis received. For both the pharmacologic and the no pharmacologic prophylaxis groups, we conducted 2-sample Student t tests to determine significant differences in demographics and LOS between patients who experienced a hospital-acquired VTE and those who did not.

For the comparison of our automated calculation to standard clinical practice, we manually calculated the PPS through chart review within the first 2 days of admission on a subsample of 300 random patients. We powered our analysis to detect a difference in mean PPS from 4.86 to 4.36, enough to alter the point value, with 90% power and α = 0.05 (two-sided) and found 300 patients to be comfortably above the required sample size. We compared APPS and manual calculation in the 300-patient cohort using: 2-sample Student t tests to compare mean scores, χ² tests to compare the frequency with which criteria were positive, and receiver operating characteristic (ROC) curves to determine capacity to predict a hospital-acquired VTE event. Pearson’s correlation was also completed to assess score agreement between APPS and manual calculation on a per-patient basis. After comparing automated calculation of APPS to manual chart review on the same 300 patients, we used APPS to calculate scores for the entire study cohort (n = 30,726). We calculated the mean of APPS by prophylaxis group and whether hospital-acquired VTE had occurred. We analyzed APPS’ ROC curve statistics by prophylaxis group to determine its overall predictive capacity in our study population. Lastly, we computed the time required to calculate APPS per patient. Statistical analyses were conducted using SPSS Statistics (IBM, Armonk, New York) and Python 2.7 (Python Software Foundation, Beaverton, Oregon); 95% confidence intervals (CI) and (SD) were reported when appropriate.

Among the 30,726 unique patients in our entire cohort (all patients admitted during the time period who met the study criteria), we found 6574 (21.4%) on pharmacologic (with or without mechanical) prophylaxis, 13,511 (44.0%) on mechanical only, and 10,641 (34.6%) on no prophylaxis. χ² tests found no significant differences in demographics, LOS, or incidence of hospital-acquired VTE between the patients who received mechanical prophylaxis only and those who received no prophylaxis (Table 1). Similarly, there were no differences in these characteristics in patients receiving pharmacologic prophylaxis with or without the addition of mechanical prophylaxis. Designation of prophylaxis group by manual chart review vs. our automated process was found to agree in categorization for 39/40 (97.5%) sampled encounters. When comparing the cohort that received pharmacologic prophylaxis against the cohort that did not, there were significant differences in racial distribution, sex, BMI, and average LOS as shown in Table 1. Those who received pharmacologic prophylaxis were found to be significantly older than those who did not (62.7 years versus 53.2 years, P < 0.001), more likely to be male (50.6% vs, 42.4%, P < 0.001), more likely to have hospital-acquired VTE (2.2% vs. 0.5%, P < 0.001), and to have a shorter LOS (7.1 days vs. 9.8, P < 0.001).

Within the cohort group receiving pharmacologic prophylaxis (n = 6574), hospital-acquired VTE occurred in patients who were significantly younger (58.2 years vs. 62.8 years, P = 0.003) with a greater LOS (23.8 days vs. 6.7, P < 0.001) than those without. Within the group receiving no pharmacologic prophylaxis (n = 24,152), hospital-acquired VTE occurred in patients who were significantly older (57.1 years vs. 53.2 years, P = 0.014) with more than twice the LOS (20.2 days vs. 9.7 days, P < 0.001) compared to those without. Sixty-six of 75 (88%) randomly selected patients in which new VTE was identified by the automated electronic query had this diagnosis confirmed during manual chart review.

As shown in Table 2, automated calculation on a subsample of 300 randomly selected patients using APPS had a mean of 5.5 (SD, 2.9) while manual calculation of the original PPS on the same patients had a mean of 5.1 (SD, 2.6). There was no significant difference in mean between manual calculation and APPS (P = 0.073). There were, however, significant differences in how often individual criteria were considered present. The largest contributors to the difference in scores between APPS and manual calculation were “prior VTE” (positive, 16% vs. 8.3%, respectively) and “reduced mobility” (positive, 74.3% vs. 66%, respectively) as shown in Table 2. In the subsample, there were a total of 6 (2.0%) hospital-acquired VTE events. APPS’ automated calculation had an AUC = 0.79 (CI, 0.63-0.95) that was significant (P = 0.016) with a cutoff value of 5. Chart review’s manual calculation of the PPS had an AUC = 0.76 (CI 0.61-0.91) that was also significant (P = 0.029).

Distribution of Patient Characteristics in Cohort

Automated calculation of APPS using EHR data from prior encounters and the first 4 hours of admission was predictive of in-hospital VTE. APPS performed as well as traditional manual score calculation of the PPS. It was able to do so with no physician input, significantly lessening the burden of calculation and potentially increasing frequency of data-driven VTE risk assessment.

While automated calculation of certain scores is becoming more common, risk calculators that require data beyond vital signs and lab results have lagged,^16-19 in part because of uncertainty about 2 issues. The first is whether EHR data accurately represent the current clinical picture. The second is if a machine-interpretable algorithm to determine a clinical status (eg, “active cancer”) would be similar to a doctor’s perception of that same concept. We attempted to better understand these 2 challenges through developing APPS. Concerning accuracy, EHR data correctly represent the clinical scenario: designations of VTEP and hospital-acquired VTE were accurate in approximately 90% of reviewed cases. Regarding the second concern, when comparing APPS to manual calculation, we found significant differences (P < 0.001) in how often 8 of the 11 criteria were positive, yet no significant difference in overall score and similar predictive capacity. Manual calculation appeared more likely to find data in the index encounter or in structured data. For example, “active cancer” may be documented only in a physician’s note, easily accounted for during a physician’s calculation but missed by APPS looking only for structured data. In contrast, automated calculation found historic criteria, such as “prior VTE” or “known thrombophilic condition,” positive more often. If the patient is being admitted for a problem unrelated to blood clots, the physician may have little time or interest to look through hundreds of EHR documents to discover a 2-year-old VTE. As patients’ records become larger and denser, more historic data can become buried and forgotten. While the 2 scores differ on individual criteria, they are similarly predictive and able to bifurcate the at-risk population to those who should and should not receive pharmacologic prophylaxis.

Disclosures

Dr. Auerbach was supported by NHLBI K24HL098372 during the period of this study. Dr. Khanna, who is an implementation scientist at the University of California San Francisco Center for Digital Health Innovation, is the principal inventor of CareWeb, and may benefit financially from its commercialization. The other authors report no financial conflicts of interest.

Hospital-acquired venous thromboembolism (VTE) continues to be a critical quality challenge for U.S. hospitals,¹ and high-risk patients are often not adequately prophylaxed. Use of VTE prophylaxis (VTEP) varies as widely as 26% to 85% of patients in various studies, as does patient outcomes and care expenditures.^2-6 The 9th edition of the American College of Chest Physicians (CHEST) guidelines⁷ recommend the Padua Prediction Score (PPS) to select individual patients who may be at high risk for venous thromboembolism (VTE) and could benefit from thromboprophylaxis. Use of the manually calculated PPS to select patients for thromboprophylaxis has been shown to help decrease 30-day and 90-day mortality associated with VTE events after hospitalization to medical services.⁸ However, the PPS requires time-consuming manual calculation by a provider, who may be focused on more immediate aspects of patient care and several other risk scores competing for his attention, potentially decreasing its use.

Other risk scores that use only discrete scalar data, such as vital signs and lab results to predict early recognition of sepsis, have been successfully automated and implemented within electronic health records (EHRs).^9-11 Successful automation of scores requiring input of diagnoses, recent medical events, and current clinical status such as the PPS remains difficult.¹² Data representing these characteristics are more prone to error, and harder to translate clearly into a single data field than discrete elements like heart rate, potentially impacting validity of the calculated result.¹³ To improve usage of guideline based VTE risk assessment and decrease physician burden, we developed an algorithm called Automated Padua Prediction Score (APPS) that automatically calculates the PPS using only EHR data available within prior encounters and the first 4 hours of admission, a similar timeframe to when admitting providers would be entering orders. Our goal was to assess if an automatically calculated version of the PPS, a score that depends on criteria more complex than vital signs and labs, would accurately assess risk for hospital-acquired VTE when compared to traditional manual calculation of the Padua Prediction Score by a provider.

Site Description and Ethics

The study was conducted at University of California, San Francisco Medical Center, a 790-bed academic hospital; its Institutional Review Board approved the study and collection of data via chart review. Handling of patient information complied with the Health Insurance Portability and Accountability Act of 1996.

Patient Inclusion

Adult patients admitted to a medical or surgical service between July 1, 2012 and April 1, 2014 were included in the study if they were candidates for VTEP, defined as: length of stay (LOS) greater than 2 days, not on hospice care, not pregnant at admission, no present on admission VTE diagnosis, no known contraindications to prophylaxis (eg, gastrointestinal bleed), and were not receiving therapeutic doses of warfarin, low molecular weight heparins, heparin, or novel anticoagulants prior to admission.

Data Sources

Clinical variables were extracted from the EHR’s enterprise data warehouse (EDW) by SQL Server query (Microsoft, Redmond, Washington) and deposited in a secure database. Chart review was conducted by a trained researcher (Mr. Jacolbia) using the EHR and a standardized protocol. Findings were recorded using REDCap (REDCap Consortium, Vanderbilt University, Nashville, Tennessee). The specific ICD-9, procedure, and lab codes used to determine each criterion of APPS are available in the Appendix.

Creation of the Automated Padua Prediction Score (APPS)

We developed APPS from the original 11 criteria that comprise the Padua Prediction Score: active cancer, previous VTE (excluding superficial vein thrombosis), reduced mobility, known thrombophilic condition, recent (1 month or less) trauma and/or surgery, age 70 years or older, heart and/or respiratory failure, acute myocardial infarction and/or ischemic stroke, acute infection and/or rheumatologic disorder, body mass index (BMI) 30 or higher, and ongoing hormonal treatment.¹³ APPS has the same scoring methodology as PPS: criteria are weighted from 1 to 3 points and summed with a maximum score of 20, representing highest risk of VTE. To automate the score calculation from data routinely available in the EHR, APPS checks pre-selected structured data fields for specific values within laboratory results, orders, nursing flowsheets and claims. Claims data included all ICD-9 and procedure codes used for billing purposes. If any of the predetermined data elements are found, then the specific criterion is considered positive; otherwise, it is scored as negative. The creators of the PPS were consulted in the generation of these data queries to replicate the original standards for deeming a criterion positive. The automated calculation required no use of natural language processing.

Characterization of Study Population

We recorded patient demographics (age, race, gender, BMI), LOS, and rate of hospital-acquired VTE. These patients were separated into 2 cohorts determined by the VTE prophylaxis they received. The risk profile of patients who received pharmacologic prophylaxis was hypothesized to be inherently different from those who had not. To evaluate APPS within this heterogeneous cohort, patients were divided into 2 major categories: pharmacologic vs. no pharmacologic prophylaxis. If they had a completed order or medication administration record on the institution’s approved formulary for pharmacologic VTEP, they were considered to have received pharmacologic prophylaxis. If they had only a completed order for usage of mechanical prophylaxis (sequential compression devices) or no evidence of any form of VTEP, they were considered to have received no pharmacologic prophylaxis. Patients with evidence of both pharmacologic and mechanical were placed in the pharmacologic prophylaxis group. To ensure that automated designation of prophylaxis group was accurate, we reviewed 40 randomly chosen charts because prior researchers were able to achieve sensitivity and specificity greater than 90% with that sample size.¹⁴

The primary outcome of hospital-acquired VTE was defined as an ICD-9 code for VTE (specific codes are found in the Appendix) paired with a “present on admission = no” flag on that encounter’s hospital billing data, abstracted from the EDW. A previous study at this institution used the same methodology and found 212/226 (94%) of patients with a VTE ICD-9 code on claim had evidence of a hospital-acquired VTE event upon chart review.¹⁴ Chart review was also completed to ensure that the primary outcome of newly discovered hospital-acquired VTE was differentiated from chronic VTE or history of VTE. Theoretically, ICD-9 codes and other data elements treat chronic VTE, history of VTE, and hospital-acquired VTE as distinct diagnoses, but it was unclear if this was true in our dataset. For 75 randomly selected cases of presumed hospital-acquired VTE, charts were reviewed for evidence that confirmed newly found VTE during that encounter.

Validation of APPS through Comparison to Manual Calculation of the Original PPS

To compare our automated calculation to standard clinical practice, we manually calculated the PPS through chart review within the first 2 days of admission on 300 random patients, a subsample of the entire study cohort. The largest study we could find had manually calculated the PPS of 1,080 hospitalized patients with a mean PPS of 4.86 (standard deviation [SD], 2.26).¹⁵ One researcher (Mr. Jacolbia) accessed the EHR with all patient information available to physicians, including admission notes, orders, labs, flowsheets, past medical history, and all prior encounters to calculate and record the PPS. To limit potential score bias, 2 authors (Drs. Elias and Davies) assessed 30 randomly selected charts from the cohort of 300. The standardized chart review protocol mimicked a physician’s approach to determine if a patient met a criterion, such as concluding if he/she had active cancer by examining medication lists for chemotherapy, procedure notes for radiation, and recent diagnoses on problem lists. After the original PPS was manually calculated, APPS was automatically calculated for the same 300 patients. We intended to characterize similarities and differences between APPS and manual calculation prior to investigating APPS’ predictive capacity for the entire study population, because it would not be feasible to manually calculate the PPS for all 30,726 patients.

Statistical Analysis

For the 75 randomly selected cases of presumed hospital-acquired VTE, the number of cases was chosen by powering our analysis to find a difference in proportion of 20% with 90% power, α = 0.05 (two-sided). We conducted χ² tests on the entire study cohort to determine if there were significant differences in demographics, LOS, and incidence of hospital-acquired VTE by prophylaxis received. For both the pharmacologic and the no pharmacologic prophylaxis groups, we conducted 2-sample Student t tests to determine significant differences in demographics and LOS between patients who experienced a hospital-acquired VTE and those who did not.

For the comparison of our automated calculation to standard clinical practice, we manually calculated the PPS through chart review within the first 2 days of admission on a subsample of 300 random patients. We powered our analysis to detect a difference in mean PPS from 4.86 to 4.36, enough to alter the point value, with 90% power and α = 0.05 (two-sided) and found 300 patients to be comfortably above the required sample size. We compared APPS and manual calculation in the 300-patient cohort using: 2-sample Student t tests to compare mean scores, χ² tests to compare the frequency with which criteria were positive, and receiver operating characteristic (ROC) curves to determine capacity to predict a hospital-acquired VTE event. Pearson’s correlation was also completed to assess score agreement between APPS and manual calculation on a per-patient basis. After comparing automated calculation of APPS to manual chart review on the same 300 patients, we used APPS to calculate scores for the entire study cohort (n = 30,726). We calculated the mean of APPS by prophylaxis group and whether hospital-acquired VTE had occurred. We analyzed APPS’ ROC curve statistics by prophylaxis group to determine its overall predictive capacity in our study population. Lastly, we computed the time required to calculate APPS per patient. Statistical analyses were conducted using SPSS Statistics (IBM, Armonk, New York) and Python 2.7 (Python Software Foundation, Beaverton, Oregon); 95% confidence intervals (CI) and (SD) were reported when appropriate.

Among the 30,726 unique patients in our entire cohort (all patients admitted during the time period who met the study criteria), we found 6574 (21.4%) on pharmacologic (with or without mechanical) prophylaxis, 13,511 (44.0%) on mechanical only, and 10,641 (34.6%) on no prophylaxis. χ² tests found no significant differences in demographics, LOS, or incidence of hospital-acquired VTE between the patients who received mechanical prophylaxis only and those who received no prophylaxis (Table 1). Similarly, there were no differences in these characteristics in patients receiving pharmacologic prophylaxis with or without the addition of mechanical prophylaxis. Designation of prophylaxis group by manual chart review vs. our automated process was found to agree in categorization for 39/40 (97.5%) sampled encounters. When comparing the cohort that received pharmacologic prophylaxis against the cohort that did not, there were significant differences in racial distribution, sex, BMI, and average LOS as shown in Table 1. Those who received pharmacologic prophylaxis were found to be significantly older than those who did not (62.7 years versus 53.2 years, P < 0.001), more likely to be male (50.6% vs, 42.4%, P < 0.001), more likely to have hospital-acquired VTE (2.2% vs. 0.5%, P < 0.001), and to have a shorter LOS (7.1 days vs. 9.8, P < 0.001).

Within the cohort group receiving pharmacologic prophylaxis (n = 6574), hospital-acquired VTE occurred in patients who were significantly younger (58.2 years vs. 62.8 years, P = 0.003) with a greater LOS (23.8 days vs. 6.7, P < 0.001) than those without. Within the group receiving no pharmacologic prophylaxis (n = 24,152), hospital-acquired VTE occurred in patients who were significantly older (57.1 years vs. 53.2 years, P = 0.014) with more than twice the LOS (20.2 days vs. 9.7 days, P < 0.001) compared to those without. Sixty-six of 75 (88%) randomly selected patients in which new VTE was identified by the automated electronic query had this diagnosis confirmed during manual chart review.

As shown in Table 2, automated calculation on a subsample of 300 randomly selected patients using APPS had a mean of 5.5 (SD, 2.9) while manual calculation of the original PPS on the same patients had a mean of 5.1 (SD, 2.6). There was no significant difference in mean between manual calculation and APPS (P = 0.073). There were, however, significant differences in how often individual criteria were considered present. The largest contributors to the difference in scores between APPS and manual calculation were “prior VTE” (positive, 16% vs. 8.3%, respectively) and “reduced mobility” (positive, 74.3% vs. 66%, respectively) as shown in Table 2. In the subsample, there were a total of 6 (2.0%) hospital-acquired VTE events. APPS’ automated calculation had an AUC = 0.79 (CI, 0.63-0.95) that was significant (P = 0.016) with a cutoff value of 5. Chart review’s manual calculation of the PPS had an AUC = 0.76 (CI 0.61-0.91) that was also significant (P = 0.029).

Distribution of Patient Characteristics in Cohort

Automated calculation of APPS using EHR data from prior encounters and the first 4 hours of admission was predictive of in-hospital VTE. APPS performed as well as traditional manual score calculation of the PPS. It was able to do so with no physician input, significantly lessening the burden of calculation and potentially increasing frequency of data-driven VTE risk assessment.

While automated calculation of certain scores is becoming more common, risk calculators that require data beyond vital signs and lab results have lagged,^16-19 in part because of uncertainty about 2 issues. The first is whether EHR data accurately represent the current clinical picture. The second is if a machine-interpretable algorithm to determine a clinical status (eg, “active cancer”) would be similar to a doctor’s perception of that same concept. We attempted to better understand these 2 challenges through developing APPS. Concerning accuracy, EHR data correctly represent the clinical scenario: designations of VTEP and hospital-acquired VTE were accurate in approximately 90% of reviewed cases. Regarding the second concern, when comparing APPS to manual calculation, we found significant differences (P < 0.001) in how often 8 of the 11 criteria were positive, yet no significant difference in overall score and similar predictive capacity. Manual calculation appeared more likely to find data in the index encounter or in structured data. For example, “active cancer” may be documented only in a physician’s note, easily accounted for during a physician’s calculation but missed by APPS looking only for structured data. In contrast, automated calculation found historic criteria, such as “prior VTE” or “known thrombophilic condition,” positive more often. If the patient is being admitted for a problem unrelated to blood clots, the physician may have little time or interest to look through hundreds of EHR documents to discover a 2-year-old VTE. As patients’ records become larger and denser, more historic data can become buried and forgotten. While the 2 scores differ on individual criteria, they are similarly predictive and able to bifurcate the at-risk population to those who should and should not receive pharmacologic prophylaxis.

Disclosures

Dr. Auerbach was supported by NHLBI K24HL098372 during the period of this study. Dr. Khanna, who is an implementation scientist at the University of California San Francisco Center for Digital Health Innovation, is the principal inventor of CareWeb, and may benefit financially from its commercialization. The other authors report no financial conflicts of interest.

Hospital-acquired venous thromboembolism (VTE) continues to be a critical quality challenge for U.S. hospitals,¹ and high-risk patients are often not adequately prophylaxed. Use of VTE prophylaxis (VTEP) varies as widely as 26% to 85% of patients in various studies, as does patient outcomes and care expenditures.^2-6 The 9th edition of the American College of Chest Physicians (CHEST) guidelines⁷ recommend the Padua Prediction Score (PPS) to select individual patients who may be at high risk for venous thromboembolism (VTE) and could benefit from thromboprophylaxis. Use of the manually calculated PPS to select patients for thromboprophylaxis has been shown to help decrease 30-day and 90-day mortality associated with VTE events after hospitalization to medical services.⁸ However, the PPS requires time-consuming manual calculation by a provider, who may be focused on more immediate aspects of patient care and several other risk scores competing for his attention, potentially decreasing its use.

Other risk scores that use only discrete scalar data, such as vital signs and lab results to predict early recognition of sepsis, have been successfully automated and implemented within electronic health records (EHRs).^9-11 Successful automation of scores requiring input of diagnoses, recent medical events, and current clinical status such as the PPS remains difficult.¹² Data representing these characteristics are more prone to error, and harder to translate clearly into a single data field than discrete elements like heart rate, potentially impacting validity of the calculated result.¹³ To improve usage of guideline based VTE risk assessment and decrease physician burden, we developed an algorithm called Automated Padua Prediction Score (APPS) that automatically calculates the PPS using only EHR data available within prior encounters and the first 4 hours of admission, a similar timeframe to when admitting providers would be entering orders. Our goal was to assess if an automatically calculated version of the PPS, a score that depends on criteria more complex than vital signs and labs, would accurately assess risk for hospital-acquired VTE when compared to traditional manual calculation of the Padua Prediction Score by a provider.

Site Description and Ethics

The study was conducted at University of California, San Francisco Medical Center, a 790-bed academic hospital; its Institutional Review Board approved the study and collection of data via chart review. Handling of patient information complied with the Health Insurance Portability and Accountability Act of 1996.

Patient Inclusion

Adult patients admitted to a medical or surgical service between July 1, 2012 and April 1, 2014 were included in the study if they were candidates for VTEP, defined as: length of stay (LOS) greater than 2 days, not on hospice care, not pregnant at admission, no present on admission VTE diagnosis, no known contraindications to prophylaxis (eg, gastrointestinal bleed), and were not receiving therapeutic doses of warfarin, low molecular weight heparins, heparin, or novel anticoagulants prior to admission.

Data Sources

Clinical variables were extracted from the EHR’s enterprise data warehouse (EDW) by SQL Server query (Microsoft, Redmond, Washington) and deposited in a secure database. Chart review was conducted by a trained researcher (Mr. Jacolbia) using the EHR and a standardized protocol. Findings were recorded using REDCap (REDCap Consortium, Vanderbilt University, Nashville, Tennessee). The specific ICD-9, procedure, and lab codes used to determine each criterion of APPS are available in the Appendix.

Creation of the Automated Padua Prediction Score (APPS)

We developed APPS from the original 11 criteria that comprise the Padua Prediction Score: active cancer, previous VTE (excluding superficial vein thrombosis), reduced mobility, known thrombophilic condition, recent (1 month or less) trauma and/or surgery, age 70 years or older, heart and/or respiratory failure, acute myocardial infarction and/or ischemic stroke, acute infection and/or rheumatologic disorder, body mass index (BMI) 30 or higher, and ongoing hormonal treatment.¹³ APPS has the same scoring methodology as PPS: criteria are weighted from 1 to 3 points and summed with a maximum score of 20, representing highest risk of VTE. To automate the score calculation from data routinely available in the EHR, APPS checks pre-selected structured data fields for specific values within laboratory results, orders, nursing flowsheets and claims. Claims data included all ICD-9 and procedure codes used for billing purposes. If any of the predetermined data elements are found, then the specific criterion is considered positive; otherwise, it is scored as negative. The creators of the PPS were consulted in the generation of these data queries to replicate the original standards for deeming a criterion positive. The automated calculation required no use of natural language processing.

Characterization of Study Population

We recorded patient demographics (age, race, gender, BMI), LOS, and rate of hospital-acquired VTE. These patients were separated into 2 cohorts determined by the VTE prophylaxis they received. The risk profile of patients who received pharmacologic prophylaxis was hypothesized to be inherently different from those who had not. To evaluate APPS within this heterogeneous cohort, patients were divided into 2 major categories: pharmacologic vs. no pharmacologic prophylaxis. If they had a completed order or medication administration record on the institution’s approved formulary for pharmacologic VTEP, they were considered to have received pharmacologic prophylaxis. If they had only a completed order for usage of mechanical prophylaxis (sequential compression devices) or no evidence of any form of VTEP, they were considered to have received no pharmacologic prophylaxis. Patients with evidence of both pharmacologic and mechanical were placed in the pharmacologic prophylaxis group. To ensure that automated designation of prophylaxis group was accurate, we reviewed 40 randomly chosen charts because prior researchers were able to achieve sensitivity and specificity greater than 90% with that sample size.¹⁴

The primary outcome of hospital-acquired VTE was defined as an ICD-9 code for VTE (specific codes are found in the Appendix) paired with a “present on admission = no” flag on that encounter’s hospital billing data, abstracted from the EDW. A previous study at this institution used the same methodology and found 212/226 (94%) of patients with a VTE ICD-9 code on claim had evidence of a hospital-acquired VTE event upon chart review.¹⁴ Chart review was also completed to ensure that the primary outcome of newly discovered hospital-acquired VTE was differentiated from chronic VTE or history of VTE. Theoretically, ICD-9 codes and other data elements treat chronic VTE, history of VTE, and hospital-acquired VTE as distinct diagnoses, but it was unclear if this was true in our dataset. For 75 randomly selected cases of presumed hospital-acquired VTE, charts were reviewed for evidence that confirmed newly found VTE during that encounter.

Validation of APPS through Comparison to Manual Calculation of the Original PPS

To compare our automated calculation to standard clinical practice, we manually calculated the PPS through chart review within the first 2 days of admission on 300 random patients, a subsample of the entire study cohort. The largest study we could find had manually calculated the PPS of 1,080 hospitalized patients with a mean PPS of 4.86 (standard deviation [SD], 2.26).¹⁵ One researcher (Mr. Jacolbia) accessed the EHR with all patient information available to physicians, including admission notes, orders, labs, flowsheets, past medical history, and all prior encounters to calculate and record the PPS. To limit potential score bias, 2 authors (Drs. Elias and Davies) assessed 30 randomly selected charts from the cohort of 300. The standardized chart review protocol mimicked a physician’s approach to determine if a patient met a criterion, such as concluding if he/she had active cancer by examining medication lists for chemotherapy, procedure notes for radiation, and recent diagnoses on problem lists. After the original PPS was manually calculated, APPS was automatically calculated for the same 300 patients. We intended to characterize similarities and differences between APPS and manual calculation prior to investigating APPS’ predictive capacity for the entire study population, because it would not be feasible to manually calculate the PPS for all 30,726 patients.

Statistical Analysis

For the 75 randomly selected cases of presumed hospital-acquired VTE, the number of cases was chosen by powering our analysis to find a difference in proportion of 20% with 90% power, α = 0.05 (two-sided). We conducted χ² tests on the entire study cohort to determine if there were significant differences in demographics, LOS, and incidence of hospital-acquired VTE by prophylaxis received. For both the pharmacologic and the no pharmacologic prophylaxis groups, we conducted 2-sample Student t tests to determine significant differences in demographics and LOS between patients who experienced a hospital-acquired VTE and those who did not.

For the comparison of our automated calculation to standard clinical practice, we manually calculated the PPS through chart review within the first 2 days of admission on a subsample of 300 random patients. We powered our analysis to detect a difference in mean PPS from 4.86 to 4.36, enough to alter the point value, with 90% power and α = 0.05 (two-sided) and found 300 patients to be comfortably above the required sample size. We compared APPS and manual calculation in the 300-patient cohort using: 2-sample Student t tests to compare mean scores, χ² tests to compare the frequency with which criteria were positive, and receiver operating characteristic (ROC) curves to determine capacity to predict a hospital-acquired VTE event. Pearson’s correlation was also completed to assess score agreement between APPS and manual calculation on a per-patient basis. After comparing automated calculation of APPS to manual chart review on the same 300 patients, we used APPS to calculate scores for the entire study cohort (n = 30,726). We calculated the mean of APPS by prophylaxis group and whether hospital-acquired VTE had occurred. We analyzed APPS’ ROC curve statistics by prophylaxis group to determine its overall predictive capacity in our study population. Lastly, we computed the time required to calculate APPS per patient. Statistical analyses were conducted using SPSS Statistics (IBM, Armonk, New York) and Python 2.7 (Python Software Foundation, Beaverton, Oregon); 95% confidence intervals (CI) and (SD) were reported when appropriate.

Among the 30,726 unique patients in our entire cohort (all patients admitted during the time period who met the study criteria), we found 6574 (21.4%) on pharmacologic (with or without mechanical) prophylaxis, 13,511 (44.0%) on mechanical only, and 10,641 (34.6%) on no prophylaxis. χ² tests found no significant differences in demographics, LOS, or incidence of hospital-acquired VTE between the patients who received mechanical prophylaxis only and those who received no prophylaxis (Table 1). Similarly, there were no differences in these characteristics in patients receiving pharmacologic prophylaxis with or without the addition of mechanical prophylaxis. Designation of prophylaxis group by manual chart review vs. our automated process was found to agree in categorization for 39/40 (97.5%) sampled encounters. When comparing the cohort that received pharmacologic prophylaxis against the cohort that did not, there were significant differences in racial distribution, sex, BMI, and average LOS as shown in Table 1. Those who received pharmacologic prophylaxis were found to be significantly older than those who did not (62.7 years versus 53.2 years, P < 0.001), more likely to be male (50.6% vs, 42.4%, P < 0.001), more likely to have hospital-acquired VTE (2.2% vs. 0.5%, P < 0.001), and to have a shorter LOS (7.1 days vs. 9.8, P < 0.001).

Within the cohort group receiving pharmacologic prophylaxis (n = 6574), hospital-acquired VTE occurred in patients who were significantly younger (58.2 years vs. 62.8 years, P = 0.003) with a greater LOS (23.8 days vs. 6.7, P < 0.001) than those without. Within the group receiving no pharmacologic prophylaxis (n = 24,152), hospital-acquired VTE occurred in patients who were significantly older (57.1 years vs. 53.2 years, P = 0.014) with more than twice the LOS (20.2 days vs. 9.7 days, P < 0.001) compared to those without. Sixty-six of 75 (88%) randomly selected patients in which new VTE was identified by the automated electronic query had this diagnosis confirmed during manual chart review.

As shown in Table 2, automated calculation on a subsample of 300 randomly selected patients using APPS had a mean of 5.5 (SD, 2.9) while manual calculation of the original PPS on the same patients had a mean of 5.1 (SD, 2.6). There was no significant difference in mean between manual calculation and APPS (P = 0.073). There were, however, significant differences in how often individual criteria were considered present. The largest contributors to the difference in scores between APPS and manual calculation were “prior VTE” (positive, 16% vs. 8.3%, respectively) and “reduced mobility” (positive, 74.3% vs. 66%, respectively) as shown in Table 2. In the subsample, there were a total of 6 (2.0%) hospital-acquired VTE events. APPS’ automated calculation had an AUC = 0.79 (CI, 0.63-0.95) that was significant (P = 0.016) with a cutoff value of 5. Chart review’s manual calculation of the PPS had an AUC = 0.76 (CI 0.61-0.91) that was also significant (P = 0.029).

Distribution of Patient Characteristics in Cohort

Automated calculation of APPS using EHR data from prior encounters and the first 4 hours of admission was predictive of in-hospital VTE. APPS performed as well as traditional manual score calculation of the PPS. It was able to do so with no physician input, significantly lessening the burden of calculation and potentially increasing frequency of data-driven VTE risk assessment.

While automated calculation of certain scores is becoming more common, risk calculators that require data beyond vital signs and lab results have lagged,^16-19 in part because of uncertainty about 2 issues. The first is whether EHR data accurately represent the current clinical picture. The second is if a machine-interpretable algorithm to determine a clinical status (eg, “active cancer”) would be similar to a doctor’s perception of that same concept. We attempted to better understand these 2 challenges through developing APPS. Concerning accuracy, EHR data correctly represent the clinical scenario: designations of VTEP and hospital-acquired VTE were accurate in approximately 90% of reviewed cases. Regarding the second concern, when comparing APPS to manual calculation, we found significant differences (P < 0.001) in how often 8 of the 11 criteria were positive, yet no significant difference in overall score and similar predictive capacity. Manual calculation appeared more likely to find data in the index encounter or in structured data. For example, “active cancer” may be documented only in a physician’s note, easily accounted for during a physician’s calculation but missed by APPS looking only for structured data. In contrast, automated calculation found historic criteria, such as “prior VTE” or “known thrombophilic condition,” positive more often. If the patient is being admitted for a problem unrelated to blood clots, the physician may have little time or interest to look through hundreds of EHR documents to discover a 2-year-old VTE. As patients’ records become larger and denser, more historic data can become buried and forgotten. While the 2 scores differ on individual criteria, they are similarly predictive and able to bifurcate the at-risk population to those who should and should not receive pharmacologic prophylaxis.

Disclosures

Dr. Auerbach was supported by NHLBI K24HL098372 during the period of this study. Dr. Khanna, who is an implementation scientist at the University of California San Francisco Center for Digital Health Innovation, is the principal inventor of CareWeb, and may benefit financially from its commercialization. The other authors report no financial conflicts of interest.

Approximately 20% of hospitalized Medicare beneficiaries in the U.S. are discharged to skilled nursing facilities (SNFs) for post-acute care,^1,2 and 23.5% of these patients are readmitted within 30 days.³ Because hospital readmissions are costly and associated with worse outcomes,^4,5 30-day readmission rates are considered a quality indicator,⁶ and there are financial penalties for hospitals with higher than expected rates.⁷ As a result, hospitals invest substantial resources in programs to reduce readmissions.^8-10 The SNFs represent an attractive target for readmission reduction efforts, since SNFs contribute a disproportionate share of readmissions.^3,4 Because SNF patients are in a monitored environment with high medication adherence, risk factors for readmission likely differ between patients discharged to SNFs and those sent home. For example, 1 study showed that among heart failure patients with cognitive impairment, those discharged to SNFs had lower readmissions during the first 20 days, likely due to better medication adherence.¹¹ Patients discharged to SNFs generally have more complex illnesses, lower functional status, and higher 1-year mortality than patients discharged to the community.^12,13 Despite this, SNF patients might have infrequent contact with physicians. Federal regulations require only that patients discharged to SNFs need to be seen within 30 days and then at least once every 30 days thereafter.¹⁴ According to the 2014 Office of Inspector General report, one-third of Medicare beneficiaries in SNFs experience adverse events from substandard treatment, inadequate resident monitoring and failure or delay of necessary care, most of which are thought to be preventable.¹⁵

To address this issue, the Cleveland Clinic developed a program called “Connected Care SNF,” in which hospital-employed physicians and advanced practice professionals visit patients in selected SNFs 4 to 5 times per week, for the purpose of reducing preventable readmissions. The aim of this study was to assess whether the program reduced 30-day readmissions, and to identify which patients benefited most from the program.

Setting and Intervention

The Cleveland Clinic main campus is a tertiary academic medical center with 1400 beds and approximately 50,000 admissions per year. In late 2012, the Cleveland Clinic implemented the Connected Care SNF program, wherein Cleveland Clinic physicians regularly visited patients who were discharged from the Cleveland Clinic main campus to 7 regional SNFs. Beginning in December 2012, these 7 high-volume referral SNFs that were not part of the Cleveland Clinic Health System (CCHS) agreed to participate in the program, which focused on reducing avoidable hospital readmissions and delivering quality care (Table 1). The Connected Care team, comprised of 2 geriatricians (1 of whom was also a palliative medicine specialist), 1 internist, 1 family physician, and 5 advanced practice professionals (nurse practitioners and physician assistants), provided medical services at the participating SNFs. These providers aimed to see patients 4 to 5 times per week, were available on site during working hours, and provided telephone coverage at nights and on weekends. All providers had access to hospital electronic medical records and could communicate with the discharging physician and with specialists familiar with the patient as needed. Prior to the admission, providers were informed about patient arrival and, at the time of admission to the SNF, providers reviewed medications and discussed goals of care with patients and their families. In the SNF, providers worked closely with staff members to deliver medications and timely treatment. They also met monthly with multidisciplinary teams for continuous quality improvement and to review outcomes. Patients at Connected Care SNFs who had their own physicians, including most long-stay and some short-stay residents, did not receive the Connected Care intervention. They constituted less than 10% of the patients discharged from Cleveland Clinic main campus.

Study Design and Population

We reviewed administrative and clinical data from a retrospective cohort of patients discharged to SNF from the Cleveland Clinic main campus from January 1, 2011 to December 31, 2014. We included all patients who were discharged to an SNF during the study period. Our main outcome measure was 30-day all-cause readmissions to any hospital in the Cleveland Clinic Health System (CCHS), including the main campus and 8 regional community hospitals. Study patients were followed until January 30, 2015 to capture 30-day readmissions. According to 2012 Medicare data, of CCHS patients who were readmitted within 30 days, 83% of pneumonia, 81% of major joint replacement, 72% of heart failure and 57% of acute myocardial infarction patients were readmitted to a CCHS facility. As the Cleveland Clinic main campus attracts cardiac patients from a 100+-mile radius, they may be more likely to seek care readmission near home and are not reflective of CCHS patients overall. Because we did not have access to readmissions data from non-CCHS hospitals, we excluded patients who were discharged to SNFs beyond a 25-mile radius from the main campus, where they may be more likely to utilize non-CCHS hospitals for acute hospitalization. We also excluded patients discharged to non-CCHS hospital-based SNFs, which may refer readmissions to their own hospital system. Because the Connected Care program began in December 2012, the years 2011-2012 served as the baseline period. The intervention was conducted at 7 SNFs. All other SNFs within the 25-mile radius were included as controls, except for 3 hospital-based SNFs that would be unlikely to admit patients to CCHS. We compared the change in all-cause 30-day readmission rates after implementation of Connected Care, using all patients discharged to SNFs within 25 miles to control for temporal changes in local readmission rates. Discharge to specific SNFs was determined solely by patient choice.

Data Collection

For each patient, we collected the following data that has been shown to be associated with readmissions:^16-18 demographics (age, race, sex, ZIP code), lab values on discharge (hemoglobin and sodium); hemodialysis status; medicine or surgical service; elective surgery or nonelective surgery; details of the index admission index (diagnosis-related group [DRG], Medicare severity-diagnosis-related groups [MS-DRG] weight, primary diagnosis code; principal procedure code; admission date; discharge date, length of stay, and post-acute care provider); and common comorbidities, as listed in Table 2. We also calculated each patient’s HOSPITAL^19,20 score. The HOSPITAL score was developed to predict risk of preventable 30-day readmissions,¹⁹ but it has also been validated to predict 30-day all-cause readmission rates for patients discharged to SNF.²¹ The model contains 7 elements (hemoglobin, oncology service, sodium, procedure, index type, admissions within the last year, length of stay) (supplemental Table).Patients with a high score (7 or higher) have a 41% chance of readmission, while those with a low score (4 or lower) have only a 15% chance. ²¹ We assessed all cause 30-day readmission status from CCHS administrative data. Observation patients and outpatient same-day surgeries were not considered to be admissions. For patients with multiple admissions, each admission was counted as a separate index hospitalization. Cleveland Clinic’s Institutional Review Board approved the study.

Statistical Analysis

For the 7 intervention SNFs, patient characteristics were summarized as means and standard deviations or frequencies and percentages for the periods of 2011-2012 and 2013-2014, respectively, and the 2 periods were compared using the Student t test or χ² test as appropriate.

Mixed-effects logistic regression models were used to model 30-day readmission rates. Since the intervention was implemented in the last quarter of 2012, we examined the difference in readmission rates before and after that time point. The model included the following fixed effects: SNF type (intervention or usual care), time points (quarters of 2011-2014), whether the time is pre- or postintervention (binary), and the 3-way interaction between SNF type, pre- or postintervention and time points, and patient characteristics. The model also contained a Gaussian random effect at the SNF level to account for possible correlations among the outcomes of patients from the same SNF. For each quarter, the mean adjusted readmission rates of 2 types of SNFs were calculated from the fitted mixed models and plotted over time. Furthermore, we compared the mean readmission rates of the 2 groups in the pre- and postintervention periods. Subgroup analyses were performed for medical and surgical patients, and for patients in the low, intermediate and high HOSPITAL score groups.

All analyses were performed using RStudio (Boston, Massachusetts). Statistical significance was established with 2-sided P values less than 0.05.

We identified 119 SNFs within a 25-mile radius of the hospital. Of these, 6 did not receive any referrals. Three non-CCHS hospital-based SNFs were excluded, leaving a total of 110 SNFs in the study sample: 7 intervention SNFs and 103 usual-care SNFs. Between January 2011 and December 2014, there were 23,408 SNF discharges from Cleveland Clinic main campus, including 13,544 who were discharged to study SNFs (Supplemental Figure). Of these, 3334 were discharged to 7 intervention SNFs and 10,210 were discharged to usual care SNFs. Characteristics of patients in both periods appear in Table 2. At baseline, patients in the intervention and control SNFs varied in a number of ways. Patients at intervention SNFs were older (75.6 vs. 70.2 years; P < 0.001), more likely to be African American (45.5% vs. 35.9%; P < 0.001), female (61% vs. 55.4%; P < 0.001) and to be insured by Medicare (85.2% vs. 71.4%; P < 0.001). Both groups had similar proportions of patients with high, intermediate, and low readmission risk as measured by HOSPITAL score. Compared to the 2011-2012 pre-intervention period, during the 2013-2014 intervention period, there were more surgeries (34.3% vs. 41.9%; P < 0.001), more elective surgeries (21.8% vs. 25.5%; P = 0.01), fewer medical patients (65.7% vs. 58.1%; P < 0.001), and an increase in comorbidities, including myocardial infarction, peripheral vascular disease, and liver disease (Table 2).

Table 3 shows adjusted 30-day readmissions rates, before and during the intervention period at the intervention and usual care SNFs. Compared to the pre-intervention period, 30-day all-cause adjusted readmission rates declined in the intervention SNFs (28.1% to 21.7%, P < 0.001), while it increased slightly at control sites (27.1% to 28.5%, P < 0.001). The Figure shows the adjusted 30-day readmission rates by quarter throughout the study period.

In this retrospective study of 4 years of discharges to 110 SNFs, we report on the impact of a Connected Care program, in which a physician visited patients on admission to the SNF and 4 to 5 times per week during their stay. Introduction of the program was followed by a 6.8% absolute reduction in all-cause 30-day readmission rates compared to usual care. The absolute reductions ranged from 4.6% for patients at low risk for readmission to 9.1% for patients at high risk, and medical patients benefited more than surgical patients.

Most studies of interventions to reduce hospital readmissions have focused on patients discharged to the community setting.^7-9 Interventions have centered on discharge planning, medication reconciliation, and close follow-up to assess for medication adherence and early signs of deterioration. Because patients in SNFs have their medications administered by staff and are under frequent surveillance, such interventions are unlikely to be helpful in this population. We found no studies that focus on short-stay or skilled patients discharged to SNF. Two studies have demonstrated that interventions can reduce hospitalization from nursing homes.^22,23 Neither study included readmissions. The Evercare model consisted of nurse practitioners providing active primary care services within the nursing home, as well as offering incentive payments to nursing homes for not hospitalizing patients.²² During a 2-year period, long term residents who enrolled in Evercare had an almost 50% reduction in incident hospitalizations compared to those who did not.²² INTERACT II was a quality improvement intervention that provided tools, education, and strategies to help identify and manage acute conditions proactively.²³ In 25 nursing homes employing INTERACT II, there was a 17% reduction in self-reported hospital admissions during the 6-month project, with higher rates of reduction among nursing homes rated as more engaged in the process.²³ Although nursing homes may serve some short-stay or skilled patients, they generally serve long-term populations, and studies have shown that short-stay patients are at higher risk for 30-day readmissions.²⁴

There are a number of reasons that short-term SNF patients are at higher risk for readmission. Although prior to admission, they were considered hospital level of care and received a physician visit daily, on transfer to the SNF, relatively little medical care is available. Current federal regulations regarding physician services at a SNF require the resident to be seen by a physician at least once every 30 days for the first 90 days after admission, and at least once every 60 days thereafter.²⁵

The Connected Care program physicians provided a smooth transition of care from hospital to SNF as well as frequent reassessment. Physicians were alerted prior to hospital discharge and performed an initial comprehensive visit generally on the day of admission to the SNF and always within 48 hours. The initial evaluation is important because miscommunication during the handoff from hospital to SNF may result in incorrect medication regimens or inaccurate assessments. By performing prompt medication reconciliation and periodic reassessments of a patient’s medical condition, the Connected Care providers recreate some of the essential elements of successful outpatient readmissions prevention programs.

They also worked together with each SNF’s interdisciplinary team to deliver quality care. There were monthly meetings at each participating Connected Care SNF. Physicians reviewed monthly 30-day readmissions and performed root-cause analysis. When they discovered challenges to timely medication and treatment delivery during daily rounds, they provided in-services to SNF nurses.

In addition, Connected Care providers discussed goals of care—something that is often overlooked on admission to a SNF. This is particularly important because patients with chronic illnesses who are discharged to SNF often have poor prognoses. For example, Medicare patients with heart failure who are discharged to SNFs have 1-year mortality in excess of 50%.¹³ By implementing a plan of care consistent with patient and family goals, inappropriate readmissions for terminal patients may be avoided.

Reducing readmissions is important for hospitals because under the Hospital Readmissions Reduction Program, hospitals now face substantial penalties for higher than expected readmissions rates. Hospitals involved in bundled payments or other total cost-of-care arrangements have additional incentive to avoid readmissions. Beginning in 2019, SNFs will also receive incentive payments based on their 30-day all-cause hospital readmissions as part of the Skilled Nursing Facility Value-Based Purchasing program.²⁵ The Connected Care model offers 1 means of achieving this goal through partnership between hospitals and SNFs.

Our study has several limitations. First, our study was observational in nature, so the observed reduction in readmissions could have been due to temporal trends unrelated to the intervention. However, no significant reduction was noted during the same time period in other area SNFs. There was also little change in the characteristics of patients admitted to the intervention SNFs. Importantly, the HOSPITAL score, which can predict 30-day readmission rates,²⁰ did not change throughout the study period. Second, the results reflect patients discharged from a single hospital and may not be generalizable to other geographic areas. However, because the program included 7 SNFs, we believe it could be reproduced in other settings. Third, our readmissions measure included only those patients who returned to a CCHS facility. Although we may have missed some readmissions to other hospital systems, such leakage is uncommon—more than 80% of CCHS patients are readmitted to CCHS facilities—and would be unlikely to differ across the short duration of the study. Finally, at the intervention SNFs, most long-stay and some short-stay residents did not receive the Connected Care intervention because they were cared for by their own physicians who did not participate in Connected Care. Had these patients’ readmissions been excluded from our results, the intervention might appear even more effective.

CONCLUSION

A Connected Care intervention reduced 30-day readmission rates among patients discharged to SNFs from a tertiary academic center. While all subgroups had substantial reductions in readmissions following the implementation of the intervention, patients who are at the highest risk of readmission benefited the most. Further study is necessary to know whether Connected Care can be reproduced in other health care systems and whether it reduces overall costs.

Acknowledgments

The authors would like to thank Michael Felver, MD, and teams for their clinical care of patients; Michael Felver, MD, William Zafirau, MD, Dan Blechschmid, MHA, and Kathy Brezine, and Seth Vilensky, MBA, for their administrative support; and Brad Souder, MPT, for assistance with data collection.

Disclosure

Nothing to report.

Approximately 20% of hospitalized Medicare beneficiaries in the U.S. are discharged to skilled nursing facilities (SNFs) for post-acute care,^1,2 and 23.5% of these patients are readmitted within 30 days.³ Because hospital readmissions are costly and associated with worse outcomes,^4,5 30-day readmission rates are considered a quality indicator,⁶ and there are financial penalties for hospitals with higher than expected rates.⁷ As a result, hospitals invest substantial resources in programs to reduce readmissions.^8-10 The SNFs represent an attractive target for readmission reduction efforts, since SNFs contribute a disproportionate share of readmissions.^3,4 Because SNF patients are in a monitored environment with high medication adherence, risk factors for readmission likely differ between patients discharged to SNFs and those sent home. For example, 1 study showed that among heart failure patients with cognitive impairment, those discharged to SNFs had lower readmissions during the first 20 days, likely due to better medication adherence.¹¹ Patients discharged to SNFs generally have more complex illnesses, lower functional status, and higher 1-year mortality than patients discharged to the community.^12,13 Despite this, SNF patients might have infrequent contact with physicians. Federal regulations require only that patients discharged to SNFs need to be seen within 30 days and then at least once every 30 days thereafter.¹⁴ According to the 2014 Office of Inspector General report, one-third of Medicare beneficiaries in SNFs experience adverse events from substandard treatment, inadequate resident monitoring and failure or delay of necessary care, most of which are thought to be preventable.¹⁵

To address this issue, the Cleveland Clinic developed a program called “Connected Care SNF,” in which hospital-employed physicians and advanced practice professionals visit patients in selected SNFs 4 to 5 times per week, for the purpose of reducing preventable readmissions. The aim of this study was to assess whether the program reduced 30-day readmissions, and to identify which patients benefited most from the program.

Setting and Intervention

The Cleveland Clinic main campus is a tertiary academic medical center with 1400 beds and approximately 50,000 admissions per year. In late 2012, the Cleveland Clinic implemented the Connected Care SNF program, wherein Cleveland Clinic physicians regularly visited patients who were discharged from the Cleveland Clinic main campus to 7 regional SNFs. Beginning in December 2012, these 7 high-volume referral SNFs that were not part of the Cleveland Clinic Health System (CCHS) agreed to participate in the program, which focused on reducing avoidable hospital readmissions and delivering quality care (Table 1). The Connected Care team, comprised of 2 geriatricians (1 of whom was also a palliative medicine specialist), 1 internist, 1 family physician, and 5 advanced practice professionals (nurse practitioners and physician assistants), provided medical services at the participating SNFs. These providers aimed to see patients 4 to 5 times per week, were available on site during working hours, and provided telephone coverage at nights and on weekends. All providers had access to hospital electronic medical records and could communicate with the discharging physician and with specialists familiar with the patient as needed. Prior to the admission, providers were informed about patient arrival and, at the time of admission to the SNF, providers reviewed medications and discussed goals of care with patients and their families. In the SNF, providers worked closely with staff members to deliver medications and timely treatment. They also met monthly with multidisciplinary teams for continuous quality improvement and to review outcomes. Patients at Connected Care SNFs who had their own physicians, including most long-stay and some short-stay residents, did not receive the Connected Care intervention. They constituted less than 10% of the patients discharged from Cleveland Clinic main campus.

Study Design and Population

We reviewed administrative and clinical data from a retrospective cohort of patients discharged to SNF from the Cleveland Clinic main campus from January 1, 2011 to December 31, 2014. We included all patients who were discharged to an SNF during the study period. Our main outcome measure was 30-day all-cause readmissions to any hospital in the Cleveland Clinic Health System (CCHS), including the main campus and 8 regional community hospitals. Study patients were followed until January 30, 2015 to capture 30-day readmissions. According to 2012 Medicare data, of CCHS patients who were readmitted within 30 days, 83% of pneumonia, 81% of major joint replacement, 72% of heart failure and 57% of acute myocardial infarction patients were readmitted to a CCHS facility. As the Cleveland Clinic main campus attracts cardiac patients from a 100+-mile radius, they may be more likely to seek care readmission near home and are not reflective of CCHS patients overall. Because we did not have access to readmissions data from non-CCHS hospitals, we excluded patients who were discharged to SNFs beyond a 25-mile radius from the main campus, where they may be more likely to utilize non-CCHS hospitals for acute hospitalization. We also excluded patients discharged to non-CCHS hospital-based SNFs, which may refer readmissions to their own hospital system. Because the Connected Care program began in December 2012, the years 2011-2012 served as the baseline period. The intervention was conducted at 7 SNFs. All other SNFs within the 25-mile radius were included as controls, except for 3 hospital-based SNFs that would be unlikely to admit patients to CCHS. We compared the change in all-cause 30-day readmission rates after implementation of Connected Care, using all patients discharged to SNFs within 25 miles to control for temporal changes in local readmission rates. Discharge to specific SNFs was determined solely by patient choice.

Data Collection

For each patient, we collected the following data that has been shown to be associated with readmissions:^16-18 demographics (age, race, sex, ZIP code), lab values on discharge (hemoglobin and sodium); hemodialysis status; medicine or surgical service; elective surgery or nonelective surgery; details of the index admission index (diagnosis-related group [DRG], Medicare severity-diagnosis-related groups [MS-DRG] weight, primary diagnosis code; principal procedure code; admission date; discharge date, length of stay, and post-acute care provider); and common comorbidities, as listed in Table 2. We also calculated each patient’s HOSPITAL^19,20 score. The HOSPITAL score was developed to predict risk of preventable 30-day readmissions,¹⁹ but it has also been validated to predict 30-day all-cause readmission rates for patients discharged to SNF.²¹ The model contains 7 elements (hemoglobin, oncology service, sodium, procedure, index type, admissions within the last year, length of stay) (supplemental Table).Patients with a high score (7 or higher) have a 41% chance of readmission, while those with a low score (4 or lower) have only a 15% chance. ²¹ We assessed all cause 30-day readmission status from CCHS administrative data. Observation patients and outpatient same-day surgeries were not considered to be admissions. For patients with multiple admissions, each admission was counted as a separate index hospitalization. Cleveland Clinic’s Institutional Review Board approved the study.

Statistical Analysis

For the 7 intervention SNFs, patient characteristics were summarized as means and standard deviations or frequencies and percentages for the periods of 2011-2012 and 2013-2014, respectively, and the 2 periods were compared using the Student t test or χ² test as appropriate.

Mixed-effects logistic regression models were used to model 30-day readmission rates. Since the intervention was implemented in the last quarter of 2012, we examined the difference in readmission rates before and after that time point. The model included the following fixed effects: SNF type (intervention or usual care), time points (quarters of 2011-2014), whether the time is pre- or postintervention (binary), and the 3-way interaction between SNF type, pre- or postintervention and time points, and patient characteristics. The model also contained a Gaussian random effect at the SNF level to account for possible correlations among the outcomes of patients from the same SNF. For each quarter, the mean adjusted readmission rates of 2 types of SNFs were calculated from the fitted mixed models and plotted over time. Furthermore, we compared the mean readmission rates of the 2 groups in the pre- and postintervention periods. Subgroup analyses were performed for medical and surgical patients, and for patients in the low, intermediate and high HOSPITAL score groups.

All analyses were performed using RStudio (Boston, Massachusetts). Statistical significance was established with 2-sided P values less than 0.05.

We identified 119 SNFs within a 25-mile radius of the hospital. Of these, 6 did not receive any referrals. Three non-CCHS hospital-based SNFs were excluded, leaving a total of 110 SNFs in the study sample: 7 intervention SNFs and 103 usual-care SNFs. Between January 2011 and December 2014, there were 23,408 SNF discharges from Cleveland Clinic main campus, including 13,544 who were discharged to study SNFs (Supplemental Figure). Of these, 3334 were discharged to 7 intervention SNFs and 10,210 were discharged to usual care SNFs. Characteristics of patients in both periods appear in Table 2. At baseline, patients in the intervention and control SNFs varied in a number of ways. Patients at intervention SNFs were older (75.6 vs. 70.2 years; P < 0.001), more likely to be African American (45.5% vs. 35.9%; P < 0.001), female (61% vs. 55.4%; P < 0.001) and to be insured by Medicare (85.2% vs. 71.4%; P < 0.001). Both groups had similar proportions of patients with high, intermediate, and low readmission risk as measured by HOSPITAL score. Compared to the 2011-2012 pre-intervention period, during the 2013-2014 intervention period, there were more surgeries (34.3% vs. 41.9%; P < 0.001), more elective surgeries (21.8% vs. 25.5%; P = 0.01), fewer medical patients (65.7% vs. 58.1%; P < 0.001), and an increase in comorbidities, including myocardial infarction, peripheral vascular disease, and liver disease (Table 2).

Table 3 shows adjusted 30-day readmissions rates, before and during the intervention period at the intervention and usual care SNFs. Compared to the pre-intervention period, 30-day all-cause adjusted readmission rates declined in the intervention SNFs (28.1% to 21.7%, P < 0.001), while it increased slightly at control sites (27.1% to 28.5%, P < 0.001). The Figure shows the adjusted 30-day readmission rates by quarter throughout the study period.

In this retrospective study of 4 years of discharges to 110 SNFs, we report on the impact of a Connected Care program, in which a physician visited patients on admission to the SNF and 4 to 5 times per week during their stay. Introduction of the program was followed by a 6.8% absolute reduction in all-cause 30-day readmission rates compared to usual care. The absolute reductions ranged from 4.6% for patients at low risk for readmission to 9.1% for patients at high risk, and medical patients benefited more than surgical patients.

Most studies of interventions to reduce hospital readmissions have focused on patients discharged to the community setting.^7-9 Interventions have centered on discharge planning, medication reconciliation, and close follow-up to assess for medication adherence and early signs of deterioration. Because patients in SNFs have their medications administered by staff and are under frequent surveillance, such interventions are unlikely to be helpful in this population. We found no studies that focus on short-stay or skilled patients discharged to SNF. Two studies have demonstrated that interventions can reduce hospitalization from nursing homes.^22,23 Neither study included readmissions. The Evercare model consisted of nurse practitioners providing active primary care services within the nursing home, as well as offering incentive payments to nursing homes for not hospitalizing patients.²² During a 2-year period, long term residents who enrolled in Evercare had an almost 50% reduction in incident hospitalizations compared to those who did not.²² INTERACT II was a quality improvement intervention that provided tools, education, and strategies to help identify and manage acute conditions proactively.²³ In 25 nursing homes employing INTERACT II, there was a 17% reduction in self-reported hospital admissions during the 6-month project, with higher rates of reduction among nursing homes rated as more engaged in the process.²³ Although nursing homes may serve some short-stay or skilled patients, they generally serve long-term populations, and studies have shown that short-stay patients are at higher risk for 30-day readmissions.²⁴

There are a number of reasons that short-term SNF patients are at higher risk for readmission. Although prior to admission, they were considered hospital level of care and received a physician visit daily, on transfer to the SNF, relatively little medical care is available. Current federal regulations regarding physician services at a SNF require the resident to be seen by a physician at least once every 30 days for the first 90 days after admission, and at least once every 60 days thereafter.²⁵

The Connected Care program physicians provided a smooth transition of care from hospital to SNF as well as frequent reassessment. Physicians were alerted prior to hospital discharge and performed an initial comprehensive visit generally on the day of admission to the SNF and always within 48 hours. The initial evaluation is important because miscommunication during the handoff from hospital to SNF may result in incorrect medication regimens or inaccurate assessments. By performing prompt medication reconciliation and periodic reassessments of a patient’s medical condition, the Connected Care providers recreate some of the essential elements of successful outpatient readmissions prevention programs.

They also worked together with each SNF’s interdisciplinary team to deliver quality care. There were monthly meetings at each participating Connected Care SNF. Physicians reviewed monthly 30-day readmissions and performed root-cause analysis. When they discovered challenges to timely medication and treatment delivery during daily rounds, they provided in-services to SNF nurses.

In addition, Connected Care providers discussed goals of care—something that is often overlooked on admission to a SNF. This is particularly important because patients with chronic illnesses who are discharged to SNF often have poor prognoses. For example, Medicare patients with heart failure who are discharged to SNFs have 1-year mortality in excess of 50%.¹³ By implementing a plan of care consistent with patient and family goals, inappropriate readmissions for terminal patients may be avoided.

Reducing readmissions is important for hospitals because under the Hospital Readmissions Reduction Program, hospitals now face substantial penalties for higher than expected readmissions rates. Hospitals involved in bundled payments or other total cost-of-care arrangements have additional incentive to avoid readmissions. Beginning in 2019, SNFs will also receive incentive payments based on their 30-day all-cause hospital readmissions as part of the Skilled Nursing Facility Value-Based Purchasing program.²⁵ The Connected Care model offers 1 means of achieving this goal through partnership between hospitals and SNFs.

Our study has several limitations. First, our study was observational in nature, so the observed reduction in readmissions could have been due to temporal trends unrelated to the intervention. However, no significant reduction was noted during the same time period in other area SNFs. There was also little change in the characteristics of patients admitted to the intervention SNFs. Importantly, the HOSPITAL score, which can predict 30-day readmission rates,²⁰ did not change throughout the study period. Second, the results reflect patients discharged from a single hospital and may not be generalizable to other geographic areas. However, because the program included 7 SNFs, we believe it could be reproduced in other settings. Third, our readmissions measure included only those patients who returned to a CCHS facility. Although we may have missed some readmissions to other hospital systems, such leakage is uncommon—more than 80% of CCHS patients are readmitted to CCHS facilities—and would be unlikely to differ across the short duration of the study. Finally, at the intervention SNFs, most long-stay and some short-stay residents did not receive the Connected Care intervention because they were cared for by their own physicians who did not participate in Connected Care. Had these patients’ readmissions been excluded from our results, the intervention might appear even more effective.

CONCLUSION

A Connected Care intervention reduced 30-day readmission rates among patients discharged to SNFs from a tertiary academic center. While all subgroups had substantial reductions in readmissions following the implementation of the intervention, patients who are at the highest risk of readmission benefited the most. Further study is necessary to know whether Connected Care can be reproduced in other health care systems and whether it reduces overall costs.

Acknowledgments

The authors would like to thank Michael Felver, MD, and teams for their clinical care of patients; Michael Felver, MD, William Zafirau, MD, Dan Blechschmid, MHA, and Kathy Brezine, and Seth Vilensky, MBA, for their administrative support; and Brad Souder, MPT, for assistance with data collection.

Disclosure

Nothing to report.

Approximately 20% of hospitalized Medicare beneficiaries in the U.S. are discharged to skilled nursing facilities (SNFs) for post-acute care,^1,2 and 23.5% of these patients are readmitted within 30 days.³ Because hospital readmissions are costly and associated with worse outcomes,^4,5 30-day readmission rates are considered a quality indicator,⁶ and there are financial penalties for hospitals with higher than expected rates.⁷ As a result, hospitals invest substantial resources in programs to reduce readmissions.^8-10 The SNFs represent an attractive target for readmission reduction efforts, since SNFs contribute a disproportionate share of readmissions.^3,4 Because SNF patients are in a monitored environment with high medication adherence, risk factors for readmission likely differ between patients discharged to SNFs and those sent home. For example, 1 study showed that among heart failure patients with cognitive impairment, those discharged to SNFs had lower readmissions during the first 20 days, likely due to better medication adherence.¹¹ Patients discharged to SNFs generally have more complex illnesses, lower functional status, and higher 1-year mortality than patients discharged to the community.^12,13 Despite this, SNF patients might have infrequent contact with physicians. Federal regulations require only that patients discharged to SNFs need to be seen within 30 days and then at least once every 30 days thereafter.¹⁴ According to the 2014 Office of Inspector General report, one-third of Medicare beneficiaries in SNFs experience adverse events from substandard treatment, inadequate resident monitoring and failure or delay of necessary care, most of which are thought to be preventable.¹⁵

To address this issue, the Cleveland Clinic developed a program called “Connected Care SNF,” in which hospital-employed physicians and advanced practice professionals visit patients in selected SNFs 4 to 5 times per week, for the purpose of reducing preventable readmissions. The aim of this study was to assess whether the program reduced 30-day readmissions, and to identify which patients benefited most from the program.

Setting and Intervention

The Cleveland Clinic main campus is a tertiary academic medical center with 1400 beds and approximately 50,000 admissions per year. In late 2012, the Cleveland Clinic implemented the Connected Care SNF program, wherein Cleveland Clinic physicians regularly visited patients who were discharged from the Cleveland Clinic main campus to 7 regional SNFs. Beginning in December 2012, these 7 high-volume referral SNFs that were not part of the Cleveland Clinic Health System (CCHS) agreed to participate in the program, which focused on reducing avoidable hospital readmissions and delivering quality care (Table 1). The Connected Care team, comprised of 2 geriatricians (1 of whom was also a palliative medicine specialist), 1 internist, 1 family physician, and 5 advanced practice professionals (nurse practitioners and physician assistants), provided medical services at the participating SNFs. These providers aimed to see patients 4 to 5 times per week, were available on site during working hours, and provided telephone coverage at nights and on weekends. All providers had access to hospital electronic medical records and could communicate with the discharging physician and with specialists familiar with the patient as needed. Prior to the admission, providers were informed about patient arrival and, at the time of admission to the SNF, providers reviewed medications and discussed goals of care with patients and their families. In the SNF, providers worked closely with staff members to deliver medications and timely treatment. They also met monthly with multidisciplinary teams for continuous quality improvement and to review outcomes. Patients at Connected Care SNFs who had their own physicians, including most long-stay and some short-stay residents, did not receive the Connected Care intervention. They constituted less than 10% of the patients discharged from Cleveland Clinic main campus.

Study Design and Population

We reviewed administrative and clinical data from a retrospective cohort of patients discharged to SNF from the Cleveland Clinic main campus from January 1, 2011 to December 31, 2014. We included all patients who were discharged to an SNF during the study period. Our main outcome measure was 30-day all-cause readmissions to any hospital in the Cleveland Clinic Health System (CCHS), including the main campus and 8 regional community hospitals. Study patients were followed until January 30, 2015 to capture 30-day readmissions. According to 2012 Medicare data, of CCHS patients who were readmitted within 30 days, 83% of pneumonia, 81% of major joint replacement, 72% of heart failure and 57% of acute myocardial infarction patients were readmitted to a CCHS facility. As the Cleveland Clinic main campus attracts cardiac patients from a 100+-mile radius, they may be more likely to seek care readmission near home and are not reflective of CCHS patients overall. Because we did not have access to readmissions data from non-CCHS hospitals, we excluded patients who were discharged to SNFs beyond a 25-mile radius from the main campus, where they may be more likely to utilize non-CCHS hospitals for acute hospitalization. We also excluded patients discharged to non-CCHS hospital-based SNFs, which may refer readmissions to their own hospital system. Because the Connected Care program began in December 2012, the years 2011-2012 served as the baseline period. The intervention was conducted at 7 SNFs. All other SNFs within the 25-mile radius were included as controls, except for 3 hospital-based SNFs that would be unlikely to admit patients to CCHS. We compared the change in all-cause 30-day readmission rates after implementation of Connected Care, using all patients discharged to SNFs within 25 miles to control for temporal changes in local readmission rates. Discharge to specific SNFs was determined solely by patient choice.

Data Collection

For each patient, we collected the following data that has been shown to be associated with readmissions:^16-18 demographics (age, race, sex, ZIP code), lab values on discharge (hemoglobin and sodium); hemodialysis status; medicine or surgical service; elective surgery or nonelective surgery; details of the index admission index (diagnosis-related group [DRG], Medicare severity-diagnosis-related groups [MS-DRG] weight, primary diagnosis code; principal procedure code; admission date; discharge date, length of stay, and post-acute care provider); and common comorbidities, as listed in Table 2. We also calculated each patient’s HOSPITAL^19,20 score. The HOSPITAL score was developed to predict risk of preventable 30-day readmissions,¹⁹ but it has also been validated to predict 30-day all-cause readmission rates for patients discharged to SNF.²¹ The model contains 7 elements (hemoglobin, oncology service, sodium, procedure, index type, admissions within the last year, length of stay) (supplemental Table).Patients with a high score (7 or higher) have a 41% chance of readmission, while those with a low score (4 or lower) have only a 15% chance. ²¹ We assessed all cause 30-day readmission status from CCHS administrative data. Observation patients and outpatient same-day surgeries were not considered to be admissions. For patients with multiple admissions, each admission was counted as a separate index hospitalization. Cleveland Clinic’s Institutional Review Board approved the study.

Statistical Analysis

For the 7 intervention SNFs, patient characteristics were summarized as means and standard deviations or frequencies and percentages for the periods of 2011-2012 and 2013-2014, respectively, and the 2 periods were compared using the Student t test or χ² test as appropriate.

Mixed-effects logistic regression models were used to model 30-day readmission rates. Since the intervention was implemented in the last quarter of 2012, we examined the difference in readmission rates before and after that time point. The model included the following fixed effects: SNF type (intervention or usual care), time points (quarters of 2011-2014), whether the time is pre- or postintervention (binary), and the 3-way interaction between SNF type, pre- or postintervention and time points, and patient characteristics. The model also contained a Gaussian random effect at the SNF level to account for possible correlations among the outcomes of patients from the same SNF. For each quarter, the mean adjusted readmission rates of 2 types of SNFs were calculated from the fitted mixed models and plotted over time. Furthermore, we compared the mean readmission rates of the 2 groups in the pre- and postintervention periods. Subgroup analyses were performed for medical and surgical patients, and for patients in the low, intermediate and high HOSPITAL score groups.

All analyses were performed using RStudio (Boston, Massachusetts). Statistical significance was established with 2-sided P values less than 0.05.

We identified 119 SNFs within a 25-mile radius of the hospital. Of these, 6 did not receive any referrals. Three non-CCHS hospital-based SNFs were excluded, leaving a total of 110 SNFs in the study sample: 7 intervention SNFs and 103 usual-care SNFs. Between January 2011 and December 2014, there were 23,408 SNF discharges from Cleveland Clinic main campus, including 13,544 who were discharged to study SNFs (Supplemental Figure). Of these, 3334 were discharged to 7 intervention SNFs and 10,210 were discharged to usual care SNFs. Characteristics of patients in both periods appear in Table 2. At baseline, patients in the intervention and control SNFs varied in a number of ways. Patients at intervention SNFs were older (75.6 vs. 70.2 years; P < 0.001), more likely to be African American (45.5% vs. 35.9%; P < 0.001), female (61% vs. 55.4%; P < 0.001) and to be insured by Medicare (85.2% vs. 71.4%; P < 0.001). Both groups had similar proportions of patients with high, intermediate, and low readmission risk as measured by HOSPITAL score. Compared to the 2011-2012 pre-intervention period, during the 2013-2014 intervention period, there were more surgeries (34.3% vs. 41.9%; P < 0.001), more elective surgeries (21.8% vs. 25.5%; P = 0.01), fewer medical patients (65.7% vs. 58.1%; P < 0.001), and an increase in comorbidities, including myocardial infarction, peripheral vascular disease, and liver disease (Table 2).

Table 3 shows adjusted 30-day readmissions rates, before and during the intervention period at the intervention and usual care SNFs. Compared to the pre-intervention period, 30-day all-cause adjusted readmission rates declined in the intervention SNFs (28.1% to 21.7%, P < 0.001), while it increased slightly at control sites (27.1% to 28.5%, P < 0.001). The Figure shows the adjusted 30-day readmission rates by quarter throughout the study period.

In this retrospective study of 4 years of discharges to 110 SNFs, we report on the impact of a Connected Care program, in which a physician visited patients on admission to the SNF and 4 to 5 times per week during their stay. Introduction of the program was followed by a 6.8% absolute reduction in all-cause 30-day readmission rates compared to usual care. The absolute reductions ranged from 4.6% for patients at low risk for readmission to 9.1% for patients at high risk, and medical patients benefited more than surgical patients.

Most studies of interventions to reduce hospital readmissions have focused on patients discharged to the community setting.^7-9 Interventions have centered on discharge planning, medication reconciliation, and close follow-up to assess for medication adherence and early signs of deterioration. Because patients in SNFs have their medications administered by staff and are under frequent surveillance, such interventions are unlikely to be helpful in this population. We found no studies that focus on short-stay or skilled patients discharged to SNF. Two studies have demonstrated that interventions can reduce hospitalization from nursing homes.^22,23 Neither study included readmissions. The Evercare model consisted of nurse practitioners providing active primary care services within the nursing home, as well as offering incentive payments to nursing homes for not hospitalizing patients.²² During a 2-year period, long term residents who enrolled in Evercare had an almost 50% reduction in incident hospitalizations compared to those who did not.²² INTERACT II was a quality improvement intervention that provided tools, education, and strategies to help identify and manage acute conditions proactively.²³ In 25 nursing homes employing INTERACT II, there was a 17% reduction in self-reported hospital admissions during the 6-month project, with higher rates of reduction among nursing homes rated as more engaged in the process.²³ Although nursing homes may serve some short-stay or skilled patients, they generally serve long-term populations, and studies have shown that short-stay patients are at higher risk for 30-day readmissions.²⁴

There are a number of reasons that short-term SNF patients are at higher risk for readmission. Although prior to admission, they were considered hospital level of care and received a physician visit daily, on transfer to the SNF, relatively little medical care is available. Current federal regulations regarding physician services at a SNF require the resident to be seen by a physician at least once every 30 days for the first 90 days after admission, and at least once every 60 days thereafter.²⁵

The Connected Care program physicians provided a smooth transition of care from hospital to SNF as well as frequent reassessment. Physicians were alerted prior to hospital discharge and performed an initial comprehensive visit generally on the day of admission to the SNF and always within 48 hours. The initial evaluation is important because miscommunication during the handoff from hospital to SNF may result in incorrect medication regimens or inaccurate assessments. By performing prompt medication reconciliation and periodic reassessments of a patient’s medical condition, the Connected Care providers recreate some of the essential elements of successful outpatient readmissions prevention programs.

They also worked together with each SNF’s interdisciplinary team to deliver quality care. There were monthly meetings at each participating Connected Care SNF. Physicians reviewed monthly 30-day readmissions and performed root-cause analysis. When they discovered challenges to timely medication and treatment delivery during daily rounds, they provided in-services to SNF nurses.

In addition, Connected Care providers discussed goals of care—something that is often overlooked on admission to a SNF. This is particularly important because patients with chronic illnesses who are discharged to SNF often have poor prognoses. For example, Medicare patients with heart failure who are discharged to SNFs have 1-year mortality in excess of 50%.¹³ By implementing a plan of care consistent with patient and family goals, inappropriate readmissions for terminal patients may be avoided.

Reducing readmissions is important for hospitals because under the Hospital Readmissions Reduction Program, hospitals now face substantial penalties for higher than expected readmissions rates. Hospitals involved in bundled payments or other total cost-of-care arrangements have additional incentive to avoid readmissions. Beginning in 2019, SNFs will also receive incentive payments based on their 30-day all-cause hospital readmissions as part of the Skilled Nursing Facility Value-Based Purchasing program.²⁵ The Connected Care model offers 1 means of achieving this goal through partnership between hospitals and SNFs.

Our study has several limitations. First, our study was observational in nature, so the observed reduction in readmissions could have been due to temporal trends unrelated to the intervention. However, no significant reduction was noted during the same time period in other area SNFs. There was also little change in the characteristics of patients admitted to the intervention SNFs. Importantly, the HOSPITAL score, which can predict 30-day readmission rates,²⁰ did not change throughout the study period. Second, the results reflect patients discharged from a single hospital and may not be generalizable to other geographic areas. However, because the program included 7 SNFs, we believe it could be reproduced in other settings. Third, our readmissions measure included only those patients who returned to a CCHS facility. Although we may have missed some readmissions to other hospital systems, such leakage is uncommon—more than 80% of CCHS patients are readmitted to CCHS facilities—and would be unlikely to differ across the short duration of the study. Finally, at the intervention SNFs, most long-stay and some short-stay residents did not receive the Connected Care intervention because they were cared for by their own physicians who did not participate in Connected Care. Had these patients’ readmissions been excluded from our results, the intervention might appear even more effective.

CONCLUSION

A Connected Care intervention reduced 30-day readmission rates among patients discharged to SNFs from a tertiary academic center. While all subgroups had substantial reductions in readmissions following the implementation of the intervention, patients who are at the highest risk of readmission benefited the most. Further study is necessary to know whether Connected Care can be reproduced in other health care systems and whether it reduces overall costs.

Acknowledgments

The authors would like to thank Michael Felver, MD, and teams for their clinical care of patients; Michael Felver, MD, William Zafirau, MD, Dan Blechschmid, MHA, and Kathy Brezine, and Seth Vilensky, MBA, for their administrative support; and Brad Souder, MPT, for assistance with data collection.

Disclosure

Nothing to report.

Hospitalists and other inpatient providers are familiar with hospitalizations classified observation. The Centers for Medicare & Medicaid Services (CMS) uses the “2-midnight rule” to distinguish between outpatient services (which include all observation stays) and inpatient services for most hospitalizations. The rule states that “inpatient admissions will generally be payable … if the admitting practitioner expected the patient to require a hospital stay that crossed two midnights and the medical record supports that reasonable expectation.”¹

Hospitalization under inpatient versus outpatient status is a billing distinction that can have significant financial consequences for patients, providers, and hospitals. The inpatient or outpatient observation orders written by hospitalists and other hospital-based providers direct billing based on CMS and other third-party regulation. However, providers may have variable expertise writing such orders. To audit the correct use of the visit-status orders by hospital providers, CMS uses recovery auditors (RAs), also referred to as recovery audit contractors.^2,3

Historically, RAs had up to 3 years from date of service (DOS) to perform an audit, which involves asking a hospital for a medical record for a particular stay. The audit timeline includes 45 days for hospitals to produce such documentation, and 60 days for the RA either to agree with the hospital’s billing or to make an “overpayment determination” that the hospital should have billed Medicare Part B (outpatient) instead of Part A (inpatient).^3,4 The hospital may either accept the RA decision, or contest it by using the pre-appeals discussion period or by directly entering the 5-level Medicare administrative appeals process.^3,4 Level 1 and Level 2 appeals are heard by a government contractor, Level 3 by an administrative law judge (ALJ), Level 4 by a Medicare appeals council, and Level 5 by a federal district court. These different appeal types have different deadlines (Appendix 1). The deadlines for hospitals and government responses beyond Level 1 are set by Congress and enforced by CMS,^3,4 and CMS sets discussion period timelines. Hospitals that miss an appeals deadline automatically default their appeals request, but there are no penalties for missed government deadlines.

Recently, there has been increased scrutiny of the audit-and-appeals process of outpatient and inpatient status determinations.⁵ Despite the 2-midnight rule, the Medicare Benefit Policy Manual (MBPM) retains the passage: “Physicians should use a 24-hour period as a benchmark, i.e., they should order admission for patients who are expected to need hospital care for 24 hours or more, and treat other patients on an outpatient basis.”⁶ Auditors often cite “medical necessity” in their decisions, which is not well defined in the MBPM and can be open to different interpretation. This lack of clarity likely contributed to the large number of status determination discrepancies between providers and RAs, thereby creating a federal appeals backlog that caused the Office of Medicare Hearings and Appeals to halt hospital appeals assignments⁷ and prompted an ongoing lawsuit against CMS regarding the lengthy appeals process.⁴ To address these problems and clear the appeals backlog, CMS proposed a “$0.68 settlement offer.”⁴ The settlement “offered an administrative agreement to any hospital willing to withdraw their pending appeals in exchange for timely partial payment (68% of the net allowable amount)”⁸ and paid out almost $1.5 billion to the third of eligible hospitals that accepted the offer.⁹ CMS also made programmatic improvements to the RA program.¹⁰

Despite these efforts, problems remain. On June 9, 2016, the U.S. Government Accountability Office (GAO) published Medicare Fee-for-Service: Opportunities Remain to Improve Appeals Process, citing an approximate 2000% increase in hospital inpatient appeals during the period 2010–2014 and the concern that appeals requests will continue to exceed adjudication capabilities.¹¹ On July 5, 2016, CMS issued its proposed rule for appeals reform that allows the Medicare Appeals Council (Level 4) to set precedents which would be binding at lower levels and allows senior attorneys to handle some cases and effectively increase manpower at the Level 3 (ALJ). In addition, CMS proposes to revise the method for calculating dollars at risk needed to schedule an ALJ hearing, and develop methods to better adjudicate similar claims, and other process improvements aimed at decreasing the more than 750,000 current claims awaiting ALJ decisions.¹²

We conducted a study to better understand the Medicare appeals process in the context of the proposed CMS reforms by investigating all appeals reaching Level 3 at Johns Hopkins Hospital (JHH), University of Wisconsin Hospitals and Clinics (UWHC), and University of Utah Hospital (UU). Because relatively few cases nationally are appealed beyond Level 3, the study focused on most-relevant data.³ We examined time spent at each appeal Level and whether it met federally mandated deadlines, as well as the percentage accountable to hospitals versus government contractors or ALJs. We also recorded the overturn rate at Level 3 and evaluated standardized text in de-identified decision letters to determine criteria cited by contractors in their decisions to deny hospital appeal requests.

The JHH, UWHC, and UU Institutional Review Boards did not require a review. The study included all complex Part A appeals involving DOS before October 1, 2013 and reaching Level 3 (ALJ) as of May 1, 2016.

Our general methods were described previously.² Briefly, the 3 academic medical centers are geographically diverse. JHH is in region A, UWHC in region B, and UU in region D (3 of the 4 RA regions are represented). The hospitals had different Medicare administrative contractors but the same qualified independent contractor until March 1, 2015 (Appendix 2).

Of 219 Level 3 cases, 135 (61.6%) concluded at Level 3. Of these 135 cases, 96 (71.1%) were decided in favor of the hospital, 11 (8.1%) were settled in the CMS $0.68 settlement offer, and 28 (20.7%) were unfavorable to the hospital (Table 1).

Mean total days since DOS was 1,663.3 (536.8) (mean [SD]), with median 1708 days. This included 560.4 (351.6) days between DOS and audit (median 556 days) and 891.3 (320.3) days in appeal (median 979 days). The hospitals were responsible for 29.3% of that time (260.7 [68.2] days) while government contractors were responsible for 70.7% (630.6 [277.2] days). Government contractors and ALJs met deadlines 47.7% of the time, meeting appeals deadlines 92.5% of the time for Discussion, 85.4% for Level 1, 38.8% for Level 2, and 0% for Level 3 (Table 1).

All “redetermination” (level 1 appeals letters) received at UU and UWHC, and all “reconsideration” (level 2 appeals letters) received by UU, UWHC, and JHH contained standardized time-based 24–hour benchmark text directly or referencing the MBPM containing such text, to describe criteria for inpatient status (Table 2 and Appendix 3).⁶ In total, 417 of 438 (95.2%) of Level 1 and Level 2 appeals results letters contained time-based 24-hour benchmark criteria for inpatient status despite 154 of 219 (70.3%) of denied cases exceeding a 24-hour LOS.

This study demonstrated process and timeliness concerns in the Medicare RA program for Level 3 cases at 3 academic medical centers. Although hospitals forfeit any appeal for which they miss a filing deadline, government contractors and ALJs met their deadlines less than half the time without default or penalty. Average time from the rendering of services to the conclusion of the audit-and-appeals process exceeded 4.5 years, which included an average 560 days between hospital stay and initial RA audit, and almost 900 days in appeals, with more than 70% of that time attributable to government contractors and ALJs.

Objective time-based 24-hour inpatient status criteria were referenced in 95% of decision letters, even though LOS exceeded 24 hours in more than 70% of these cases, suggesting that objective LOS data played only a small role in contractor decisions, or that contractors did not actually audit for LOS when reviewing cases. Unclear criteria likely contributed to payment denials and improper payments, despite admitting providers’ best efforts to comply with Medicare rules when writing visit-status orders. There was also a significant cost to hospitals; our prior study found that navigating the appeals process required 5 full-time equivalents per institution.²

At the 2 study hospitals with Level 3 decisions, more than two thirds of the decisions favored the hospital, suggesting the hospitals were justified in appealing RA Level 1 and Level 2 determinations. This proportion is consistent with the 43% ALJ overturn rate (including RA- and non-RA-derived appeals) cited in the recent U.S. Court of Appeals for the DC Circuit decision.⁹

This study potentially was limited by contractor and hospital use of the nonstandardized LOS calculation during the study period. That the majority of JHH and UU cases cited the 24-hour benchmark in their letters but nevertheless exceeded 24-hour LOS (using the most conservative definition of LOS) suggests contractors did not audit for or consider LOS in their decisions.

Our results support recent steps taken by CMS to reform the appeals process, including shortening the RA “look-back period” from 3 years to 6 months,¹⁰ which will markedly shorten the 560-day lag between DOS and audit found in this study. In addition, CMS has replaced RAs with beneficiary and family-centered care quality improvement organizations (BFCC-QIOs)^1,8 for initial status determination audits. Although it is too soon to tell, the hope is that BFCC-QIOs will decrease the volume of audits and denials that have overwhelmed the system and most probably contributed to process delays and the appeals backlog.

However, our data demonstrate several areas of concern not addressed in the recent GAO report¹¹ or in the rule proposed by CMS.¹² Most important, CMS could consider an appeals deadline missed by a government contractor as a decision for the hospital, in the same way a hospital’s missed deadline defaults its appeal. Such equity would ensure due process and prevent another appeals backlog. In addition, the large number of Level 3 decisions favoring hospitals suggests a need for process improvement at the Medicare administrative contractor and qualified independent contractor Level of appeals—such as mandatory review of Level 1 and Level 2 decision letters for appeals overturned at Level 3, accountability for Level 1 and Level 2 contractors with high rates of Level 3 overturn, and clarification of criteria used to judge determinations.

Medicare fraud cannot be tolerated, and a robust auditing process is essential to the integrity of the Medicare program. CMS’s current and proposed reforms may not be enough to eliminate the appeals backlog and restore a timely and fair appeals process. As CMS explores bundled payments and other reimbursement reforms, perhaps the need to distinguish observation hospital care will be eliminated. Short of that, additional actions must be taken so that a just and efficient Medicare appeals system can be realized for observation hospitalizations.

Acknowledgments

For invaluable assistance in data preparation and presentation, the authors thank Becky Borchert, RN, MS, MBA, Program Manager for Medicare/Medicaid Utilization Review, University of Wisconsin Hospital and Clinics; Carol Duhaney, Calvin Young, and Joan Kratz, RN, Johns Hopkins Hospital; and Morgan Walker and Lisa Whittaker, RN, University of Utah.

Disclosure

Nothing to report.

Hospitalists and other inpatient providers are familiar with hospitalizations classified observation. The Centers for Medicare & Medicaid Services (CMS) uses the “2-midnight rule” to distinguish between outpatient services (which include all observation stays) and inpatient services for most hospitalizations. The rule states that “inpatient admissions will generally be payable … if the admitting practitioner expected the patient to require a hospital stay that crossed two midnights and the medical record supports that reasonable expectation.”¹

Hospitalization under inpatient versus outpatient status is a billing distinction that can have significant financial consequences for patients, providers, and hospitals. The inpatient or outpatient observation orders written by hospitalists and other hospital-based providers direct billing based on CMS and other third-party regulation. However, providers may have variable expertise writing such orders. To audit the correct use of the visit-status orders by hospital providers, CMS uses recovery auditors (RAs), also referred to as recovery audit contractors.^2,3

Historically, RAs had up to 3 years from date of service (DOS) to perform an audit, which involves asking a hospital for a medical record for a particular stay. The audit timeline includes 45 days for hospitals to produce such documentation, and 60 days for the RA either to agree with the hospital’s billing or to make an “overpayment determination” that the hospital should have billed Medicare Part B (outpatient) instead of Part A (inpatient).^3,4 The hospital may either accept the RA decision, or contest it by using the pre-appeals discussion period or by directly entering the 5-level Medicare administrative appeals process.^3,4 Level 1 and Level 2 appeals are heard by a government contractor, Level 3 by an administrative law judge (ALJ), Level 4 by a Medicare appeals council, and Level 5 by a federal district court. These different appeal types have different deadlines (Appendix 1). The deadlines for hospitals and government responses beyond Level 1 are set by Congress and enforced by CMS,^3,4 and CMS sets discussion period timelines. Hospitals that miss an appeals deadline automatically default their appeals request, but there are no penalties for missed government deadlines.

Recently, there has been increased scrutiny of the audit-and-appeals process of outpatient and inpatient status determinations.⁵ Despite the 2-midnight rule, the Medicare Benefit Policy Manual (MBPM) retains the passage: “Physicians should use a 24-hour period as a benchmark, i.e., they should order admission for patients who are expected to need hospital care for 24 hours or more, and treat other patients on an outpatient basis.”⁶ Auditors often cite “medical necessity” in their decisions, which is not well defined in the MBPM and can be open to different interpretation. This lack of clarity likely contributed to the large number of status determination discrepancies between providers and RAs, thereby creating a federal appeals backlog that caused the Office of Medicare Hearings and Appeals to halt hospital appeals assignments⁷ and prompted an ongoing lawsuit against CMS regarding the lengthy appeals process.⁴ To address these problems and clear the appeals backlog, CMS proposed a “$0.68 settlement offer.”⁴ The settlement “offered an administrative agreement to any hospital willing to withdraw their pending appeals in exchange for timely partial payment (68% of the net allowable amount)”⁸ and paid out almost $1.5 billion to the third of eligible hospitals that accepted the offer.⁹ CMS also made programmatic improvements to the RA program.¹⁰

Despite these efforts, problems remain. On June 9, 2016, the U.S. Government Accountability Office (GAO) published Medicare Fee-for-Service: Opportunities Remain to Improve Appeals Process, citing an approximate 2000% increase in hospital inpatient appeals during the period 2010–2014 and the concern that appeals requests will continue to exceed adjudication capabilities.¹¹ On July 5, 2016, CMS issued its proposed rule for appeals reform that allows the Medicare Appeals Council (Level 4) to set precedents which would be binding at lower levels and allows senior attorneys to handle some cases and effectively increase manpower at the Level 3 (ALJ). In addition, CMS proposes to revise the method for calculating dollars at risk needed to schedule an ALJ hearing, and develop methods to better adjudicate similar claims, and other process improvements aimed at decreasing the more than 750,000 current claims awaiting ALJ decisions.¹²

We conducted a study to better understand the Medicare appeals process in the context of the proposed CMS reforms by investigating all appeals reaching Level 3 at Johns Hopkins Hospital (JHH), University of Wisconsin Hospitals and Clinics (UWHC), and University of Utah Hospital (UU). Because relatively few cases nationally are appealed beyond Level 3, the study focused on most-relevant data.³ We examined time spent at each appeal Level and whether it met federally mandated deadlines, as well as the percentage accountable to hospitals versus government contractors or ALJs. We also recorded the overturn rate at Level 3 and evaluated standardized text in de-identified decision letters to determine criteria cited by contractors in their decisions to deny hospital appeal requests.

The JHH, UWHC, and UU Institutional Review Boards did not require a review. The study included all complex Part A appeals involving DOS before October 1, 2013 and reaching Level 3 (ALJ) as of May 1, 2016.

Our general methods were described previously.² Briefly, the 3 academic medical centers are geographically diverse. JHH is in region A, UWHC in region B, and UU in region D (3 of the 4 RA regions are represented). The hospitals had different Medicare administrative contractors but the same qualified independent contractor until March 1, 2015 (Appendix 2).

Of 219 Level 3 cases, 135 (61.6%) concluded at Level 3. Of these 135 cases, 96 (71.1%) were decided in favor of the hospital, 11 (8.1%) were settled in the CMS $0.68 settlement offer, and 28 (20.7%) were unfavorable to the hospital (Table 1).

Mean total days since DOS was 1,663.3 (536.8) (mean [SD]), with median 1708 days. This included 560.4 (351.6) days between DOS and audit (median 556 days) and 891.3 (320.3) days in appeal (median 979 days). The hospitals were responsible for 29.3% of that time (260.7 [68.2] days) while government contractors were responsible for 70.7% (630.6 [277.2] days). Government contractors and ALJs met deadlines 47.7% of the time, meeting appeals deadlines 92.5% of the time for Discussion, 85.4% for Level 1, 38.8% for Level 2, and 0% for Level 3 (Table 1).

All “redetermination” (level 1 appeals letters) received at UU and UWHC, and all “reconsideration” (level 2 appeals letters) received by UU, UWHC, and JHH contained standardized time-based 24–hour benchmark text directly or referencing the MBPM containing such text, to describe criteria for inpatient status (Table 2 and Appendix 3).⁶ In total, 417 of 438 (95.2%) of Level 1 and Level 2 appeals results letters contained time-based 24-hour benchmark criteria for inpatient status despite 154 of 219 (70.3%) of denied cases exceeding a 24-hour LOS.

This study demonstrated process and timeliness concerns in the Medicare RA program for Level 3 cases at 3 academic medical centers. Although hospitals forfeit any appeal for which they miss a filing deadline, government contractors and ALJs met their deadlines less than half the time without default or penalty. Average time from the rendering of services to the conclusion of the audit-and-appeals process exceeded 4.5 years, which included an average 560 days between hospital stay and initial RA audit, and almost 900 days in appeals, with more than 70% of that time attributable to government contractors and ALJs.

Objective time-based 24-hour inpatient status criteria were referenced in 95% of decision letters, even though LOS exceeded 24 hours in more than 70% of these cases, suggesting that objective LOS data played only a small role in contractor decisions, or that contractors did not actually audit for LOS when reviewing cases. Unclear criteria likely contributed to payment denials and improper payments, despite admitting providers’ best efforts to comply with Medicare rules when writing visit-status orders. There was also a significant cost to hospitals; our prior study found that navigating the appeals process required 5 full-time equivalents per institution.²

At the 2 study hospitals with Level 3 decisions, more than two thirds of the decisions favored the hospital, suggesting the hospitals were justified in appealing RA Level 1 and Level 2 determinations. This proportion is consistent with the 43% ALJ overturn rate (including RA- and non-RA-derived appeals) cited in the recent U.S. Court of Appeals for the DC Circuit decision.⁹

This study potentially was limited by contractor and hospital use of the nonstandardized LOS calculation during the study period. That the majority of JHH and UU cases cited the 24-hour benchmark in their letters but nevertheless exceeded 24-hour LOS (using the most conservative definition of LOS) suggests contractors did not audit for or consider LOS in their decisions.

Our results support recent steps taken by CMS to reform the appeals process, including shortening the RA “look-back period” from 3 years to 6 months,¹⁰ which will markedly shorten the 560-day lag between DOS and audit found in this study. In addition, CMS has replaced RAs with beneficiary and family-centered care quality improvement organizations (BFCC-QIOs)^1,8 for initial status determination audits. Although it is too soon to tell, the hope is that BFCC-QIOs will decrease the volume of audits and denials that have overwhelmed the system and most probably contributed to process delays and the appeals backlog.

However, our data demonstrate several areas of concern not addressed in the recent GAO report¹¹ or in the rule proposed by CMS.¹² Most important, CMS could consider an appeals deadline missed by a government contractor as a decision for the hospital, in the same way a hospital’s missed deadline defaults its appeal. Such equity would ensure due process and prevent another appeals backlog. In addition, the large number of Level 3 decisions favoring hospitals suggests a need for process improvement at the Medicare administrative contractor and qualified independent contractor Level of appeals—such as mandatory review of Level 1 and Level 2 decision letters for appeals overturned at Level 3, accountability for Level 1 and Level 2 contractors with high rates of Level 3 overturn, and clarification of criteria used to judge determinations.

Medicare fraud cannot be tolerated, and a robust auditing process is essential to the integrity of the Medicare program. CMS’s current and proposed reforms may not be enough to eliminate the appeals backlog and restore a timely and fair appeals process. As CMS explores bundled payments and other reimbursement reforms, perhaps the need to distinguish observation hospital care will be eliminated. Short of that, additional actions must be taken so that a just and efficient Medicare appeals system can be realized for observation hospitalizations.

Acknowledgments

For invaluable assistance in data preparation and presentation, the authors thank Becky Borchert, RN, MS, MBA, Program Manager for Medicare/Medicaid Utilization Review, University of Wisconsin Hospital and Clinics; Carol Duhaney, Calvin Young, and Joan Kratz, RN, Johns Hopkins Hospital; and Morgan Walker and Lisa Whittaker, RN, University of Utah.

Disclosure

Nothing to report.

Hospitalists and other inpatient providers are familiar with hospitalizations classified observation. The Centers for Medicare & Medicaid Services (CMS) uses the “2-midnight rule” to distinguish between outpatient services (which include all observation stays) and inpatient services for most hospitalizations. The rule states that “inpatient admissions will generally be payable … if the admitting practitioner expected the patient to require a hospital stay that crossed two midnights and the medical record supports that reasonable expectation.”¹

Hospitalization under inpatient versus outpatient status is a billing distinction that can have significant financial consequences for patients, providers, and hospitals. The inpatient or outpatient observation orders written by hospitalists and other hospital-based providers direct billing based on CMS and other third-party regulation. However, providers may have variable expertise writing such orders. To audit the correct use of the visit-status orders by hospital providers, CMS uses recovery auditors (RAs), also referred to as recovery audit contractors.^2,3

Historically, RAs had up to 3 years from date of service (DOS) to perform an audit, which involves asking a hospital for a medical record for a particular stay. The audit timeline includes 45 days for hospitals to produce such documentation, and 60 days for the RA either to agree with the hospital’s billing or to make an “overpayment determination” that the hospital should have billed Medicare Part B (outpatient) instead of Part A (inpatient).^3,4 The hospital may either accept the RA decision, or contest it by using the pre-appeals discussion period or by directly entering the 5-level Medicare administrative appeals process.^3,4 Level 1 and Level 2 appeals are heard by a government contractor, Level 3 by an administrative law judge (ALJ), Level 4 by a Medicare appeals council, and Level 5 by a federal district court. These different appeal types have different deadlines (Appendix 1). The deadlines for hospitals and government responses beyond Level 1 are set by Congress and enforced by CMS,^3,4 and CMS sets discussion period timelines. Hospitals that miss an appeals deadline automatically default their appeals request, but there are no penalties for missed government deadlines.

Recently, there has been increased scrutiny of the audit-and-appeals process of outpatient and inpatient status determinations.⁵ Despite the 2-midnight rule, the Medicare Benefit Policy Manual (MBPM) retains the passage: “Physicians should use a 24-hour period as a benchmark, i.e., they should order admission for patients who are expected to need hospital care for 24 hours or more, and treat other patients on an outpatient basis.”⁶ Auditors often cite “medical necessity” in their decisions, which is not well defined in the MBPM and can be open to different interpretation. This lack of clarity likely contributed to the large number of status determination discrepancies between providers and RAs, thereby creating a federal appeals backlog that caused the Office of Medicare Hearings and Appeals to halt hospital appeals assignments⁷ and prompted an ongoing lawsuit against CMS regarding the lengthy appeals process.⁴ To address these problems and clear the appeals backlog, CMS proposed a “$0.68 settlement offer.”⁴ The settlement “offered an administrative agreement to any hospital willing to withdraw their pending appeals in exchange for timely partial payment (68% of the net allowable amount)”⁸ and paid out almost $1.5 billion to the third of eligible hospitals that accepted the offer.⁹ CMS also made programmatic improvements to the RA program.¹⁰

Despite these efforts, problems remain. On June 9, 2016, the U.S. Government Accountability Office (GAO) published Medicare Fee-for-Service: Opportunities Remain to Improve Appeals Process, citing an approximate 2000% increase in hospital inpatient appeals during the period 2010–2014 and the concern that appeals requests will continue to exceed adjudication capabilities.¹¹ On July 5, 2016, CMS issued its proposed rule for appeals reform that allows the Medicare Appeals Council (Level 4) to set precedents which would be binding at lower levels and allows senior attorneys to handle some cases and effectively increase manpower at the Level 3 (ALJ). In addition, CMS proposes to revise the method for calculating dollars at risk needed to schedule an ALJ hearing, and develop methods to better adjudicate similar claims, and other process improvements aimed at decreasing the more than 750,000 current claims awaiting ALJ decisions.¹²

We conducted a study to better understand the Medicare appeals process in the context of the proposed CMS reforms by investigating all appeals reaching Level 3 at Johns Hopkins Hospital (JHH), University of Wisconsin Hospitals and Clinics (UWHC), and University of Utah Hospital (UU). Because relatively few cases nationally are appealed beyond Level 3, the study focused on most-relevant data.³ We examined time spent at each appeal Level and whether it met federally mandated deadlines, as well as the percentage accountable to hospitals versus government contractors or ALJs. We also recorded the overturn rate at Level 3 and evaluated standardized text in de-identified decision letters to determine criteria cited by contractors in their decisions to deny hospital appeal requests.

The JHH, UWHC, and UU Institutional Review Boards did not require a review. The study included all complex Part A appeals involving DOS before October 1, 2013 and reaching Level 3 (ALJ) as of May 1, 2016.

Our general methods were described previously.² Briefly, the 3 academic medical centers are geographically diverse. JHH is in region A, UWHC in region B, and UU in region D (3 of the 4 RA regions are represented). The hospitals had different Medicare administrative contractors but the same qualified independent contractor until March 1, 2015 (Appendix 2).

Of 219 Level 3 cases, 135 (61.6%) concluded at Level 3. Of these 135 cases, 96 (71.1%) were decided in favor of the hospital, 11 (8.1%) were settled in the CMS $0.68 settlement offer, and 28 (20.7%) were unfavorable to the hospital (Table 1).

Mean total days since DOS was 1,663.3 (536.8) (mean [SD]), with median 1708 days. This included 560.4 (351.6) days between DOS and audit (median 556 days) and 891.3 (320.3) days in appeal (median 979 days). The hospitals were responsible for 29.3% of that time (260.7 [68.2] days) while government contractors were responsible for 70.7% (630.6 [277.2] days). Government contractors and ALJs met deadlines 47.7% of the time, meeting appeals deadlines 92.5% of the time for Discussion, 85.4% for Level 1, 38.8% for Level 2, and 0% for Level 3 (Table 1).

All “redetermination” (level 1 appeals letters) received at UU and UWHC, and all “reconsideration” (level 2 appeals letters) received by UU, UWHC, and JHH contained standardized time-based 24–hour benchmark text directly or referencing the MBPM containing such text, to describe criteria for inpatient status (Table 2 and Appendix 3).⁶ In total, 417 of 438 (95.2%) of Level 1 and Level 2 appeals results letters contained time-based 24-hour benchmark criteria for inpatient status despite 154 of 219 (70.3%) of denied cases exceeding a 24-hour LOS.

This study demonstrated process and timeliness concerns in the Medicare RA program for Level 3 cases at 3 academic medical centers. Although hospitals forfeit any appeal for which they miss a filing deadline, government contractors and ALJs met their deadlines less than half the time without default or penalty. Average time from the rendering of services to the conclusion of the audit-and-appeals process exceeded 4.5 years, which included an average 560 days between hospital stay and initial RA audit, and almost 900 days in appeals, with more than 70% of that time attributable to government contractors and ALJs.

Objective time-based 24-hour inpatient status criteria were referenced in 95% of decision letters, even though LOS exceeded 24 hours in more than 70% of these cases, suggesting that objective LOS data played only a small role in contractor decisions, or that contractors did not actually audit for LOS when reviewing cases. Unclear criteria likely contributed to payment denials and improper payments, despite admitting providers’ best efforts to comply with Medicare rules when writing visit-status orders. There was also a significant cost to hospitals; our prior study found that navigating the appeals process required 5 full-time equivalents per institution.²

At the 2 study hospitals with Level 3 decisions, more than two thirds of the decisions favored the hospital, suggesting the hospitals were justified in appealing RA Level 1 and Level 2 determinations. This proportion is consistent with the 43% ALJ overturn rate (including RA- and non-RA-derived appeals) cited in the recent U.S. Court of Appeals for the DC Circuit decision.⁹

This study potentially was limited by contractor and hospital use of the nonstandardized LOS calculation during the study period. That the majority of JHH and UU cases cited the 24-hour benchmark in their letters but nevertheless exceeded 24-hour LOS (using the most conservative definition of LOS) suggests contractors did not audit for or consider LOS in their decisions.

Our results support recent steps taken by CMS to reform the appeals process, including shortening the RA “look-back period” from 3 years to 6 months,¹⁰ which will markedly shorten the 560-day lag between DOS and audit found in this study. In addition, CMS has replaced RAs with beneficiary and family-centered care quality improvement organizations (BFCC-QIOs)^1,8 for initial status determination audits. Although it is too soon to tell, the hope is that BFCC-QIOs will decrease the volume of audits and denials that have overwhelmed the system and most probably contributed to process delays and the appeals backlog.

However, our data demonstrate several areas of concern not addressed in the recent GAO report¹¹ or in the rule proposed by CMS.¹² Most important, CMS could consider an appeals deadline missed by a government contractor as a decision for the hospital, in the same way a hospital’s missed deadline defaults its appeal. Such equity would ensure due process and prevent another appeals backlog. In addition, the large number of Level 3 decisions favoring hospitals suggests a need for process improvement at the Medicare administrative contractor and qualified independent contractor Level of appeals—such as mandatory review of Level 1 and Level 2 decision letters for appeals overturned at Level 3, accountability for Level 1 and Level 2 contractors with high rates of Level 3 overturn, and clarification of criteria used to judge determinations.

Medicare fraud cannot be tolerated, and a robust auditing process is essential to the integrity of the Medicare program. CMS’s current and proposed reforms may not be enough to eliminate the appeals backlog and restore a timely and fair appeals process. As CMS explores bundled payments and other reimbursement reforms, perhaps the need to distinguish observation hospital care will be eliminated. Short of that, additional actions must be taken so that a just and efficient Medicare appeals system can be realized for observation hospitalizations.

Acknowledgments

For invaluable assistance in data preparation and presentation, the authors thank Becky Borchert, RN, MS, MBA, Program Manager for Medicare/Medicaid Utilization Review, University of Wisconsin Hospital and Clinics; Carol Duhaney, Calvin Young, and Joan Kratz, RN, Johns Hopkins Hospital; and Morgan Walker and Lisa Whittaker, RN, University of Utah.

Disclosure

Nothing to report.

WASHINGTON – Patients who experience an acute MI at age 65 or older have unsettlingly high 5- and 10-year mortality in community practice settings despite excellent rates of evidence-based medications being prescribed at discharge, Ajar Kochar, MD, reported at the annual meeting of the American College of Cardiology.

This observation is based upon more than 22,000 patients aged 65 years or older treated for an acute MI during 2004-2006 at 344 U.S. hospitals participating in the CRUSADE registry. Their median age at the time of MI was 77 years. But 10-year all-cause mortality remained high even among relatively younger patients aged 65-74 whom one would expect to have a favorable long-term prognosis because they had additional survival-enhancing factors working in their favor, including having undergone coronary revascularization during their index hospitalization and surviving their first year post-MI, observed Dr. Kochar of the Duke Clinical Research Institute in Durham, N.C.

Bruce Jancin/Frontline Medical News

• Unadjusted 10-year all-cause mortality was significantly greater in patients with NSTEMI than STEMI, by a margin of 73% versus 60%. Notably, however, NSTEMI patients were far less likely to undergo coronary revascularization: 32% of them had percutaneous coronary intervention during their index hospitalization, and 8.7% underwent coronary artery bypass grafting, in contrast to rates of 65.5% for PCI and 8.0% for CABG in the STEMI patients. After adjustment for these and other differences in care, NSTEMI patients actually had a 7% lower risk of long-term mortality than the STEMI group.

• Even after limiting the analysis to the youngest elderly – patients aged 65-74 when their MI occurred – 10-year mortality remained high, at 53%.

• After excluding the 24% of patients who died within 1 year after MI, 10-year mortality was still quite high, at 63%. Dr. Kochar and his coinvestigators chose to reanalyze the data in this way because the 1-year mark is an important time point clinically, since it’s when decisions regarding extended dual-antiplatelet therapy are made.

Patients who underwent coronary revascularization during their index hospitalization had a much-improved long-term prognosis, compared with those with medical management only. The 10-year cumulative mortality rate was 57% in patients who had PCI, identical at 57% in those who received CABG, and 84% in medically managed patients.

Ninety-five percent of patients were discharged on aspirin, 94% on a beta blocker, 81% on a statin, and 73% on clopidogrel. Discharge prescriptions for statins and clopidogrel were more common for the STEMI group. Unfortunately, the CRUSADE registry doesn’t include data on long-term medication adherence or prescription refill rates.

Dr. Kochar named several potential strategies aimed at reducing the high long-term mortality rates in older patients with MI as documented in this study. These include structured efforts to improve adherence to evidence-based medications for secondary prevention, as well as making percutaneous revascularization more widely available for older patients with NSTEMI. He noted that while in 2004-2006, 32% of CRUSADE participants with NSTEMI underwent PCI during their index hospitalization, by 2011-2012 that rate had inched upward only to 36%.

Several physicians commented that the high long-term all-cause mortality rates in older CRUSADE participants may paint a grim picture, in part because the aged face growing risks of cancer and other noncardiovascular competing causes of death. But Dr. Kochar replied that while the lack of information on specific causes of death is a study limitation, he and his coinvestigators are convinced based upon data from other studies that most of the deaths in CRUSADE were cardiovascular in nature.

He reported having no financial conflicts regarding his study.

[email protected]

WASHINGTON – Patients who experience an acute MI at age 65 or older have unsettlingly high 5- and 10-year mortality in community practice settings despite excellent rates of evidence-based medications being prescribed at discharge, Ajar Kochar, MD, reported at the annual meeting of the American College of Cardiology.

This observation is based upon more than 22,000 patients aged 65 years or older treated for an acute MI during 2004-2006 at 344 U.S. hospitals participating in the CRUSADE registry. Their median age at the time of MI was 77 years. But 10-year all-cause mortality remained high even among relatively younger patients aged 65-74 whom one would expect to have a favorable long-term prognosis because they had additional survival-enhancing factors working in their favor, including having undergone coronary revascularization during their index hospitalization and surviving their first year post-MI, observed Dr. Kochar of the Duke Clinical Research Institute in Durham, N.C.

Bruce Jancin/Frontline Medical News

• Unadjusted 10-year all-cause mortality was significantly greater in patients with NSTEMI than STEMI, by a margin of 73% versus 60%. Notably, however, NSTEMI patients were far less likely to undergo coronary revascularization: 32% of them had percutaneous coronary intervention during their index hospitalization, and 8.7% underwent coronary artery bypass grafting, in contrast to rates of 65.5% for PCI and 8.0% for CABG in the STEMI patients. After adjustment for these and other differences in care, NSTEMI patients actually had a 7% lower risk of long-term mortality than the STEMI group.

• Even after limiting the analysis to the youngest elderly – patients aged 65-74 when their MI occurred – 10-year mortality remained high, at 53%.

• After excluding the 24% of patients who died within 1 year after MI, 10-year mortality was still quite high, at 63%. Dr. Kochar and his coinvestigators chose to reanalyze the data in this way because the 1-year mark is an important time point clinically, since it’s when decisions regarding extended dual-antiplatelet therapy are made.

Patients who underwent coronary revascularization during their index hospitalization had a much-improved long-term prognosis, compared with those with medical management only. The 10-year cumulative mortality rate was 57% in patients who had PCI, identical at 57% in those who received CABG, and 84% in medically managed patients.

Ninety-five percent of patients were discharged on aspirin, 94% on a beta blocker, 81% on a statin, and 73% on clopidogrel. Discharge prescriptions for statins and clopidogrel were more common for the STEMI group. Unfortunately, the CRUSADE registry doesn’t include data on long-term medication adherence or prescription refill rates.

Dr. Kochar named several potential strategies aimed at reducing the high long-term mortality rates in older patients with MI as documented in this study. These include structured efforts to improve adherence to evidence-based medications for secondary prevention, as well as making percutaneous revascularization more widely available for older patients with NSTEMI. He noted that while in 2004-2006, 32% of CRUSADE participants with NSTEMI underwent PCI during their index hospitalization, by 2011-2012 that rate had inched upward only to 36%.

Several physicians commented that the high long-term all-cause mortality rates in older CRUSADE participants may paint a grim picture, in part because the aged face growing risks of cancer and other noncardiovascular competing causes of death. But Dr. Kochar replied that while the lack of information on specific causes of death is a study limitation, he and his coinvestigators are convinced based upon data from other studies that most of the deaths in CRUSADE were cardiovascular in nature.

He reported having no financial conflicts regarding his study.

[email protected]

WASHINGTON – Patients who experience an acute MI at age 65 or older have unsettlingly high 5- and 10-year mortality in community practice settings despite excellent rates of evidence-based medications being prescribed at discharge, Ajar Kochar, MD, reported at the annual meeting of the American College of Cardiology.

This observation is based upon more than 22,000 patients aged 65 years or older treated for an acute MI during 2004-2006 at 344 U.S. hospitals participating in the CRUSADE registry. Their median age at the time of MI was 77 years. But 10-year all-cause mortality remained high even among relatively younger patients aged 65-74 whom one would expect to have a favorable long-term prognosis because they had additional survival-enhancing factors working in their favor, including having undergone coronary revascularization during their index hospitalization and surviving their first year post-MI, observed Dr. Kochar of the Duke Clinical Research Institute in Durham, N.C.

Bruce Jancin/Frontline Medical News

• Unadjusted 10-year all-cause mortality was significantly greater in patients with NSTEMI than STEMI, by a margin of 73% versus 60%. Notably, however, NSTEMI patients were far less likely to undergo coronary revascularization: 32% of them had percutaneous coronary intervention during their index hospitalization, and 8.7% underwent coronary artery bypass grafting, in contrast to rates of 65.5% for PCI and 8.0% for CABG in the STEMI patients. After adjustment for these and other differences in care, NSTEMI patients actually had a 7% lower risk of long-term mortality than the STEMI group.

• Even after limiting the analysis to the youngest elderly – patients aged 65-74 when their MI occurred – 10-year mortality remained high, at 53%.

• After excluding the 24% of patients who died within 1 year after MI, 10-year mortality was still quite high, at 63%. Dr. Kochar and his coinvestigators chose to reanalyze the data in this way because the 1-year mark is an important time point clinically, since it’s when decisions regarding extended dual-antiplatelet therapy are made.

Patients who underwent coronary revascularization during their index hospitalization had a much-improved long-term prognosis, compared with those with medical management only. The 10-year cumulative mortality rate was 57% in patients who had PCI, identical at 57% in those who received CABG, and 84% in medically managed patients.

Ninety-five percent of patients were discharged on aspirin, 94% on a beta blocker, 81% on a statin, and 73% on clopidogrel. Discharge prescriptions for statins and clopidogrel were more common for the STEMI group. Unfortunately, the CRUSADE registry doesn’t include data on long-term medication adherence or prescription refill rates.

Dr. Kochar named several potential strategies aimed at reducing the high long-term mortality rates in older patients with MI as documented in this study. These include structured efforts to improve adherence to evidence-based medications for secondary prevention, as well as making percutaneous revascularization more widely available for older patients with NSTEMI. He noted that while in 2004-2006, 32% of CRUSADE participants with NSTEMI underwent PCI during their index hospitalization, by 2011-2012 that rate had inched upward only to 36%.

Several physicians commented that the high long-term all-cause mortality rates in older CRUSADE participants may paint a grim picture, in part because the aged face growing risks of cancer and other noncardiovascular competing causes of death. But Dr. Kochar replied that while the lack of information on specific causes of death is a study limitation, he and his coinvestigators are convinced based upon data from other studies that most of the deaths in CRUSADE were cardiovascular in nature.

He reported having no financial conflicts regarding his study.

[email protected]

Sepsis is a leading cause of hospital mortality in the United States, contributing to up to half of all deaths.¹ If the infection is identified and treated early, however, its associated morbidity and mortality can be significantly reduced.² The 2001 sepsis guidelines define sepsis as the suspicion of infection plus meeting 2 or more systemic inflammatory response syndrome (SIRS) criteria.³ Although the utility of SIRS criteria has been extensively debated, providers’ accuracy and agreement regarding suspicion of infection are not yet fully characterized. This is very important, as the source of infection is often not identified in patients with severe sepsis or septic shock.⁴

Although much attention recently has been given to ideal objective criteria for accurately identifying sepsis, less is known about what constitutes ideal subjective criteria and who can best make that assessment.^5-7 We conducted a study to measure providers’ agreement regarding this subjective assessment and the impact of that agreement on patient outcomes.

We performed a secondary analysis of prospectively collected data on consecutive adults hospitalized on a general medicine ward at an academic medical center between April 1, 2014 and March 31, 2015. This study was approved by the University of Chicago Institutional Review Board with a waiver of consent.

A sepsis screening tool was developed locally as part of the Surviving Sepsis Campaign Quality Improvement Learning Collaborative⁸ (Supplemental Figure). This tool was completed by bedside nurses for each patient during each shift. Bedside registered nurse (RN) suspicion of infection was deemed positive if the nurse answered yes to question 2: “Does the patient have evidence of an active infection?” We compared RN assessment with assessment by the ordering provider, a medical doctor or advanced practice professionals (MD/APP), using an existing order for antibiotics or a new order for either blood or urine cultures placed within 12 hours before nursing screen time to indicate MD/APP suspicion of infection.

All nursing screens were transcribed into an electronic database, excluding screens not performed, or missing RN suspicion of infection. For quality purposes, screening data were merged with electronic health record data to verify SIRS criteria at the time of the screens as well as the presence of culture and/or antibiotic orders preceding the screens. Outcome data were obtained from an administrative database and confirmed by chart review using the 2001 sepsis definitions.⁶ Data were de-identified and time-shifted before this analysis. SIRS-positive criteria were defined as meeting 2 or more of the following: temperature higher than 38°C or lower than 36°C; heart rate higher than 90 beats per minute; respiratory rate more than 20 breaths per minute; and white blood cell count more than 2,000/mm³ or less than 4,000/mm³.The primary clinical outcome was progression to severe sepsis or septic shock. Secondary outcomes included transfer to intensive care unit (ICU) and in-hospital mortality. Given that RN and MD/APP suspicion of infection can vary over time, only the initial screen for each patient was used in assessing progression to severe sepsis or septic shock and in-hospital mortality. All available screens were used to investigate the association between each provider’s suspicion of infection over time and ICU transfer.

Demographic characteristics were compared using the χ² test and analysis of variance, as appropriate. Provider agreement was evaluated with a weighted κ statistic. Fisher exact tests were used to compare proportions of mortality and severe sepsis/septic shock, and the McNemar test was used to compare proportions of ICU transfers. The association of outcomes based on provider agreement was evaluated with a nonparametric test for trend.

During the study period, 1386 distinct patients had 13,223 screening opportunities, with a 95.4% compliance rate. A total of 1127 screens were excluded for missing nursing documentation of suspicion of infection, leaving 1192 first screens and 11,489 total screens for analysis. Of the completed screens, 3744 (32.6%) met SIRS criteria; suspicion of infection was noted by both RN and MD/APP in 5.8% of cases, by RN only in 22.2%, by MD/APP only in 7.2%, and by neither provider in 64.7% (Figure 1). Overall agreement rate was 80.7% for suspicion of infection (κ = 0.11, P < 0.001). Demographics by subgroup are shown in the Supplemental Table. Progression to severe sepsis or shock was highest when both providers suspected infection in a SIRS-positive patient (17.7%), was substantially reduced with single-provider suspicion (6.0%), and was lowest when neither provider suspected infection (1.5%) (P < 0.001). A similar trend was found for in-hospital mortality (both providers, 6.3%; single provider, 2.7%; neither provider, 2.5%; P = 0.01). Compared with MD/APP-only suspicion, SIRS-positive patients in whom only RNs suspected infection had similar frequency of progression to severe sepsis or septic shock (6.5% vs 5.6%; P = 0.52) and higher mortality (5.0% vs 1.1%; P = 0.32), though these findings were not statistically significant.

For the 121 patients (10.2%) transferred to ICU, RNs were more likely than MD/APPs to suspect infection at all time points (Figure 2). The difference was small (P = 0.29) 48 hours before transfer (RN, 12.5%; MD/APP, 5.6%) but became more pronounced (P = 0.06) by 3 hours before transfer (RN, 46.3%; MD/APP, 33.1%). Nursing assessments were not available after transfer, but 3 hours after transfer the proportion of patients who met MD/APP suspicion-of-infection criteria (44.6%) was similar (P = 0.90) to that of the RNs 3 hours before transfer (46.3%).

Our findings reveal that bedside nurses and ordering providers routinely have discordant assessments regarding presence of infection. Specifically, when RNs are asked to screen patients on the wards, they are suspicious of infection more often than MD/APPs are, and they suspect infection earlier in ICU transfer patients. These findings have significant implications for patient care, compliance with the new national SEP-1 Centers for Medicare & Medicaid Services quality measure, and identification of appropriate patients for enrollment in sepsis-related clinical trials.

To our knowledge, this is the first study to explore agreement between bedside RN and MD/APP suspicion of infection in sepsis screening and its association with patient outcomes. Studies on nurse and physician concordance in other domains have had mixed findings.^9-11 The high discordance rate found in our study points to the highly subjective nature of suspicion of infection.

Our finding that RNs suspect infection earlier in patients transferred to ICU suggests nursing suspicion has value above and beyond current practice. A possible explanation for the higher rate of RN suspicion, and earlier RN suspicion, is that bedside nurses spend substantially more time with their patients and are more attuned to subtle changes that often occur before any objective signs of deterioration. This phenomenon is well documented and accounts for why rapid response calling criteria often include “nurse worry or concern.”^12,13 Thus, nurse intuition may be an important signal for early identification of patients at high risk for sepsis.

That about one third of all screens met SIRS criteria and that almost two thirds of those screens were not thought by RN or MD/APP to be caused by infection add to the literature demonstrating the limited value of SIRS as a screening tool for sepsis.¹⁴ To address this issue, the 2016 sepsis definitions propose using the quick Sepsis-Related Organ Failure Assessment (qSOFA) to identify patients at high risk for clinical deterioration; however, the Surviving Sepsis Campaign continues to encourage sepsis screening using the SIRS criteria.¹⁵

Limitations of this study include its lack of generalizability, as it was conducted with general medical patients at a single center. Second, we did not specifically ask the MD/APPs whether they suspected infection; instead, we relied on their ordering practices. Third, RN and MD/APP assessments were not independent, as RNs had access to MD/APP orders before making their own assessments, which could bias our results.

Discordance in provider suspicion of infection is common, with RNs documenting suspicion more often than MD/APPs, and earlier in patients transferred to ICU. Suspicion by either provider alone is associated with higher risk for sepsis progression and in-hospital mortality than is the case when neither provider suspects infection. Thus, a collaborative method that includes both RNs and MD/APPs may improve the accuracy and timing of sepsis detection on the wards.

Acknowledgments

The authors thank the members of the Surviving Sepsis Campaign (SSC) Quality Improvement Learning Collaborative at the University of Chicago for their help in data collection and review, especially Meredith Borak, Rita Lanier, Mary Ann Francisco, and Bill Marsack. The authors also thank Thomas Best and Mary-Kate Springman for their assistance in data entry and Nicole Twu for administrative support. Data from this study were provided by the Clinical Research Data Warehouse (CRDW) maintained by the Center for Research Informatics (CRI) at the University of Chicago. CRI is funded by the Biological Sciences Division of the Institute for Translational Medicine/Clinical and Translational Science Award (CTSA) (National Institutes of Health UL1 TR000430) at the University of Chicago.

Disclosures

Dr. Bhattacharjee is supported by postdoctoral training grant 4T32HS000078 from the Agency for Healthcare Research and Quality. Drs. Churpek and Edelson have a patent pending (ARCD.P0535US.P2) for risk stratification algorithms for hospitalized patients. Dr. Churpek is supported by career development award K08 HL121080 from the National Heart, Lung, and Blood Institute. Dr. Edelson has received research support from Philips Healthcare (Andover, Massachusetts), American Heart Association (Dallas, Texas), and Laerdal Medical (Stavanger, Norway) and has ownership interest in Quant HC (Chicago, Illinois), which is developing products for risk stratification of hospitalized patients. The other authors report no conflicts of interest.

Sepsis is a leading cause of hospital mortality in the United States, contributing to up to half of all deaths.¹ If the infection is identified and treated early, however, its associated morbidity and mortality can be significantly reduced.² The 2001 sepsis guidelines define sepsis as the suspicion of infection plus meeting 2 or more systemic inflammatory response syndrome (SIRS) criteria.³ Although the utility of SIRS criteria has been extensively debated, providers’ accuracy and agreement regarding suspicion of infection are not yet fully characterized. This is very important, as the source of infection is often not identified in patients with severe sepsis or septic shock.⁴

Although much attention recently has been given to ideal objective criteria for accurately identifying sepsis, less is known about what constitutes ideal subjective criteria and who can best make that assessment.^5-7 We conducted a study to measure providers’ agreement regarding this subjective assessment and the impact of that agreement on patient outcomes.

We performed a secondary analysis of prospectively collected data on consecutive adults hospitalized on a general medicine ward at an academic medical center between April 1, 2014 and March 31, 2015. This study was approved by the University of Chicago Institutional Review Board with a waiver of consent.

A sepsis screening tool was developed locally as part of the Surviving Sepsis Campaign Quality Improvement Learning Collaborative⁸ (Supplemental Figure). This tool was completed by bedside nurses for each patient during each shift. Bedside registered nurse (RN) suspicion of infection was deemed positive if the nurse answered yes to question 2: “Does the patient have evidence of an active infection?” We compared RN assessment with assessment by the ordering provider, a medical doctor or advanced practice professionals (MD/APP), using an existing order for antibiotics or a new order for either blood or urine cultures placed within 12 hours before nursing screen time to indicate MD/APP suspicion of infection.

All nursing screens were transcribed into an electronic database, excluding screens not performed, or missing RN suspicion of infection. For quality purposes, screening data were merged with electronic health record data to verify SIRS criteria at the time of the screens as well as the presence of culture and/or antibiotic orders preceding the screens. Outcome data were obtained from an administrative database and confirmed by chart review using the 2001 sepsis definitions.⁶ Data were de-identified and time-shifted before this analysis. SIRS-positive criteria were defined as meeting 2 or more of the following: temperature higher than 38°C or lower than 36°C; heart rate higher than 90 beats per minute; respiratory rate more than 20 breaths per minute; and white blood cell count more than 2,000/mm³ or less than 4,000/mm³.The primary clinical outcome was progression to severe sepsis or septic shock. Secondary outcomes included transfer to intensive care unit (ICU) and in-hospital mortality. Given that RN and MD/APP suspicion of infection can vary over time, only the initial screen for each patient was used in assessing progression to severe sepsis or septic shock and in-hospital mortality. All available screens were used to investigate the association between each provider’s suspicion of infection over time and ICU transfer.

Demographic characteristics were compared using the χ² test and analysis of variance, as appropriate. Provider agreement was evaluated with a weighted κ statistic. Fisher exact tests were used to compare proportions of mortality and severe sepsis/septic shock, and the McNemar test was used to compare proportions of ICU transfers. The association of outcomes based on provider agreement was evaluated with a nonparametric test for trend.

During the study period, 1386 distinct patients had 13,223 screening opportunities, with a 95.4% compliance rate. A total of 1127 screens were excluded for missing nursing documentation of suspicion of infection, leaving 1192 first screens and 11,489 total screens for analysis. Of the completed screens, 3744 (32.6%) met SIRS criteria; suspicion of infection was noted by both RN and MD/APP in 5.8% of cases, by RN only in 22.2%, by MD/APP only in 7.2%, and by neither provider in 64.7% (Figure 1). Overall agreement rate was 80.7% for suspicion of infection (κ = 0.11, P < 0.001). Demographics by subgroup are shown in the Supplemental Table. Progression to severe sepsis or shock was highest when both providers suspected infection in a SIRS-positive patient (17.7%), was substantially reduced with single-provider suspicion (6.0%), and was lowest when neither provider suspected infection (1.5%) (P < 0.001). A similar trend was found for in-hospital mortality (both providers, 6.3%; single provider, 2.7%; neither provider, 2.5%; P = 0.01). Compared with MD/APP-only suspicion, SIRS-positive patients in whom only RNs suspected infection had similar frequency of progression to severe sepsis or septic shock (6.5% vs 5.6%; P = 0.52) and higher mortality (5.0% vs 1.1%; P = 0.32), though these findings were not statistically significant.

For the 121 patients (10.2%) transferred to ICU, RNs were more likely than MD/APPs to suspect infection at all time points (Figure 2). The difference was small (P = 0.29) 48 hours before transfer (RN, 12.5%; MD/APP, 5.6%) but became more pronounced (P = 0.06) by 3 hours before transfer (RN, 46.3%; MD/APP, 33.1%). Nursing assessments were not available after transfer, but 3 hours after transfer the proportion of patients who met MD/APP suspicion-of-infection criteria (44.6%) was similar (P = 0.90) to that of the RNs 3 hours before transfer (46.3%).

Our findings reveal that bedside nurses and ordering providers routinely have discordant assessments regarding presence of infection. Specifically, when RNs are asked to screen patients on the wards, they are suspicious of infection more often than MD/APPs are, and they suspect infection earlier in ICU transfer patients. These findings have significant implications for patient care, compliance with the new national SEP-1 Centers for Medicare & Medicaid Services quality measure, and identification of appropriate patients for enrollment in sepsis-related clinical trials.

To our knowledge, this is the first study to explore agreement between bedside RN and MD/APP suspicion of infection in sepsis screening and its association with patient outcomes. Studies on nurse and physician concordance in other domains have had mixed findings.^9-11 The high discordance rate found in our study points to the highly subjective nature of suspicion of infection.

Our finding that RNs suspect infection earlier in patients transferred to ICU suggests nursing suspicion has value above and beyond current practice. A possible explanation for the higher rate of RN suspicion, and earlier RN suspicion, is that bedside nurses spend substantially more time with their patients and are more attuned to subtle changes that often occur before any objective signs of deterioration. This phenomenon is well documented and accounts for why rapid response calling criteria often include “nurse worry or concern.”^12,13 Thus, nurse intuition may be an important signal for early identification of patients at high risk for sepsis.

That about one third of all screens met SIRS criteria and that almost two thirds of those screens were not thought by RN or MD/APP to be caused by infection add to the literature demonstrating the limited value of SIRS as a screening tool for sepsis.¹⁴ To address this issue, the 2016 sepsis definitions propose using the quick Sepsis-Related Organ Failure Assessment (qSOFA) to identify patients at high risk for clinical deterioration; however, the Surviving Sepsis Campaign continues to encourage sepsis screening using the SIRS criteria.¹⁵

Limitations of this study include its lack of generalizability, as it was conducted with general medical patients at a single center. Second, we did not specifically ask the MD/APPs whether they suspected infection; instead, we relied on their ordering practices. Third, RN and MD/APP assessments were not independent, as RNs had access to MD/APP orders before making their own assessments, which could bias our results.

Discordance in provider suspicion of infection is common, with RNs documenting suspicion more often than MD/APPs, and earlier in patients transferred to ICU. Suspicion by either provider alone is associated with higher risk for sepsis progression and in-hospital mortality than is the case when neither provider suspects infection. Thus, a collaborative method that includes both RNs and MD/APPs may improve the accuracy and timing of sepsis detection on the wards.

Acknowledgments

The authors thank the members of the Surviving Sepsis Campaign (SSC) Quality Improvement Learning Collaborative at the University of Chicago for their help in data collection and review, especially Meredith Borak, Rita Lanier, Mary Ann Francisco, and Bill Marsack. The authors also thank Thomas Best and Mary-Kate Springman for their assistance in data entry and Nicole Twu for administrative support. Data from this study were provided by the Clinical Research Data Warehouse (CRDW) maintained by the Center for Research Informatics (CRI) at the University of Chicago. CRI is funded by the Biological Sciences Division of the Institute for Translational Medicine/Clinical and Translational Science Award (CTSA) (National Institutes of Health UL1 TR000430) at the University of Chicago.

Disclosures

Dr. Bhattacharjee is supported by postdoctoral training grant 4T32HS000078 from the Agency for Healthcare Research and Quality. Drs. Churpek and Edelson have a patent pending (ARCD.P0535US.P2) for risk stratification algorithms for hospitalized patients. Dr. Churpek is supported by career development award K08 HL121080 from the National Heart, Lung, and Blood Institute. Dr. Edelson has received research support from Philips Healthcare (Andover, Massachusetts), American Heart Association (Dallas, Texas), and Laerdal Medical (Stavanger, Norway) and has ownership interest in Quant HC (Chicago, Illinois), which is developing products for risk stratification of hospitalized patients. The other authors report no conflicts of interest.

Sepsis is a leading cause of hospital mortality in the United States, contributing to up to half of all deaths.¹ If the infection is identified and treated early, however, its associated morbidity and mortality can be significantly reduced.² The 2001 sepsis guidelines define sepsis as the suspicion of infection plus meeting 2 or more systemic inflammatory response syndrome (SIRS) criteria.³ Although the utility of SIRS criteria has been extensively debated, providers’ accuracy and agreement regarding suspicion of infection are not yet fully characterized. This is very important, as the source of infection is often not identified in patients with severe sepsis or septic shock.⁴

Although much attention recently has been given to ideal objective criteria for accurately identifying sepsis, less is known about what constitutes ideal subjective criteria and who can best make that assessment.^5-7 We conducted a study to measure providers’ agreement regarding this subjective assessment and the impact of that agreement on patient outcomes.

We performed a secondary analysis of prospectively collected data on consecutive adults hospitalized on a general medicine ward at an academic medical center between April 1, 2014 and March 31, 2015. This study was approved by the University of Chicago Institutional Review Board with a waiver of consent.

A sepsis screening tool was developed locally as part of the Surviving Sepsis Campaign Quality Improvement Learning Collaborative⁸ (Supplemental Figure). This tool was completed by bedside nurses for each patient during each shift. Bedside registered nurse (RN) suspicion of infection was deemed positive if the nurse answered yes to question 2: “Does the patient have evidence of an active infection?” We compared RN assessment with assessment by the ordering provider, a medical doctor or advanced practice professionals (MD/APP), using an existing order for antibiotics or a new order for either blood or urine cultures placed within 12 hours before nursing screen time to indicate MD/APP suspicion of infection.

All nursing screens were transcribed into an electronic database, excluding screens not performed, or missing RN suspicion of infection. For quality purposes, screening data were merged with electronic health record data to verify SIRS criteria at the time of the screens as well as the presence of culture and/or antibiotic orders preceding the screens. Outcome data were obtained from an administrative database and confirmed by chart review using the 2001 sepsis definitions.⁶ Data were de-identified and time-shifted before this analysis. SIRS-positive criteria were defined as meeting 2 or more of the following: temperature higher than 38°C or lower than 36°C; heart rate higher than 90 beats per minute; respiratory rate more than 20 breaths per minute; and white blood cell count more than 2,000/mm³ or less than 4,000/mm³.The primary clinical outcome was progression to severe sepsis or septic shock. Secondary outcomes included transfer to intensive care unit (ICU) and in-hospital mortality. Given that RN and MD/APP suspicion of infection can vary over time, only the initial screen for each patient was used in assessing progression to severe sepsis or septic shock and in-hospital mortality. All available screens were used to investigate the association between each provider’s suspicion of infection over time and ICU transfer.

Demographic characteristics were compared using the χ² test and analysis of variance, as appropriate. Provider agreement was evaluated with a weighted κ statistic. Fisher exact tests were used to compare proportions of mortality and severe sepsis/septic shock, and the McNemar test was used to compare proportions of ICU transfers. The association of outcomes based on provider agreement was evaluated with a nonparametric test for trend.

During the study period, 1386 distinct patients had 13,223 screening opportunities, with a 95.4% compliance rate. A total of 1127 screens were excluded for missing nursing documentation of suspicion of infection, leaving 1192 first screens and 11,489 total screens for analysis. Of the completed screens, 3744 (32.6%) met SIRS criteria; suspicion of infection was noted by both RN and MD/APP in 5.8% of cases, by RN only in 22.2%, by MD/APP only in 7.2%, and by neither provider in 64.7% (Figure 1). Overall agreement rate was 80.7% for suspicion of infection (κ = 0.11, P < 0.001). Demographics by subgroup are shown in the Supplemental Table. Progression to severe sepsis or shock was highest when both providers suspected infection in a SIRS-positive patient (17.7%), was substantially reduced with single-provider suspicion (6.0%), and was lowest when neither provider suspected infection (1.5%) (P < 0.001). A similar trend was found for in-hospital mortality (both providers, 6.3%; single provider, 2.7%; neither provider, 2.5%; P = 0.01). Compared with MD/APP-only suspicion, SIRS-positive patients in whom only RNs suspected infection had similar frequency of progression to severe sepsis or septic shock (6.5% vs 5.6%; P = 0.52) and higher mortality (5.0% vs 1.1%; P = 0.32), though these findings were not statistically significant.

For the 121 patients (10.2%) transferred to ICU, RNs were more likely than MD/APPs to suspect infection at all time points (Figure 2). The difference was small (P = 0.29) 48 hours before transfer (RN, 12.5%; MD/APP, 5.6%) but became more pronounced (P = 0.06) by 3 hours before transfer (RN, 46.3%; MD/APP, 33.1%). Nursing assessments were not available after transfer, but 3 hours after transfer the proportion of patients who met MD/APP suspicion-of-infection criteria (44.6%) was similar (P = 0.90) to that of the RNs 3 hours before transfer (46.3%).

Our findings reveal that bedside nurses and ordering providers routinely have discordant assessments regarding presence of infection. Specifically, when RNs are asked to screen patients on the wards, they are suspicious of infection more often than MD/APPs are, and they suspect infection earlier in ICU transfer patients. These findings have significant implications for patient care, compliance with the new national SEP-1 Centers for Medicare & Medicaid Services quality measure, and identification of appropriate patients for enrollment in sepsis-related clinical trials.

To our knowledge, this is the first study to explore agreement between bedside RN and MD/APP suspicion of infection in sepsis screening and its association with patient outcomes. Studies on nurse and physician concordance in other domains have had mixed findings.^9-11 The high discordance rate found in our study points to the highly subjective nature of suspicion of infection.

Our finding that RNs suspect infection earlier in patients transferred to ICU suggests nursing suspicion has value above and beyond current practice. A possible explanation for the higher rate of RN suspicion, and earlier RN suspicion, is that bedside nurses spend substantially more time with their patients and are more attuned to subtle changes that often occur before any objective signs of deterioration. This phenomenon is well documented and accounts for why rapid response calling criteria often include “nurse worry or concern.”^12,13 Thus, nurse intuition may be an important signal for early identification of patients at high risk for sepsis.

That about one third of all screens met SIRS criteria and that almost two thirds of those screens were not thought by RN or MD/APP to be caused by infection add to the literature demonstrating the limited value of SIRS as a screening tool for sepsis.¹⁴ To address this issue, the 2016 sepsis definitions propose using the quick Sepsis-Related Organ Failure Assessment (qSOFA) to identify patients at high risk for clinical deterioration; however, the Surviving Sepsis Campaign continues to encourage sepsis screening using the SIRS criteria.¹⁵

Limitations of this study include its lack of generalizability, as it was conducted with general medical patients at a single center. Second, we did not specifically ask the MD/APPs whether they suspected infection; instead, we relied on their ordering practices. Third, RN and MD/APP assessments were not independent, as RNs had access to MD/APP orders before making their own assessments, which could bias our results.

Discordance in provider suspicion of infection is common, with RNs documenting suspicion more often than MD/APPs, and earlier in patients transferred to ICU. Suspicion by either provider alone is associated with higher risk for sepsis progression and in-hospital mortality than is the case when neither provider suspects infection. Thus, a collaborative method that includes both RNs and MD/APPs may improve the accuracy and timing of sepsis detection on the wards.

Acknowledgments

The authors thank the members of the Surviving Sepsis Campaign (SSC) Quality Improvement Learning Collaborative at the University of Chicago for their help in data collection and review, especially Meredith Borak, Rita Lanier, Mary Ann Francisco, and Bill Marsack. The authors also thank Thomas Best and Mary-Kate Springman for their assistance in data entry and Nicole Twu for administrative support. Data from this study were provided by the Clinical Research Data Warehouse (CRDW) maintained by the Center for Research Informatics (CRI) at the University of Chicago. CRI is funded by the Biological Sciences Division of the Institute for Translational Medicine/Clinical and Translational Science Award (CTSA) (National Institutes of Health UL1 TR000430) at the University of Chicago.

Disclosures

Dr. Bhattacharjee is supported by postdoctoral training grant 4T32HS000078 from the Agency for Healthcare Research and Quality. Drs. Churpek and Edelson have a patent pending (ARCD.P0535US.P2) for risk stratification algorithms for hospitalized patients. Dr. Churpek is supported by career development award K08 HL121080 from the National Heart, Lung, and Blood Institute. Dr. Edelson has received research support from Philips Healthcare (Andover, Massachusetts), American Heart Association (Dallas, Texas), and Laerdal Medical (Stavanger, Norway) and has ownership interest in Quant HC (Chicago, Illinois), which is developing products for risk stratification of hospitalized patients. The other authors report no conflicts of interest.

The “Things We Do for No Reason” series reviews practices which have become common parts of hospital care but which may provide little value to our patients. Practices reviewed in the TWDFNR series do not represent “black and white” conclusions or clinical practice standards, but are meant as a starting place for research and active discussions among hospitalists and patients. We invite you to be part of that discussion. https://www.choosingwisely.org/

A 50-year-old man presented to the emergency department with a 3-day-old cut on his anterior right shin. Associated redness, warmth, pain, and swelling had progressed. The patient had no history of prior DVT or pulmonary embolism (PE). His temperature was 38.5°C, and his white blood cell count of 18,000. On review of systems, he denied shortness of breath and chest pain. He was diagnosed with cellulitis and administered intravenous fluids and cefazolin. The clinician wondered whether to perform lower extremity ultrasound to rule out concurrent DVT.

WHY YOU MIGHT THINK ULTRASOUND IS HELPFUL IN RULING OUT DVT IN CELLULITIS

Lower extremity cellulitis, a common infection of the skin and subcutaneous tissues, is characterized by unilateral erythema, pain, warmth, and swelling. The infection usually follows a skin breach that allows bacteria to enter. DVT may present similarly, and symptoms can include mild leukocytosis and elevated temperature. Because of the clinical similarities, clinicians often order compression ultrasound of the extremity to rule out concurrent DVT in cellulitis. Further impetus for testing stems from fear of the potential complications of untreated DVT, including post-thrombotic syndrome, chronic venous insufficiency, and venous ulceration. A subsequent PE can be fatal, or can cause significant morbidity, including chronic VTE with associated pulmonary hypertension. An estimated quarter of all PEs present as sudden death.¹

WHY ULTRASOUND IS NOT HELPFUL IN THIS SETTING

Studies have shown that ultrasound is ordered for 16% to 73% of patients with a cellulitis diagnosis.^2,3 Although testing is commonly performed, a meta-analysis of 9 studies of cellulitis patients who underwent ultrasound testing for concurrent DVT revealed a low pooled incidence of total DVT (3.1%) and proximal DVT (2.1%).⁴ Maze et al.² retrospectively reviewed 1515 cellulitis cases (identified by International Classification of Diseases, Ninth Revision codes) at a single center in New Zealand over 3 years. Of the 1515 patients, 240 (16%) had ultrasound performed, and only 3 (1.3%) were found to have DVT. Two of the 3 had active malignancy, and the third had injected battery acid into the area. In a 5-year retrospective cohort study at a Veterans Administration hospital in Connecticut, Gunderson and Chang³ reviewed the cases of 183 patients with cellulitis and found ultrasound testing commonly performed (73% of cases) to assess for DVT. Only 1 patient (<1%) was diagnosed with new DVT in the ipsilateral leg, and acute DVT was diagnosed in the contralateral leg of 2 other patients. Overall, these studies indicate the incidence of concurrent DVT in cellulitis is low, regardless of the frequency of ultrasound testing.

Although the cost of a single ultrasound test is not prohibitive, annual total costs hospital-wide and nationally are large. In the United States, the charge for a unilateral duplex ultrasound of the extremity ranges from $260 to $1300, and there is an additional charge for interpretation by a radiologist.⁵ In a retrospective study spanning 3.5 years and involving 2 community hospitals in Michigan, an estimated $290,000 was spent on ultrasound tests defined as unnecessary for patients with cellulitis.⁶ A limitation of the study was defining a test as unnecessary based on its result being negative.

DOES WELLS SCORE WITH D-DIMER HELP DEFINE A LOW-RISK POPULATION?

The Wells clinical prediction rule is commonly used to assess the pretest probability of DVT in patients presenting with unilateral leg symptoms. The Wells score is often combined with D-dimer testing to help determine whether ultrasound is necessary. Studies of patients with suspected DVT have found that those considered low risk according to the Wells criteria had a 6.5% incidence of DVT.⁷ However, the predictive value is lower in the setting of presumed cellulitis. In a prospective cohort study of 200 patients with cellulitis, Maze et al.⁸ reported that use of the Wells score with D-dimer testing overestimated the DVT risk. D-dimer level was elevated for 74% of patients, and 20.5% were high-risk by Wells criteria. An algorithm determined that—among patients with a high-risk Wells score, a positive D-dimer result, or both—only 1 (0.5%) was diagnosed with ipsilateral DVT after ultrasound testing. Two patients were diagnosed with DVT in the contralateral leg. These results suggest that a strategy that incorporates the Wells score and D-dimer testing in the setting of acute cellulitis provides little value. The authors concluded that, in the absence of a known hypercoagulable state, ultrasound is not warranted. However, their study did not assess whether there are any specific hypercoagulable states for which further testing may be indicated.

WHEN MIGHT ULTRASOUND BE HELPFUL IN CELLULITIS?

Investigators have described possible DVT risk factors in patients with cellulitis, but definitive associations are lacking because of the insufficient number of patients studied.^8,9 The most consistently identified DVT risk factor is history of previous thromboembolism. In a retrospective analysis of patients with cellulitis, Afzal et al.⁶ found that, of the 66.8% who underwent ultrasound testing, 5.5% were identified as having concurrent DVT. The authors performed univariate analyses of 15 potential risk factors, including active malignancy, oral contraceptive pill use, recent hospitalization, and surgery. A higher incidence of DVT was found for patients with history of VTE (odds ratio [OR], 5.7; 95% confidence interval [CI], 2.3-13.7), calf swelling (OR, 4.5; 95% CI, 1.3-15.8), CVA (OR, 3.5; 95% CI, 1.2-10.1), or hypertension (OR, 3.5; 95% CI, 0.98-12.2). Given the wide confidence intervals, paucity of studies, and lack of definitive data in the setting of cellulitis, clinicians may want to consider the risk factors established in larger trials in other settings, including known immobility (OR, <2); thrombophilia, CHF, and CVA with hemiparesis (OR, 2-9); and trauma and recent surgery (OR, >10).¹⁰

WHAT YOU SHOULD DO INSTEAD

As the incidence of concurrent VTE in patients with cellulitis is low, the essential step is to make a clear diagnosis of cellulitis based on its established signs and symptoms. A 2-center trial of 145 patients found that cellulitis was diagnosed accurately by general medicine and emergency medicine physicians 72% of the time, with evaluation by dermatologists and infectious disease specialists used as the gold standard. Only 5% of the misdiagnosed patients were diagnosed with DVT; stasis dermatitis was the most common alternative diagnosis. Taking a thorough history may elicit risk factors consistent with cellulitis, such as a recent injury with a break in the skin. On examination, cellulitis should be suspected for patients with fever and localized pain, redness, swelling, and warmth—the cardinal signs of dolor, rubor, tumor, and calor. An injury or entry site and leukocytosis also support the diagnosis of cellulitis. Distinct margins of erythema on the skin are highly suspicious for erysipelas.¹¹ Other physical findings (eg, laceration, purulent drainage, lymphangitic spread, fluctuating mass) also are consistent with a diagnosis of cellulitis.

The patient’s history is also essential in determining whether any DVT risk factors are present. Past medical history of VTE or CVA, or recent history of surgery, immobility, or trauma, should alert the clinician to the possibility of DVT. Family history of VTE increases the likelihood of DVT. Acute shortness of breath or chest pain in the setting of concerning lower extremity findings for DVT should raise concern for DVT and concurrent PE.

If the classic features of cellulitis are present, empiric antibiotics should be initiated. Routine ultrasound testing for all patients with cellulitis is of low value. However, as the incidence of DVT in this population is not negligible, those with VTE risk factors should be targeted for testing. Studies in the setting of cellulitis provide little guidance regarding specific risk factors that can be used to determine who should undergo further testing. Given this limitation, we suggest that clinicians incorporate into their decision making the well-established VTE risk factors identified for large populations studied in other settings, such as the postoperative period. Specifically, clinicians should consider ultrasound testing for patients with cellulitis and prior history of VTE; immobility; thrombophilia, CHF, and CVA with hemiparesis; or trauma and recent surgery.^10-12 Ultrasound should also be considered for patients with cellulitis that does not improve and for patients whose localized symptoms worsen despite use of antibiotics.

• Do not routinely perform ultrasound to rule out concurrent DVT in cases of cellulitis.

• Consider compression ultrasound if there is a history of VTE; immobility; thrombophilia, CHF, and CVA with hemiparesis; or trauma and recent surgery. Also consider it for patients who do not respond to antibiotics.

• In cases of cellulitis, avoid use of the Wells score alone or with D-dimer testing, as it likely overestimates the DVT risk.

The current evidence shows that, for most patients with cellulitis, routine ultrasound testing for DVT is unnecessary. Ultrasound should be considered for patients with potent VTE risk factors. If symptoms do not improve, or if they worsen despite use of antibiotics, clinicians should be alert to potential anchoring bias and consider DVT. The Wells clinical prediction rule overestimates the incidence of DVT in cellulitis and has little value in this setting.

Disclosure

Nothing to report.

Do you think this is a low-value practice? Is this truly a “Thing We Do for No Reason”? Let us know what you do in your practice and propose ideas for other “Things We Do for No Reason” topics. Please join in the conversation online at Twitter (#TWDFNR)/Facebook and don’t forget to “Like It” on Facebook or retweet it on Twitter. We invite you to propose ideas for other “Things We Do for No Reason” topics by emailing [email protected].

The “Things We Do for No Reason” series reviews practices which have become common parts of hospital care but which may provide little value to our patients. Practices reviewed in the TWDFNR series do not represent “black and white” conclusions or clinical practice standards, but are meant as a starting place for research and active discussions among hospitalists and patients. We invite you to be part of that discussion. https://www.choosingwisely.org/

A 50-year-old man presented to the emergency department with a 3-day-old cut on his anterior right shin. Associated redness, warmth, pain, and swelling had progressed. The patient had no history of prior DVT or pulmonary embolism (PE). His temperature was 38.5°C, and his white blood cell count of 18,000. On review of systems, he denied shortness of breath and chest pain. He was diagnosed with cellulitis and administered intravenous fluids and cefazolin. The clinician wondered whether to perform lower extremity ultrasound to rule out concurrent DVT.

WHY YOU MIGHT THINK ULTRASOUND IS HELPFUL IN RULING OUT DVT IN CELLULITIS

Lower extremity cellulitis, a common infection of the skin and subcutaneous tissues, is characterized by unilateral erythema, pain, warmth, and swelling. The infection usually follows a skin breach that allows bacteria to enter. DVT may present similarly, and symptoms can include mild leukocytosis and elevated temperature. Because of the clinical similarities, clinicians often order compression ultrasound of the extremity to rule out concurrent DVT in cellulitis. Further impetus for testing stems from fear of the potential complications of untreated DVT, including post-thrombotic syndrome, chronic venous insufficiency, and venous ulceration. A subsequent PE can be fatal, or can cause significant morbidity, including chronic VTE with associated pulmonary hypertension. An estimated quarter of all PEs present as sudden death.¹

WHY ULTRASOUND IS NOT HELPFUL IN THIS SETTING

Studies have shown that ultrasound is ordered for 16% to 73% of patients with a cellulitis diagnosis.^2,3 Although testing is commonly performed, a meta-analysis of 9 studies of cellulitis patients who underwent ultrasound testing for concurrent DVT revealed a low pooled incidence of total DVT (3.1%) and proximal DVT (2.1%).⁴ Maze et al.² retrospectively reviewed 1515 cellulitis cases (identified by International Classification of Diseases, Ninth Revision codes) at a single center in New Zealand over 3 years. Of the 1515 patients, 240 (16%) had ultrasound performed, and only 3 (1.3%) were found to have DVT. Two of the 3 had active malignancy, and the third had injected battery acid into the area. In a 5-year retrospective cohort study at a Veterans Administration hospital in Connecticut, Gunderson and Chang³ reviewed the cases of 183 patients with cellulitis and found ultrasound testing commonly performed (73% of cases) to assess for DVT. Only 1 patient (<1%) was diagnosed with new DVT in the ipsilateral leg, and acute DVT was diagnosed in the contralateral leg of 2 other patients. Overall, these studies indicate the incidence of concurrent DVT in cellulitis is low, regardless of the frequency of ultrasound testing.

Although the cost of a single ultrasound test is not prohibitive, annual total costs hospital-wide and nationally are large. In the United States, the charge for a unilateral duplex ultrasound of the extremity ranges from $260 to $1300, and there is an additional charge for interpretation by a radiologist.⁵ In a retrospective study spanning 3.5 years and involving 2 community hospitals in Michigan, an estimated $290,000 was spent on ultrasound tests defined as unnecessary for patients with cellulitis.⁶ A limitation of the study was defining a test as unnecessary based on its result being negative.

DOES WELLS SCORE WITH D-DIMER HELP DEFINE A LOW-RISK POPULATION?

The Wells clinical prediction rule is commonly used to assess the pretest probability of DVT in patients presenting with unilateral leg symptoms. The Wells score is often combined with D-dimer testing to help determine whether ultrasound is necessary. Studies of patients with suspected DVT have found that those considered low risk according to the Wells criteria had a 6.5% incidence of DVT.⁷ However, the predictive value is lower in the setting of presumed cellulitis. In a prospective cohort study of 200 patients with cellulitis, Maze et al.⁸ reported that use of the Wells score with D-dimer testing overestimated the DVT risk. D-dimer level was elevated for 74% of patients, and 20.5% were high-risk by Wells criteria. An algorithm determined that—among patients with a high-risk Wells score, a positive D-dimer result, or both—only 1 (0.5%) was diagnosed with ipsilateral DVT after ultrasound testing. Two patients were diagnosed with DVT in the contralateral leg. These results suggest that a strategy that incorporates the Wells score and D-dimer testing in the setting of acute cellulitis provides little value. The authors concluded that, in the absence of a known hypercoagulable state, ultrasound is not warranted. However, their study did not assess whether there are any specific hypercoagulable states for which further testing may be indicated.

WHEN MIGHT ULTRASOUND BE HELPFUL IN CELLULITIS?

Investigators have described possible DVT risk factors in patients with cellulitis, but definitive associations are lacking because of the insufficient number of patients studied.^8,9 The most consistently identified DVT risk factor is history of previous thromboembolism. In a retrospective analysis of patients with cellulitis, Afzal et al.⁶ found that, of the 66.8% who underwent ultrasound testing, 5.5% were identified as having concurrent DVT. The authors performed univariate analyses of 15 potential risk factors, including active malignancy, oral contraceptive pill use, recent hospitalization, and surgery. A higher incidence of DVT was found for patients with history of VTE (odds ratio [OR], 5.7; 95% confidence interval [CI], 2.3-13.7), calf swelling (OR, 4.5; 95% CI, 1.3-15.8), CVA (OR, 3.5; 95% CI, 1.2-10.1), or hypertension (OR, 3.5; 95% CI, 0.98-12.2). Given the wide confidence intervals, paucity of studies, and lack of definitive data in the setting of cellulitis, clinicians may want to consider the risk factors established in larger trials in other settings, including known immobility (OR, <2); thrombophilia, CHF, and CVA with hemiparesis (OR, 2-9); and trauma and recent surgery (OR, >10).¹⁰

WHAT YOU SHOULD DO INSTEAD

As the incidence of concurrent VTE in patients with cellulitis is low, the essential step is to make a clear diagnosis of cellulitis based on its established signs and symptoms. A 2-center trial of 145 patients found that cellulitis was diagnosed accurately by general medicine and emergency medicine physicians 72% of the time, with evaluation by dermatologists and infectious disease specialists used as the gold standard. Only 5% of the misdiagnosed patients were diagnosed with DVT; stasis dermatitis was the most common alternative diagnosis. Taking a thorough history may elicit risk factors consistent with cellulitis, such as a recent injury with a break in the skin. On examination, cellulitis should be suspected for patients with fever and localized pain, redness, swelling, and warmth—the cardinal signs of dolor, rubor, tumor, and calor. An injury or entry site and leukocytosis also support the diagnosis of cellulitis. Distinct margins of erythema on the skin are highly suspicious for erysipelas.¹¹ Other physical findings (eg, laceration, purulent drainage, lymphangitic spread, fluctuating mass) also are consistent with a diagnosis of cellulitis.

The patient’s history is also essential in determining whether any DVT risk factors are present. Past medical history of VTE or CVA, or recent history of surgery, immobility, or trauma, should alert the clinician to the possibility of DVT. Family history of VTE increases the likelihood of DVT. Acute shortness of breath or chest pain in the setting of concerning lower extremity findings for DVT should raise concern for DVT and concurrent PE.

If the classic features of cellulitis are present, empiric antibiotics should be initiated. Routine ultrasound testing for all patients with cellulitis is of low value. However, as the incidence of DVT in this population is not negligible, those with VTE risk factors should be targeted for testing. Studies in the setting of cellulitis provide little guidance regarding specific risk factors that can be used to determine who should undergo further testing. Given this limitation, we suggest that clinicians incorporate into their decision making the well-established VTE risk factors identified for large populations studied in other settings, such as the postoperative period. Specifically, clinicians should consider ultrasound testing for patients with cellulitis and prior history of VTE; immobility; thrombophilia, CHF, and CVA with hemiparesis; or trauma and recent surgery.^10-12 Ultrasound should also be considered for patients with cellulitis that does not improve and for patients whose localized symptoms worsen despite use of antibiotics.

• Do not routinely perform ultrasound to rule out concurrent DVT in cases of cellulitis.

• Consider compression ultrasound if there is a history of VTE; immobility; thrombophilia, CHF, and CVA with hemiparesis; or trauma and recent surgery. Also consider it for patients who do not respond to antibiotics.

• In cases of cellulitis, avoid use of the Wells score alone or with D-dimer testing, as it likely overestimates the DVT risk.

The current evidence shows that, for most patients with cellulitis, routine ultrasound testing for DVT is unnecessary. Ultrasound should be considered for patients with potent VTE risk factors. If symptoms do not improve, or if they worsen despite use of antibiotics, clinicians should be alert to potential anchoring bias and consider DVT. The Wells clinical prediction rule overestimates the incidence of DVT in cellulitis and has little value in this setting.

Disclosure

Nothing to report.

Do you think this is a low-value practice? Is this truly a “Thing We Do for No Reason”? Let us know what you do in your practice and propose ideas for other “Things We Do for No Reason” topics. Please join in the conversation online at Twitter (#TWDFNR)/Facebook and don’t forget to “Like It” on Facebook or retweet it on Twitter. We invite you to propose ideas for other “Things We Do for No Reason” topics by emailing [email protected].

The “Things We Do for No Reason” series reviews practices which have become common parts of hospital care but which may provide little value to our patients. Practices reviewed in the TWDFNR series do not represent “black and white” conclusions or clinical practice standards, but are meant as a starting place for research and active discussions among hospitalists and patients. We invite you to be part of that discussion. https://www.choosingwisely.org/

A 50-year-old man presented to the emergency department with a 3-day-old cut on his anterior right shin. Associated redness, warmth, pain, and swelling had progressed. The patient had no history of prior DVT or pulmonary embolism (PE). His temperature was 38.5°C, and his white blood cell count of 18,000. On review of systems, he denied shortness of breath and chest pain. He was diagnosed with cellulitis and administered intravenous fluids and cefazolin. The clinician wondered whether to perform lower extremity ultrasound to rule out concurrent DVT.

WHY YOU MIGHT THINK ULTRASOUND IS HELPFUL IN RULING OUT DVT IN CELLULITIS

Lower extremity cellulitis, a common infection of the skin and subcutaneous tissues, is characterized by unilateral erythema, pain, warmth, and swelling. The infection usually follows a skin breach that allows bacteria to enter. DVT may present similarly, and symptoms can include mild leukocytosis and elevated temperature. Because of the clinical similarities, clinicians often order compression ultrasound of the extremity to rule out concurrent DVT in cellulitis. Further impetus for testing stems from fear of the potential complications of untreated DVT, including post-thrombotic syndrome, chronic venous insufficiency, and venous ulceration. A subsequent PE can be fatal, or can cause significant morbidity, including chronic VTE with associated pulmonary hypertension. An estimated quarter of all PEs present as sudden death.¹

WHY ULTRASOUND IS NOT HELPFUL IN THIS SETTING

Studies have shown that ultrasound is ordered for 16% to 73% of patients with a cellulitis diagnosis.^2,3 Although testing is commonly performed, a meta-analysis of 9 studies of cellulitis patients who underwent ultrasound testing for concurrent DVT revealed a low pooled incidence of total DVT (3.1%) and proximal DVT (2.1%).⁴ Maze et al.² retrospectively reviewed 1515 cellulitis cases (identified by International Classification of Diseases, Ninth Revision codes) at a single center in New Zealand over 3 years. Of the 1515 patients, 240 (16%) had ultrasound performed, and only 3 (1.3%) were found to have DVT. Two of the 3 had active malignancy, and the third had injected battery acid into the area. In a 5-year retrospective cohort study at a Veterans Administration hospital in Connecticut, Gunderson and Chang³ reviewed the cases of 183 patients with cellulitis and found ultrasound testing commonly performed (73% of cases) to assess for DVT. Only 1 patient (<1%) was diagnosed with new DVT in the ipsilateral leg, and acute DVT was diagnosed in the contralateral leg of 2 other patients. Overall, these studies indicate the incidence of concurrent DVT in cellulitis is low, regardless of the frequency of ultrasound testing.

Although the cost of a single ultrasound test is not prohibitive, annual total costs hospital-wide and nationally are large. In the United States, the charge for a unilateral duplex ultrasound of the extremity ranges from $260 to $1300, and there is an additional charge for interpretation by a radiologist.⁵ In a retrospective study spanning 3.5 years and involving 2 community hospitals in Michigan, an estimated $290,000 was spent on ultrasound tests defined as unnecessary for patients with cellulitis.⁶ A limitation of the study was defining a test as unnecessary based on its result being negative.

DOES WELLS SCORE WITH D-DIMER HELP DEFINE A LOW-RISK POPULATION?

The Wells clinical prediction rule is commonly used to assess the pretest probability of DVT in patients presenting with unilateral leg symptoms. The Wells score is often combined with D-dimer testing to help determine whether ultrasound is necessary. Studies of patients with suspected DVT have found that those considered low risk according to the Wells criteria had a 6.5% incidence of DVT.⁷ However, the predictive value is lower in the setting of presumed cellulitis. In a prospective cohort study of 200 patients with cellulitis, Maze et al.⁸ reported that use of the Wells score with D-dimer testing overestimated the DVT risk. D-dimer level was elevated for 74% of patients, and 20.5% were high-risk by Wells criteria. An algorithm determined that—among patients with a high-risk Wells score, a positive D-dimer result, or both—only 1 (0.5%) was diagnosed with ipsilateral DVT after ultrasound testing. Two patients were diagnosed with DVT in the contralateral leg. These results suggest that a strategy that incorporates the Wells score and D-dimer testing in the setting of acute cellulitis provides little value. The authors concluded that, in the absence of a known hypercoagulable state, ultrasound is not warranted. However, their study did not assess whether there are any specific hypercoagulable states for which further testing may be indicated.

WHEN MIGHT ULTRASOUND BE HELPFUL IN CELLULITIS?

Investigators have described possible DVT risk factors in patients with cellulitis, but definitive associations are lacking because of the insufficient number of patients studied.^8,9 The most consistently identified DVT risk factor is history of previous thromboembolism. In a retrospective analysis of patients with cellulitis, Afzal et al.⁶ found that, of the 66.8% who underwent ultrasound testing, 5.5% were identified as having concurrent DVT. The authors performed univariate analyses of 15 potential risk factors, including active malignancy, oral contraceptive pill use, recent hospitalization, and surgery. A higher incidence of DVT was found for patients with history of VTE (odds ratio [OR], 5.7; 95% confidence interval [CI], 2.3-13.7), calf swelling (OR, 4.5; 95% CI, 1.3-15.8), CVA (OR, 3.5; 95% CI, 1.2-10.1), or hypertension (OR, 3.5; 95% CI, 0.98-12.2). Given the wide confidence intervals, paucity of studies, and lack of definitive data in the setting of cellulitis, clinicians may want to consider the risk factors established in larger trials in other settings, including known immobility (OR, <2); thrombophilia, CHF, and CVA with hemiparesis (OR, 2-9); and trauma and recent surgery (OR, >10).¹⁰

WHAT YOU SHOULD DO INSTEAD

As the incidence of concurrent VTE in patients with cellulitis is low, the essential step is to make a clear diagnosis of cellulitis based on its established signs and symptoms. A 2-center trial of 145 patients found that cellulitis was diagnosed accurately by general medicine and emergency medicine physicians 72% of the time, with evaluation by dermatologists and infectious disease specialists used as the gold standard. Only 5% of the misdiagnosed patients were diagnosed with DVT; stasis dermatitis was the most common alternative diagnosis. Taking a thorough history may elicit risk factors consistent with cellulitis, such as a recent injury with a break in the skin. On examination, cellulitis should be suspected for patients with fever and localized pain, redness, swelling, and warmth—the cardinal signs of dolor, rubor, tumor, and calor. An injury or entry site and leukocytosis also support the diagnosis of cellulitis. Distinct margins of erythema on the skin are highly suspicious for erysipelas.¹¹ Other physical findings (eg, laceration, purulent drainage, lymphangitic spread, fluctuating mass) also are consistent with a diagnosis of cellulitis.

The patient’s history is also essential in determining whether any DVT risk factors are present. Past medical history of VTE or CVA, or recent history of surgery, immobility, or trauma, should alert the clinician to the possibility of DVT. Family history of VTE increases the likelihood of DVT. Acute shortness of breath or chest pain in the setting of concerning lower extremity findings for DVT should raise concern for DVT and concurrent PE.

If the classic features of cellulitis are present, empiric antibiotics should be initiated. Routine ultrasound testing for all patients with cellulitis is of low value. However, as the incidence of DVT in this population is not negligible, those with VTE risk factors should be targeted for testing. Studies in the setting of cellulitis provide little guidance regarding specific risk factors that can be used to determine who should undergo further testing. Given this limitation, we suggest that clinicians incorporate into their decision making the well-established VTE risk factors identified for large populations studied in other settings, such as the postoperative period. Specifically, clinicians should consider ultrasound testing for patients with cellulitis and prior history of VTE; immobility; thrombophilia, CHF, and CVA with hemiparesis; or trauma and recent surgery.^10-12 Ultrasound should also be considered for patients with cellulitis that does not improve and for patients whose localized symptoms worsen despite use of antibiotics.

• Do not routinely perform ultrasound to rule out concurrent DVT in cases of cellulitis.

• Consider compression ultrasound if there is a history of VTE; immobility; thrombophilia, CHF, and CVA with hemiparesis; or trauma and recent surgery. Also consider it for patients who do not respond to antibiotics.

• In cases of cellulitis, avoid use of the Wells score alone or with D-dimer testing, as it likely overestimates the DVT risk.

The current evidence shows that, for most patients with cellulitis, routine ultrasound testing for DVT is unnecessary. Ultrasound should be considered for patients with potent VTE risk factors. If symptoms do not improve, or if they worsen despite use of antibiotics, clinicians should be alert to potential anchoring bias and consider DVT. The Wells clinical prediction rule overestimates the incidence of DVT in cellulitis and has little value in this setting.

Disclosure

Nothing to report.

Do you think this is a low-value practice? Is this truly a “Thing We Do for No Reason”? Let us know what you do in your practice and propose ideas for other “Things We Do for No Reason” topics. Please join in the conversation online at Twitter (#TWDFNR)/Facebook and don’t forget to “Like It” on Facebook or retweet it on Twitter. We invite you to propose ideas for other “Things We Do for No Reason” topics by emailing [email protected].