Brief Reports

Viral testing for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) of all patients admitted to the hospital is an appealing objective given the recognition of asymptomatic or minimally symptomatic infections. Yet such testing requires that all admitted patients be classified as persons under investigation (PUIs) until their test results are known. If an outside laboratory is used for the SARS-CoV-2 testing, the delay in obtaining results for these PUIs may cause significant personal protective equipment (PPE) use, postpone some care for non-coronavirus disease 2019 (COVID-19) conditions, block beds, and produce anxiety among staff and other patients. Rapid in-house testing of all admitted patients may resolve these issues but may be limited by the supply of reagents. To address this challenge, we piloted a pooled testing strategy for patients at low risk for SARS-CoV-2 admitted to a community hospital.

From April 17, 2020, to May 11, 2020, we implemented a pooled testing strategy using the GeneXpert^® System (Cepheid, Sunnyvale, California) at Saratoga Hospital, a 171-bed community hospital in upstate New York. Under normal procedures for this system, a single patient swab is placed in a vial containing viral transport media (VTM). An aliquot of this media is then transferred into a Xpert^® Xpress SARS CoV-2 test cartridge and assayed on the GeneXpert® instrument in our laboratory. Obtaining immediate results allowed us to assign admitted patients to either a COVID-19 or a non–COVID-19 unit, improving the issues associated with PUIs. Unfortunately, we did not have enough test cartridges to sustain this strategy of rapid individual testing of all admitted patients, and supply lines have remained uncertain.

We sought to conserve our limited Xpert Xpress SARS CoV-2 test cartridges using the strategy of pooled testing, a technique reported in Germany and by the University of Nebraska.^1,2 In this method, variable numbers of tests are pooled for a single analysis. If the test from the pooled vial is negative, these patients are all considered negative. If the pooled test is positive, all those patients need individual testing. This pooling method has been theorized to preserve test cartridges when the expected frequency of positive results is low.³

All patients admitted or placed on observation underwent SARS-CoV-2 PCR testing. The Emergency Department (ED) staff stratified patients into high or low risk to determine if they would be tested in a single send-out test (high risk) or a rapid in-house pooled group (low risk). High-risk patients were those with compatible history, physical exam, laboratory markers, and radiographic studies for COVID-19 disease. This often included increased supplemental oxygen requirement, multiple elevated inflammatory markers (including D-dimer, C-reactive protein, erythrocyte sedimentation rate, and ferritin levels), lymphopenia, and findings on chest radiograph or computed tomography scan including ground glass changes, multifocal pneumonia, or pneumonia. High-risk patients were admitted to the COVID unit or intensive care unit, had a send-out SARS-CoV-2 polymerase chain reaction (PCR) test, and were treated as a PUI until the results of their testing was known and correlated with their clinical course. Low-risk patients were those without complaints suggestive of COVID-19 infection and who may have had negative inflammatory markers, no significant lymphopenia, and negative imaging. 

The samples from 3 admitted patients thought to be at low-risk for COVID-19 using the clinical judgement of our ED staff were pooled for testing. All samples were obtained using nasopharyngeal swabs by experienced staff. The swabs from these patients were placed into a single vial of 3 mL VTM, maintaining the recommended 1 swab per mL of VTM. An aliquot of this media was then transferred into an Xpert Xpress SARS CoV-2 test cartridge and assayed on the GeneXpert instrument in our laboratory following manufacturer’s instructions. Based on analytic laboratory studies of the Cepheid Xpert Express SARS-CoV-2 test,⁴ we assume a clinical performance comparable to other reverse-transcriptase PCR (RT-PCR) tests, which have so far demonstrated sensitivities of 60% to 80% and specificities of 95% to 99%.⁵

Validation studies were performed on pools made from samples obtained from admitted patients with previously known positive and negative samples tested at the New York State Department of Health, Wadsworth Center laboratory (Albany, New York). A total of 14 samples were used for the instrument validation study, including three samples for pooled testing. The cycle threshold (Ct) value is defined as the number of PCR cycles required for the signal to be detectable. Ct values for each nucleic acid target of a known positive sample tested singly and in the pool with known negative patients were compared. A small shift in Ct values was noted between single and pooled testing, demonstrating no decrease in analytic sensitivity and suggesting that we would experience no decrease in clinical sensitivity.

We selected the pooling of 3 samples into 1 cartridge for several reasons: (1) 3-sample pools are well within the appropriate pooling size for the percentage positive rate in the population being tested. The use of larger pool sizes results in the need for more repeat testing when a positive result is obtained; (2) Given our supply lines, the projected savings would allow us to continue this strategy; and (3) Holding 3 patients in the ED until a pool was ready was manageable given our rate of admissions and ED volume.

The strategy required patients being held in the ED until a pooled group of 3 could be tested. On select occasions when holding patients in the ED to obtain a pool of 3 was not practical, 2 patients were tested in the pool. These decisions required close coordination between the laboratory, ED, and nursing staff.

This strategy resulted in 530 unique patient tests in 179 cartridges (172 with three swabs and 7 with two swabs). We had 4 positive pooled tests, requiring the use of 11 additional cartridges, for a positive rate of 0.8% (4/530) in this low-risk population (patients without COVID-19–related symptoms). There were no patients from negative pools who developed evidence of COVID-19 disease or tested positive for SARS-CoV-2 during their hospitalization. The total number of cartridges used was 190 and the number saved was 340.

The strategy of pooled testing for SARS-CoV-2 in patients admitted to our community hospital allowed us to continue rapid testing of admitted patients at low risk for COVID-19 disease during a period when supplies would otherwise not have been sufficient. We believe this strategy conserved PPE, led to a marked reduction in staff and patient anxiety, and improved patient care. Our impression is that testing all admitted patients has also been reassuring to our community. Like many others, we have observed that public fear of entering the hospital during this pandemic has caused delays in patients seeking care for non–COVID-19 conditions. We believe this strategy will help reduce those fears.

This strategy may require modification as the pandemic progresses. Our ED physicians were able to identify patients who they felt to be low risk for having COVID-19 disease based on signs, symptoms, and clinical impression during a time when we had an 8% positive rate among symptomatic outpatients and an estimated community positive rate in the range of 1% to 2%. If the rate of positive tests in our community rises, the use of pooling may need to be limited or the pool size reduced. If our supply of reagents is further limited or patient testing demand increases, the pool size may need to be increased. This will need to be balanced with our ability to hold patients in the ED while waiting for the pool size to be reached.

The strategy of pooled testing for SARS-CoV-2 has allowed us to continue to immediately test all admitted patients, thus improving patient care. It has required close coordination between multiple members of our laboratory and clinical staff and may require adjustment as the pandemic progresses. We believe it is a valuable tool during a time of limited resources that may have application in testing other low-risk groups, including healthcare workers and clients of occupational medicine services.

The authors gratefully acknowledge the support of Kirsten St. George, MAppSc, PhD, Director, Virology Laboratory, Wadsworth, NYSDOH, and the services supplied by the Wadsworth laboratory to our region.

Viral testing for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) of all patients admitted to the hospital is an appealing objective given the recognition of asymptomatic or minimally symptomatic infections. Yet such testing requires that all admitted patients be classified as persons under investigation (PUIs) until their test results are known. If an outside laboratory is used for the SARS-CoV-2 testing, the delay in obtaining results for these PUIs may cause significant personal protective equipment (PPE) use, postpone some care for non-coronavirus disease 2019 (COVID-19) conditions, block beds, and produce anxiety among staff and other patients. Rapid in-house testing of all admitted patients may resolve these issues but may be limited by the supply of reagents. To address this challenge, we piloted a pooled testing strategy for patients at low risk for SARS-CoV-2 admitted to a community hospital.

From April 17, 2020, to May 11, 2020, we implemented a pooled testing strategy using the GeneXpert^® System (Cepheid, Sunnyvale, California) at Saratoga Hospital, a 171-bed community hospital in upstate New York. Under normal procedures for this system, a single patient swab is placed in a vial containing viral transport media (VTM). An aliquot of this media is then transferred into a Xpert^® Xpress SARS CoV-2 test cartridge and assayed on the GeneXpert® instrument in our laboratory. Obtaining immediate results allowed us to assign admitted patients to either a COVID-19 or a non–COVID-19 unit, improving the issues associated with PUIs. Unfortunately, we did not have enough test cartridges to sustain this strategy of rapid individual testing of all admitted patients, and supply lines have remained uncertain.

We sought to conserve our limited Xpert Xpress SARS CoV-2 test cartridges using the strategy of pooled testing, a technique reported in Germany and by the University of Nebraska.^1,2 In this method, variable numbers of tests are pooled for a single analysis. If the test from the pooled vial is negative, these patients are all considered negative. If the pooled test is positive, all those patients need individual testing. This pooling method has been theorized to preserve test cartridges when the expected frequency of positive results is low.³

All patients admitted or placed on observation underwent SARS-CoV-2 PCR testing. The Emergency Department (ED) staff stratified patients into high or low risk to determine if they would be tested in a single send-out test (high risk) or a rapid in-house pooled group (low risk). High-risk patients were those with compatible history, physical exam, laboratory markers, and radiographic studies for COVID-19 disease. This often included increased supplemental oxygen requirement, multiple elevated inflammatory markers (including D-dimer, C-reactive protein, erythrocyte sedimentation rate, and ferritin levels), lymphopenia, and findings on chest radiograph or computed tomography scan including ground glass changes, multifocal pneumonia, or pneumonia. High-risk patients were admitted to the COVID unit or intensive care unit, had a send-out SARS-CoV-2 polymerase chain reaction (PCR) test, and were treated as a PUI until the results of their testing was known and correlated with their clinical course. Low-risk patients were those without complaints suggestive of COVID-19 infection and who may have had negative inflammatory markers, no significant lymphopenia, and negative imaging. 

The samples from 3 admitted patients thought to be at low-risk for COVID-19 using the clinical judgement of our ED staff were pooled for testing. All samples were obtained using nasopharyngeal swabs by experienced staff. The swabs from these patients were placed into a single vial of 3 mL VTM, maintaining the recommended 1 swab per mL of VTM. An aliquot of this media was then transferred into an Xpert Xpress SARS CoV-2 test cartridge and assayed on the GeneXpert instrument in our laboratory following manufacturer’s instructions. Based on analytic laboratory studies of the Cepheid Xpert Express SARS-CoV-2 test,⁴ we assume a clinical performance comparable to other reverse-transcriptase PCR (RT-PCR) tests, which have so far demonstrated sensitivities of 60% to 80% and specificities of 95% to 99%.⁵

Validation studies were performed on pools made from samples obtained from admitted patients with previously known positive and negative samples tested at the New York State Department of Health, Wadsworth Center laboratory (Albany, New York). A total of 14 samples were used for the instrument validation study, including three samples for pooled testing. The cycle threshold (Ct) value is defined as the number of PCR cycles required for the signal to be detectable. Ct values for each nucleic acid target of a known positive sample tested singly and in the pool with known negative patients were compared. A small shift in Ct values was noted between single and pooled testing, demonstrating no decrease in analytic sensitivity and suggesting that we would experience no decrease in clinical sensitivity.

We selected the pooling of 3 samples into 1 cartridge for several reasons: (1) 3-sample pools are well within the appropriate pooling size for the percentage positive rate in the population being tested. The use of larger pool sizes results in the need for more repeat testing when a positive result is obtained; (2) Given our supply lines, the projected savings would allow us to continue this strategy; and (3) Holding 3 patients in the ED until a pool was ready was manageable given our rate of admissions and ED volume.

The strategy required patients being held in the ED until a pooled group of 3 could be tested. On select occasions when holding patients in the ED to obtain a pool of 3 was not practical, 2 patients were tested in the pool. These decisions required close coordination between the laboratory, ED, and nursing staff.

This strategy resulted in 530 unique patient tests in 179 cartridges (172 with three swabs and 7 with two swabs). We had 4 positive pooled tests, requiring the use of 11 additional cartridges, for a positive rate of 0.8% (4/530) in this low-risk population (patients without COVID-19–related symptoms). There were no patients from negative pools who developed evidence of COVID-19 disease or tested positive for SARS-CoV-2 during their hospitalization. The total number of cartridges used was 190 and the number saved was 340.

The strategy of pooled testing for SARS-CoV-2 in patients admitted to our community hospital allowed us to continue rapid testing of admitted patients at low risk for COVID-19 disease during a period when supplies would otherwise not have been sufficient. We believe this strategy conserved PPE, led to a marked reduction in staff and patient anxiety, and improved patient care. Our impression is that testing all admitted patients has also been reassuring to our community. Like many others, we have observed that public fear of entering the hospital during this pandemic has caused delays in patients seeking care for non–COVID-19 conditions. We believe this strategy will help reduce those fears.

This strategy may require modification as the pandemic progresses. Our ED physicians were able to identify patients who they felt to be low risk for having COVID-19 disease based on signs, symptoms, and clinical impression during a time when we had an 8% positive rate among symptomatic outpatients and an estimated community positive rate in the range of 1% to 2%. If the rate of positive tests in our community rises, the use of pooling may need to be limited or the pool size reduced. If our supply of reagents is further limited or patient testing demand increases, the pool size may need to be increased. This will need to be balanced with our ability to hold patients in the ED while waiting for the pool size to be reached.

The strategy of pooled testing for SARS-CoV-2 has allowed us to continue to immediately test all admitted patients, thus improving patient care. It has required close coordination between multiple members of our laboratory and clinical staff and may require adjustment as the pandemic progresses. We believe it is a valuable tool during a time of limited resources that may have application in testing other low-risk groups, including healthcare workers and clients of occupational medicine services.

The authors gratefully acknowledge the support of Kirsten St. George, MAppSc, PhD, Director, Virology Laboratory, Wadsworth, NYSDOH, and the services supplied by the Wadsworth laboratory to our region.

Viral testing for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) of all patients admitted to the hospital is an appealing objective given the recognition of asymptomatic or minimally symptomatic infections. Yet such testing requires that all admitted patients be classified as persons under investigation (PUIs) until their test results are known. If an outside laboratory is used for the SARS-CoV-2 testing, the delay in obtaining results for these PUIs may cause significant personal protective equipment (PPE) use, postpone some care for non-coronavirus disease 2019 (COVID-19) conditions, block beds, and produce anxiety among staff and other patients. Rapid in-house testing of all admitted patients may resolve these issues but may be limited by the supply of reagents. To address this challenge, we piloted a pooled testing strategy for patients at low risk for SARS-CoV-2 admitted to a community hospital.

From April 17, 2020, to May 11, 2020, we implemented a pooled testing strategy using the GeneXpert^® System (Cepheid, Sunnyvale, California) at Saratoga Hospital, a 171-bed community hospital in upstate New York. Under normal procedures for this system, a single patient swab is placed in a vial containing viral transport media (VTM). An aliquot of this media is then transferred into a Xpert^® Xpress SARS CoV-2 test cartridge and assayed on the GeneXpert® instrument in our laboratory. Obtaining immediate results allowed us to assign admitted patients to either a COVID-19 or a non–COVID-19 unit, improving the issues associated with PUIs. Unfortunately, we did not have enough test cartridges to sustain this strategy of rapid individual testing of all admitted patients, and supply lines have remained uncertain.

We sought to conserve our limited Xpert Xpress SARS CoV-2 test cartridges using the strategy of pooled testing, a technique reported in Germany and by the University of Nebraska.^1,2 In this method, variable numbers of tests are pooled for a single analysis. If the test from the pooled vial is negative, these patients are all considered negative. If the pooled test is positive, all those patients need individual testing. This pooling method has been theorized to preserve test cartridges when the expected frequency of positive results is low.³

All patients admitted or placed on observation underwent SARS-CoV-2 PCR testing. The Emergency Department (ED) staff stratified patients into high or low risk to determine if they would be tested in a single send-out test (high risk) or a rapid in-house pooled group (low risk). High-risk patients were those with compatible history, physical exam, laboratory markers, and radiographic studies for COVID-19 disease. This often included increased supplemental oxygen requirement, multiple elevated inflammatory markers (including D-dimer, C-reactive protein, erythrocyte sedimentation rate, and ferritin levels), lymphopenia, and findings on chest radiograph or computed tomography scan including ground glass changes, multifocal pneumonia, or pneumonia. High-risk patients were admitted to the COVID unit or intensive care unit, had a send-out SARS-CoV-2 polymerase chain reaction (PCR) test, and were treated as a PUI until the results of their testing was known and correlated with their clinical course. Low-risk patients were those without complaints suggestive of COVID-19 infection and who may have had negative inflammatory markers, no significant lymphopenia, and negative imaging. 

The samples from 3 admitted patients thought to be at low-risk for COVID-19 using the clinical judgement of our ED staff were pooled for testing. All samples were obtained using nasopharyngeal swabs by experienced staff. The swabs from these patients were placed into a single vial of 3 mL VTM, maintaining the recommended 1 swab per mL of VTM. An aliquot of this media was then transferred into an Xpert Xpress SARS CoV-2 test cartridge and assayed on the GeneXpert instrument in our laboratory following manufacturer’s instructions. Based on analytic laboratory studies of the Cepheid Xpert Express SARS-CoV-2 test,⁴ we assume a clinical performance comparable to other reverse-transcriptase PCR (RT-PCR) tests, which have so far demonstrated sensitivities of 60% to 80% and specificities of 95% to 99%.⁵

Validation studies were performed on pools made from samples obtained from admitted patients with previously known positive and negative samples tested at the New York State Department of Health, Wadsworth Center laboratory (Albany, New York). A total of 14 samples were used for the instrument validation study, including three samples for pooled testing. The cycle threshold (Ct) value is defined as the number of PCR cycles required for the signal to be detectable. Ct values for each nucleic acid target of a known positive sample tested singly and in the pool with known negative patients were compared. A small shift in Ct values was noted between single and pooled testing, demonstrating no decrease in analytic sensitivity and suggesting that we would experience no decrease in clinical sensitivity.

We selected the pooling of 3 samples into 1 cartridge for several reasons: (1) 3-sample pools are well within the appropriate pooling size for the percentage positive rate in the population being tested. The use of larger pool sizes results in the need for more repeat testing when a positive result is obtained; (2) Given our supply lines, the projected savings would allow us to continue this strategy; and (3) Holding 3 patients in the ED until a pool was ready was manageable given our rate of admissions and ED volume.

The strategy required patients being held in the ED until a pooled group of 3 could be tested. On select occasions when holding patients in the ED to obtain a pool of 3 was not practical, 2 patients were tested in the pool. These decisions required close coordination between the laboratory, ED, and nursing staff.

This strategy resulted in 530 unique patient tests in 179 cartridges (172 with three swabs and 7 with two swabs). We had 4 positive pooled tests, requiring the use of 11 additional cartridges, for a positive rate of 0.8% (4/530) in this low-risk population (patients without COVID-19–related symptoms). There were no patients from negative pools who developed evidence of COVID-19 disease or tested positive for SARS-CoV-2 during their hospitalization. The total number of cartridges used was 190 and the number saved was 340.

The strategy of pooled testing for SARS-CoV-2 in patients admitted to our community hospital allowed us to continue rapid testing of admitted patients at low risk for COVID-19 disease during a period when supplies would otherwise not have been sufficient. We believe this strategy conserved PPE, led to a marked reduction in staff and patient anxiety, and improved patient care. Our impression is that testing all admitted patients has also been reassuring to our community. Like many others, we have observed that public fear of entering the hospital during this pandemic has caused delays in patients seeking care for non–COVID-19 conditions. We believe this strategy will help reduce those fears.

This strategy may require modification as the pandemic progresses. Our ED physicians were able to identify patients who they felt to be low risk for having COVID-19 disease based on signs, symptoms, and clinical impression during a time when we had an 8% positive rate among symptomatic outpatients and an estimated community positive rate in the range of 1% to 2%. If the rate of positive tests in our community rises, the use of pooling may need to be limited or the pool size reduced. If our supply of reagents is further limited or patient testing demand increases, the pool size may need to be increased. This will need to be balanced with our ability to hold patients in the ED while waiting for the pool size to be reached.

The strategy of pooled testing for SARS-CoV-2 has allowed us to continue to immediately test all admitted patients, thus improving patient care. It has required close coordination between multiple members of our laboratory and clinical staff and may require adjustment as the pandemic progresses. We believe it is a valuable tool during a time of limited resources that may have application in testing other low-risk groups, including healthcare workers and clients of occupational medicine services.

The authors gratefully acknowledge the support of Kirsten St. George, MAppSc, PhD, Director, Virology Laboratory, Wadsworth, NYSDOH, and the services supplied by the Wadsworth laboratory to our region.

The Centers for Medicare & Medicaid Services (CMS) Hospital Compare overall hospital ratings was originally released in 2016 and was recently updated in February 2019.^1,2 The program is designed to provide a consumer-friendly global rating system for hospitals, with hospitals rated on a scale from one star (worst) to five stars (best). The ratings are based on a formula that combines scores on 57 performance measures into seven groups, with the groups of mortality, safety, readmission, and patient experience given weights of 22% each in the overall scoring, and groups of effectiveness of care, timeliness of care, and efficient use of medical imaging equally contributing to the rest of the score.

Concerns have been raised since the introduction of the program regarding the methodology and possible unfairly high or low star ratings for certain types of hospitals.^3,4 It has been noted that five-star hospitals are disproportionately small, specialty-focused hospitals that may not have Emergency Departments or significant volumes of Medicaid patients.⁵ Hospitals that report fewer measures and thus receive scores for fewer measure groups (in general, smaller or specialty hospitals) are more likely to receive higher star ratings than are hospitals that receive scores for all measure groups.^6,7 Teaching hospitals, on average, have received lower star ratings than nonteaching hospitals.^8,9

Multihospital systems generally designate one of their hospitals as a “flagship” hospital and often use the name of that hospital to identify the system as a whole (eg, Mayo Clinic Health System, University of Pittsburgh Medical Center). There is not a set of objective criteria to designate a “flagship” hospital of a multihospital health system. Flagships could be the founding hospitals of the systems or the largest hospitals in the systems, and they are usually (although not always) large teaching hospitals. There is therefore a potential paradox in which a set of hospitals that tend to get lower ratings in the CMS star rating system may also be the set frequently identified as system flagship hospitals and whose reputation is used as a brand identity for multihospital systems.

It is possible, though, that the hospitals designated as flagship hospitals in multihospital systems are exceptions to the general rule of lower star ratings for major teaching hospitals. The flagship designation may reflect excellence that is then reflected in the star rating system, or it may reflect some other kind of excellence (eg, reputation for research or teaching, diverse medical services provided) that is not reflected in the star rating system. The primary aim of this study was to compare the average star ratings and hospital characteristics of designated flagship hospitals in multihospital systems with those of (1) major teaching hospitals generally and (2) “nonflagship” hospitals across and within the same systems specifically. We sought to determine whether a flagship designation would be associated with higher star ratings than those of major teaching hospitals in general and with higher star ratings than other, nonflagship hospitals in the same system.

The use of a prestigious flagship hospital name to identify a multihospital system suggests that some aspects of high quality in the flagship are extended in some way to other hospitals in the system. If that is so, then the star ratings of hospitals in organized multihospital systems with a flagship may be more similar to each other than those of sets of hospitals selected at random. As a secondary aim, to determine whether this type of consistent quality throughout a system could be identified in the CMS hospital star rating system, we compared the variation in star ratings between organized multihospital systems with flagship hospitals to those of artificially created “pseudo systems” of unaffiliated hospitals.

We used the Agency for Healthcare Research and Quality (AHRQ) Compendium of U.S. Health Systems, 2016, database and hospital file to identify multihospital health systems and their member hospitals.¹⁰ The database also provides information about health system characteristics such as systemwide teaching intensity and total number of acute care hospitals. We linked the AHRQ files to the CMS Hospital Compare datasets and Hospital Inpatient Prospective Payment System (IPPS) 2018 Final Rule Impact File to obtain star ratings and other information about specific hospitals (eg, resident to bed ratio, uncompensated care payment). Throughout the study, we followed the AHRQ’s definition of “major teaching hospitals” as hospitals with a high resident to bed ratio (≥0.25).

For purposes of this study, the primary criterion for identification of flagship hospitals was an explicit designation by the parent health systems on their websites, in the systems’ official documents, or in press releases or through major media reports. In the few cases in which parent systems did not designate their flagships, we searched reliable online sources such as major newspapers and hospital reviews to see if there was an agreement among sources on the flagship status. If we could not unambiguously identify a flagship hospital in a multihospital system using these methods, the system was not included in the study. A health system could have more than one flagship hospital.

Because the concept of “flagship” often involves a role as a referral center for complex cases in a regional area small enough to have referrals from hospital to hospital within the same system, we excluded multistate national health systems (eg, Catholic Health Initiatives, Community Health Systems, Inc.) and health systems with no major teaching hospitals or no flagship(s) identified by the systems themselves. Non-acute care and stand-alone hospitals, hospitals with missing CMS Certification Numbers (CCNs) or unmatched CCNs or hospital types across different data files, and hospitals without a star rating, were excluded.

Our analyses were performed at both hospital and health system levels. In the hospital-level analysis, we grouped hospitals into “1-2 star,” “3 star,” and “4-5 star” rating categories. We first compared star ratings of flagship hospitals with those of major teaching hospitals in general (ie, hospitals in the CMS Hospital Compare database with resident to bed ratios ≥0.25 that were not designated as system flagship hospitals). We then compared the average flagship hospital and average nonflagship hospital star ratings pooled across all the health systems. To explore hospital-level characteristics that might be associated with flagship hospitals’ performance on star ratings, we compared hospitals’ teaching intensity, bed size, charity care, and disproportionate share hospital (DSH) patient percentage between flagship and major teaching hospitals and between flagship and nonflagship hospitals. Differences were tested using two-sample t test with equal variances. We also compared hospital characteristics among hospitals with 1-2 stars, 3 stars, and 4-5 stars with use of one-way analysis of variance (ANOVA) with Bonferroni adjustment for multiple comparisons.

In the system-level analysis, we examined flagship hospitals’ star ratings relative to the star ratings for other member hospitals in the same system. We assigned health systems to the following three groups according to their flagship hospitals’ star ratings in comparison to other hospitals within their own systems: health systems in which flagship hospitals were rated the lowest among all member hospitals, health systems in which flagship hospitals were rated neither highest nor lowest or all hospitals within the system had the same star rating, and health systems in which flagship hospitals were rated the highest among all member hospitals. We compared system-level characteristics of the three groups. We calculated the average differences in uncompensated care payment, resident to bed ratio, DSH patient percentage, and total beds between flagship hospitals and nonflagship hospitals of the same health systems, and we also compared the differences across the three health system groups defined previously. We conducted an analysis of covariance (ANCOVA) to take system-level factors into consideration, including system size (total number of acute care hospitals in the system), systemwide teaching intensity, and systemwide charity care. The Bonferroni correction was used to adjust for potential problems of multiple comparisons.

Finally, to compare the diversity of star ratings within health systems and the diversity of star ratings nationwide, we generated a set of 100 pseudo systems each comprising six member hospitals (corresponding to the average number of member hospitals per “true” health system included in the study) that were randomly selected from all hospitals excluded from this study. We calculated and compared the average standard deviations of star ratings between the true health systems and this set of pseudo systems. Differences were tested using two-sample t test with equal variances.

Data management and statistical analyses were conducted using Stata SE, version 13.0 (StataCorp LLC, College Station, Texas).

Our final analysis included 599 hospitals in 113 health systems; 119 hospitals were flagships (four health systems each had two flagship hospitals, and one health system had three flagship hospitals). All other hospitals (n = 480) were designated as nonflaghips. On average, each health system had 6 member hospitals with star ratings, with a range from 2 to 22.

Flagship hospitals did have higher average star ratings than major teaching hospitals (mean star rating, 2.8 vs 2.3, respectively; P < .01; Figure). A larger proportion of flagship hospitals received four or five stars than did major teaching hospitals (29% vs 20%, respectively), and a smaller proportion of them received one or two stars (44% vs 59%, respectively; P < .05).

Flagship hospitals had lower star ratings on average, across all systems, than did nonflagship hospitals (mean star rating, 2.8 vs 3.3, respectively; P < .001). A smaller proportion of flagships received four or five stars than did nonflagships (29% vs 44%, respectively), and a larger proportion of them received one or two stars (44% vs 23%, respectively; P < .001).

As expected, flagship hospitals had significantly higher teaching intensity, larger bed size, higher DSH patient percentage, and higher value of uncompensated care payments than did nonflagship hospitals (P < .001 for all). On average, flagship hospitals were significantly larger but had lower DSH patient percentage and lower value of uncompensated care payments than did major teaching hospitals in general (P < .01 for all). In all types of hospitals, four- or five-star hospitals consistently had significantly lower DSH patient percentage (P < .001) and lower value of uncompensated care payment per claim (P < .05) than did other hospitals (Table).

In half of all health systems (n = 56), flagship hospitals were rated the lowest of all hospitals within that system; in approximately 20% of all health systems (n = 22), flagship hospitals were rated the highest. Flagship hospitals were more likely to have the lowest star rating in the system if the within-system difference in DSH patient percentage between flagship and nonflagship hospitals was relatively large. Within-system DSH patient percentage differences between flagship and nonflagship hospitals were 12.4%, 5.4%, and 3.5% in “flagship rated lowest,” “flagship rated middle,” and “flagship rated highest” systems, respectively (P < .05).

Average standardized deviations of star ratings for the 113 true health systems and 100 randomly generated pseudo health systems were 0.86 and 0.97, respectively (P < .05).

System-designated flagship hospitals did not generally have higher star ratings than did the other, smaller, community hospitals, either on average or within their own systems. In fact, the most common pattern observed was the system-designated flagship hospitals had the lowest star rating in their system. Flagship hospitals in multihospital systems were, however, rated higher than major teaching hospitals in general. The safety-net role of many of the system flagship hospitals, as captured by relative DSH percentage, was the most important determinant of low star ratings. A high bed number and teaching status were not as strongly associated with low star ratings.

It is already well established that the CMS star rating system does not correspond to other global hospital ratings systems like those of US News & World Report, Healthgrades, or the Leapfrog Group.¹¹ Each global rating system uses a unique set of measures and weighting systems for those measures, so discrepancies among these systems are inevitable. Multihospital systems may feel that the positive reputation for tertiary care excellence held by a flagship hospital is captured in a rating system like US News that has an explicit reputation component¹² and that the US News rankings are more prominent in the public eye than are those of CMS. To the extent that the CMS star ratings do become more widely used by the public or by payers to establish narrow provider networks, the relatively low ratings of multisystem flagship hospitals may become a cause for concern for those hospitals and systems.

System-designated flagship hospitals are typically large teaching hospitals with higher levels of technology, more highly specialized services and medical staff, more extensive research programs and active clinical trials programs, and the ability to treat cases that are difficult or complex or instances of rare conditions. They are not generally, as it turns out, the hospitals in a given system that the CMS star rating system identifies as “best.” In a number of multihospital systems, the system name is derived from the name of the flagship hospital (eg, Yale New Haven Health System and Montefiore Health System), which suggests that the system finds a marketing or branding advantage in being publicly identified with the name and positive reputation of the flagship hospital. Flagship hospitals may be designated as such because they have other attributes that patients, the community, and the system value, which may not be represented by the CMS quality metrics summarized by star ratings.

We did find a somewhat lower level of variation in star ratings in actual multihospital systems than in a set of randomly created “pseudo systems,” suggesting the presence of some mechanism for quality management in those systems leading to a more similar set of star ratings than one would find in hospitals selected at random.

Our study has a few limitations. First, we excluded multihospital health systems without any major teaching member hospital, which was based on our observation that they do not usually designate their flagship hospitals or they do not have any identifiable flagship hospitals. There may be a small number of such health systems that have designated their flagship hospitals and were excluded from the study, but we do not believe it will change our key findings. Second, it was possible that multiple hospitals in the same health system reported under the same CCN (multicampuses will often use the flagship facility’s IDs for the purposes of claims processing or cost and measure reporting), and therefore, the star ratings for the flagship hospitals reflected the performance of both the flagship hospital and the other member hospitals sharing the same CCN. We cannot fix the underlying reporting issue, and as a result, part of our analysis was probably more of a comparison of the “financial” flagship with other more loosely associated hospitals in the system. We could have in fact overestimated the flagships’ star rating performance by including data of other better performing nonflagship hospitals.

System-designated flagship hospitals tended to have lower CMS Hospital Compare overall hospital quality star ratings than did nonflagship hospitals in the same multihospital systems. The characteristics of hospitals identified as system flagships do not seem well aligned with those associated with better performance in the star rating system.

The authors declared no conflicts of interest.

The Centers for Medicare & Medicaid Services (CMS) Hospital Compare overall hospital ratings was originally released in 2016 and was recently updated in February 2019.^1,2 The program is designed to provide a consumer-friendly global rating system for hospitals, with hospitals rated on a scale from one star (worst) to five stars (best). The ratings are based on a formula that combines scores on 57 performance measures into seven groups, with the groups of mortality, safety, readmission, and patient experience given weights of 22% each in the overall scoring, and groups of effectiveness of care, timeliness of care, and efficient use of medical imaging equally contributing to the rest of the score.

Concerns have been raised since the introduction of the program regarding the methodology and possible unfairly high or low star ratings for certain types of hospitals.^3,4 It has been noted that five-star hospitals are disproportionately small, specialty-focused hospitals that may not have Emergency Departments or significant volumes of Medicaid patients.⁵ Hospitals that report fewer measures and thus receive scores for fewer measure groups (in general, smaller or specialty hospitals) are more likely to receive higher star ratings than are hospitals that receive scores for all measure groups.^6,7 Teaching hospitals, on average, have received lower star ratings than nonteaching hospitals.^8,9

Multihospital systems generally designate one of their hospitals as a “flagship” hospital and often use the name of that hospital to identify the system as a whole (eg, Mayo Clinic Health System, University of Pittsburgh Medical Center). There is not a set of objective criteria to designate a “flagship” hospital of a multihospital health system. Flagships could be the founding hospitals of the systems or the largest hospitals in the systems, and they are usually (although not always) large teaching hospitals. There is therefore a potential paradox in which a set of hospitals that tend to get lower ratings in the CMS star rating system may also be the set frequently identified as system flagship hospitals and whose reputation is used as a brand identity for multihospital systems.

It is possible, though, that the hospitals designated as flagship hospitals in multihospital systems are exceptions to the general rule of lower star ratings for major teaching hospitals. The flagship designation may reflect excellence that is then reflected in the star rating system, or it may reflect some other kind of excellence (eg, reputation for research or teaching, diverse medical services provided) that is not reflected in the star rating system. The primary aim of this study was to compare the average star ratings and hospital characteristics of designated flagship hospitals in multihospital systems with those of (1) major teaching hospitals generally and (2) “nonflagship” hospitals across and within the same systems specifically. We sought to determine whether a flagship designation would be associated with higher star ratings than those of major teaching hospitals in general and with higher star ratings than other, nonflagship hospitals in the same system.

The use of a prestigious flagship hospital name to identify a multihospital system suggests that some aspects of high quality in the flagship are extended in some way to other hospitals in the system. If that is so, then the star ratings of hospitals in organized multihospital systems with a flagship may be more similar to each other than those of sets of hospitals selected at random. As a secondary aim, to determine whether this type of consistent quality throughout a system could be identified in the CMS hospital star rating system, we compared the variation in star ratings between organized multihospital systems with flagship hospitals to those of artificially created “pseudo systems” of unaffiliated hospitals.

We used the Agency for Healthcare Research and Quality (AHRQ) Compendium of U.S. Health Systems, 2016, database and hospital file to identify multihospital health systems and their member hospitals.¹⁰ The database also provides information about health system characteristics such as systemwide teaching intensity and total number of acute care hospitals. We linked the AHRQ files to the CMS Hospital Compare datasets and Hospital Inpatient Prospective Payment System (IPPS) 2018 Final Rule Impact File to obtain star ratings and other information about specific hospitals (eg, resident to bed ratio, uncompensated care payment). Throughout the study, we followed the AHRQ’s definition of “major teaching hospitals” as hospitals with a high resident to bed ratio (≥0.25).

For purposes of this study, the primary criterion for identification of flagship hospitals was an explicit designation by the parent health systems on their websites, in the systems’ official documents, or in press releases or through major media reports. In the few cases in which parent systems did not designate their flagships, we searched reliable online sources such as major newspapers and hospital reviews to see if there was an agreement among sources on the flagship status. If we could not unambiguously identify a flagship hospital in a multihospital system using these methods, the system was not included in the study. A health system could have more than one flagship hospital.

Because the concept of “flagship” often involves a role as a referral center for complex cases in a regional area small enough to have referrals from hospital to hospital within the same system, we excluded multistate national health systems (eg, Catholic Health Initiatives, Community Health Systems, Inc.) and health systems with no major teaching hospitals or no flagship(s) identified by the systems themselves. Non-acute care and stand-alone hospitals, hospitals with missing CMS Certification Numbers (CCNs) or unmatched CCNs or hospital types across different data files, and hospitals without a star rating, were excluded.

Our analyses were performed at both hospital and health system levels. In the hospital-level analysis, we grouped hospitals into “1-2 star,” “3 star,” and “4-5 star” rating categories. We first compared star ratings of flagship hospitals with those of major teaching hospitals in general (ie, hospitals in the CMS Hospital Compare database with resident to bed ratios ≥0.25 that were not designated as system flagship hospitals). We then compared the average flagship hospital and average nonflagship hospital star ratings pooled across all the health systems. To explore hospital-level characteristics that might be associated with flagship hospitals’ performance on star ratings, we compared hospitals’ teaching intensity, bed size, charity care, and disproportionate share hospital (DSH) patient percentage between flagship and major teaching hospitals and between flagship and nonflagship hospitals. Differences were tested using two-sample t test with equal variances. We also compared hospital characteristics among hospitals with 1-2 stars, 3 stars, and 4-5 stars with use of one-way analysis of variance (ANOVA) with Bonferroni adjustment for multiple comparisons.

In the system-level analysis, we examined flagship hospitals’ star ratings relative to the star ratings for other member hospitals in the same system. We assigned health systems to the following three groups according to their flagship hospitals’ star ratings in comparison to other hospitals within their own systems: health systems in which flagship hospitals were rated the lowest among all member hospitals, health systems in which flagship hospitals were rated neither highest nor lowest or all hospitals within the system had the same star rating, and health systems in which flagship hospitals were rated the highest among all member hospitals. We compared system-level characteristics of the three groups. We calculated the average differences in uncompensated care payment, resident to bed ratio, DSH patient percentage, and total beds between flagship hospitals and nonflagship hospitals of the same health systems, and we also compared the differences across the three health system groups defined previously. We conducted an analysis of covariance (ANCOVA) to take system-level factors into consideration, including system size (total number of acute care hospitals in the system), systemwide teaching intensity, and systemwide charity care. The Bonferroni correction was used to adjust for potential problems of multiple comparisons.

Finally, to compare the diversity of star ratings within health systems and the diversity of star ratings nationwide, we generated a set of 100 pseudo systems each comprising six member hospitals (corresponding to the average number of member hospitals per “true” health system included in the study) that were randomly selected from all hospitals excluded from this study. We calculated and compared the average standard deviations of star ratings between the true health systems and this set of pseudo systems. Differences were tested using two-sample t test with equal variances.

Data management and statistical analyses were conducted using Stata SE, version 13.0 (StataCorp LLC, College Station, Texas).

Our final analysis included 599 hospitals in 113 health systems; 119 hospitals were flagships (four health systems each had two flagship hospitals, and one health system had three flagship hospitals). All other hospitals (n = 480) were designated as nonflaghips. On average, each health system had 6 member hospitals with star ratings, with a range from 2 to 22.

Flagship hospitals did have higher average star ratings than major teaching hospitals (mean star rating, 2.8 vs 2.3, respectively; P < .01; Figure). A larger proportion of flagship hospitals received four or five stars than did major teaching hospitals (29% vs 20%, respectively), and a smaller proportion of them received one or two stars (44% vs 59%, respectively; P < .05).

Flagship hospitals had lower star ratings on average, across all systems, than did nonflagship hospitals (mean star rating, 2.8 vs 3.3, respectively; P < .001). A smaller proportion of flagships received four or five stars than did nonflagships (29% vs 44%, respectively), and a larger proportion of them received one or two stars (44% vs 23%, respectively; P < .001).

As expected, flagship hospitals had significantly higher teaching intensity, larger bed size, higher DSH patient percentage, and higher value of uncompensated care payments than did nonflagship hospitals (P < .001 for all). On average, flagship hospitals were significantly larger but had lower DSH patient percentage and lower value of uncompensated care payments than did major teaching hospitals in general (P < .01 for all). In all types of hospitals, four- or five-star hospitals consistently had significantly lower DSH patient percentage (P < .001) and lower value of uncompensated care payment per claim (P < .05) than did other hospitals (Table).

In half of all health systems (n = 56), flagship hospitals were rated the lowest of all hospitals within that system; in approximately 20% of all health systems (n = 22), flagship hospitals were rated the highest. Flagship hospitals were more likely to have the lowest star rating in the system if the within-system difference in DSH patient percentage between flagship and nonflagship hospitals was relatively large. Within-system DSH patient percentage differences between flagship and nonflagship hospitals were 12.4%, 5.4%, and 3.5% in “flagship rated lowest,” “flagship rated middle,” and “flagship rated highest” systems, respectively (P < .05).

Average standardized deviations of star ratings for the 113 true health systems and 100 randomly generated pseudo health systems were 0.86 and 0.97, respectively (P < .05).

System-designated flagship hospitals did not generally have higher star ratings than did the other, smaller, community hospitals, either on average or within their own systems. In fact, the most common pattern observed was the system-designated flagship hospitals had the lowest star rating in their system. Flagship hospitals in multihospital systems were, however, rated higher than major teaching hospitals in general. The safety-net role of many of the system flagship hospitals, as captured by relative DSH percentage, was the most important determinant of low star ratings. A high bed number and teaching status were not as strongly associated with low star ratings.

It is already well established that the CMS star rating system does not correspond to other global hospital ratings systems like those of US News & World Report, Healthgrades, or the Leapfrog Group.¹¹ Each global rating system uses a unique set of measures and weighting systems for those measures, so discrepancies among these systems are inevitable. Multihospital systems may feel that the positive reputation for tertiary care excellence held by a flagship hospital is captured in a rating system like US News that has an explicit reputation component¹² and that the US News rankings are more prominent in the public eye than are those of CMS. To the extent that the CMS star ratings do become more widely used by the public or by payers to establish narrow provider networks, the relatively low ratings of multisystem flagship hospitals may become a cause for concern for those hospitals and systems.

System-designated flagship hospitals are typically large teaching hospitals with higher levels of technology, more highly specialized services and medical staff, more extensive research programs and active clinical trials programs, and the ability to treat cases that are difficult or complex or instances of rare conditions. They are not generally, as it turns out, the hospitals in a given system that the CMS star rating system identifies as “best.” In a number of multihospital systems, the system name is derived from the name of the flagship hospital (eg, Yale New Haven Health System and Montefiore Health System), which suggests that the system finds a marketing or branding advantage in being publicly identified with the name and positive reputation of the flagship hospital. Flagship hospitals may be designated as such because they have other attributes that patients, the community, and the system value, which may not be represented by the CMS quality metrics summarized by star ratings.

We did find a somewhat lower level of variation in star ratings in actual multihospital systems than in a set of randomly created “pseudo systems,” suggesting the presence of some mechanism for quality management in those systems leading to a more similar set of star ratings than one would find in hospitals selected at random.

Our study has a few limitations. First, we excluded multihospital health systems without any major teaching member hospital, which was based on our observation that they do not usually designate their flagship hospitals or they do not have any identifiable flagship hospitals. There may be a small number of such health systems that have designated their flagship hospitals and were excluded from the study, but we do not believe it will change our key findings. Second, it was possible that multiple hospitals in the same health system reported under the same CCN (multicampuses will often use the flagship facility’s IDs for the purposes of claims processing or cost and measure reporting), and therefore, the star ratings for the flagship hospitals reflected the performance of both the flagship hospital and the other member hospitals sharing the same CCN. We cannot fix the underlying reporting issue, and as a result, part of our analysis was probably more of a comparison of the “financial” flagship with other more loosely associated hospitals in the system. We could have in fact overestimated the flagships’ star rating performance by including data of other better performing nonflagship hospitals.

System-designated flagship hospitals tended to have lower CMS Hospital Compare overall hospital quality star ratings than did nonflagship hospitals in the same multihospital systems. The characteristics of hospitals identified as system flagships do not seem well aligned with those associated with better performance in the star rating system.

The authors declared no conflicts of interest.

The Centers for Medicare & Medicaid Services (CMS) Hospital Compare overall hospital ratings was originally released in 2016 and was recently updated in February 2019.^1,2 The program is designed to provide a consumer-friendly global rating system for hospitals, with hospitals rated on a scale from one star (worst) to five stars (best). The ratings are based on a formula that combines scores on 57 performance measures into seven groups, with the groups of mortality, safety, readmission, and patient experience given weights of 22% each in the overall scoring, and groups of effectiveness of care, timeliness of care, and efficient use of medical imaging equally contributing to the rest of the score.

Concerns have been raised since the introduction of the program regarding the methodology and possible unfairly high or low star ratings for certain types of hospitals.^3,4 It has been noted that five-star hospitals are disproportionately small, specialty-focused hospitals that may not have Emergency Departments or significant volumes of Medicaid patients.⁵ Hospitals that report fewer measures and thus receive scores for fewer measure groups (in general, smaller or specialty hospitals) are more likely to receive higher star ratings than are hospitals that receive scores for all measure groups.^6,7 Teaching hospitals, on average, have received lower star ratings than nonteaching hospitals.^8,9

Multihospital systems generally designate one of their hospitals as a “flagship” hospital and often use the name of that hospital to identify the system as a whole (eg, Mayo Clinic Health System, University of Pittsburgh Medical Center). There is not a set of objective criteria to designate a “flagship” hospital of a multihospital health system. Flagships could be the founding hospitals of the systems or the largest hospitals in the systems, and they are usually (although not always) large teaching hospitals. There is therefore a potential paradox in which a set of hospitals that tend to get lower ratings in the CMS star rating system may also be the set frequently identified as system flagship hospitals and whose reputation is used as a brand identity for multihospital systems.

It is possible, though, that the hospitals designated as flagship hospitals in multihospital systems are exceptions to the general rule of lower star ratings for major teaching hospitals. The flagship designation may reflect excellence that is then reflected in the star rating system, or it may reflect some other kind of excellence (eg, reputation for research or teaching, diverse medical services provided) that is not reflected in the star rating system. The primary aim of this study was to compare the average star ratings and hospital characteristics of designated flagship hospitals in multihospital systems with those of (1) major teaching hospitals generally and (2) “nonflagship” hospitals across and within the same systems specifically. We sought to determine whether a flagship designation would be associated with higher star ratings than those of major teaching hospitals in general and with higher star ratings than other, nonflagship hospitals in the same system.

The use of a prestigious flagship hospital name to identify a multihospital system suggests that some aspects of high quality in the flagship are extended in some way to other hospitals in the system. If that is so, then the star ratings of hospitals in organized multihospital systems with a flagship may be more similar to each other than those of sets of hospitals selected at random. As a secondary aim, to determine whether this type of consistent quality throughout a system could be identified in the CMS hospital star rating system, we compared the variation in star ratings between organized multihospital systems with flagship hospitals to those of artificially created “pseudo systems” of unaffiliated hospitals.

We used the Agency for Healthcare Research and Quality (AHRQ) Compendium of U.S. Health Systems, 2016, database and hospital file to identify multihospital health systems and their member hospitals.¹⁰ The database also provides information about health system characteristics such as systemwide teaching intensity and total number of acute care hospitals. We linked the AHRQ files to the CMS Hospital Compare datasets and Hospital Inpatient Prospective Payment System (IPPS) 2018 Final Rule Impact File to obtain star ratings and other information about specific hospitals (eg, resident to bed ratio, uncompensated care payment). Throughout the study, we followed the AHRQ’s definition of “major teaching hospitals” as hospitals with a high resident to bed ratio (≥0.25).

For purposes of this study, the primary criterion for identification of flagship hospitals was an explicit designation by the parent health systems on their websites, in the systems’ official documents, or in press releases or through major media reports. In the few cases in which parent systems did not designate their flagships, we searched reliable online sources such as major newspapers and hospital reviews to see if there was an agreement among sources on the flagship status. If we could not unambiguously identify a flagship hospital in a multihospital system using these methods, the system was not included in the study. A health system could have more than one flagship hospital.

Because the concept of “flagship” often involves a role as a referral center for complex cases in a regional area small enough to have referrals from hospital to hospital within the same system, we excluded multistate national health systems (eg, Catholic Health Initiatives, Community Health Systems, Inc.) and health systems with no major teaching hospitals or no flagship(s) identified by the systems themselves. Non-acute care and stand-alone hospitals, hospitals with missing CMS Certification Numbers (CCNs) or unmatched CCNs or hospital types across different data files, and hospitals without a star rating, were excluded.

Our analyses were performed at both hospital and health system levels. In the hospital-level analysis, we grouped hospitals into “1-2 star,” “3 star,” and “4-5 star” rating categories. We first compared star ratings of flagship hospitals with those of major teaching hospitals in general (ie, hospitals in the CMS Hospital Compare database with resident to bed ratios ≥0.25 that were not designated as system flagship hospitals). We then compared the average flagship hospital and average nonflagship hospital star ratings pooled across all the health systems. To explore hospital-level characteristics that might be associated with flagship hospitals’ performance on star ratings, we compared hospitals’ teaching intensity, bed size, charity care, and disproportionate share hospital (DSH) patient percentage between flagship and major teaching hospitals and between flagship and nonflagship hospitals. Differences were tested using two-sample t test with equal variances. We also compared hospital characteristics among hospitals with 1-2 stars, 3 stars, and 4-5 stars with use of one-way analysis of variance (ANOVA) with Bonferroni adjustment for multiple comparisons.

In the system-level analysis, we examined flagship hospitals’ star ratings relative to the star ratings for other member hospitals in the same system. We assigned health systems to the following three groups according to their flagship hospitals’ star ratings in comparison to other hospitals within their own systems: health systems in which flagship hospitals were rated the lowest among all member hospitals, health systems in which flagship hospitals were rated neither highest nor lowest or all hospitals within the system had the same star rating, and health systems in which flagship hospitals were rated the highest among all member hospitals. We compared system-level characteristics of the three groups. We calculated the average differences in uncompensated care payment, resident to bed ratio, DSH patient percentage, and total beds between flagship hospitals and nonflagship hospitals of the same health systems, and we also compared the differences across the three health system groups defined previously. We conducted an analysis of covariance (ANCOVA) to take system-level factors into consideration, including system size (total number of acute care hospitals in the system), systemwide teaching intensity, and systemwide charity care. The Bonferroni correction was used to adjust for potential problems of multiple comparisons.

Finally, to compare the diversity of star ratings within health systems and the diversity of star ratings nationwide, we generated a set of 100 pseudo systems each comprising six member hospitals (corresponding to the average number of member hospitals per “true” health system included in the study) that were randomly selected from all hospitals excluded from this study. We calculated and compared the average standard deviations of star ratings between the true health systems and this set of pseudo systems. Differences were tested using two-sample t test with equal variances.

Data management and statistical analyses were conducted using Stata SE, version 13.0 (StataCorp LLC, College Station, Texas).

Our final analysis included 599 hospitals in 113 health systems; 119 hospitals were flagships (four health systems each had two flagship hospitals, and one health system had three flagship hospitals). All other hospitals (n = 480) were designated as nonflaghips. On average, each health system had 6 member hospitals with star ratings, with a range from 2 to 22.

Flagship hospitals did have higher average star ratings than major teaching hospitals (mean star rating, 2.8 vs 2.3, respectively; P < .01; Figure). A larger proportion of flagship hospitals received four or five stars than did major teaching hospitals (29% vs 20%, respectively), and a smaller proportion of them received one or two stars (44% vs 59%, respectively; P < .05).

Flagship hospitals had lower star ratings on average, across all systems, than did nonflagship hospitals (mean star rating, 2.8 vs 3.3, respectively; P < .001). A smaller proportion of flagships received four or five stars than did nonflagships (29% vs 44%, respectively), and a larger proportion of them received one or two stars (44% vs 23%, respectively; P < .001).

As expected, flagship hospitals had significantly higher teaching intensity, larger bed size, higher DSH patient percentage, and higher value of uncompensated care payments than did nonflagship hospitals (P < .001 for all). On average, flagship hospitals were significantly larger but had lower DSH patient percentage and lower value of uncompensated care payments than did major teaching hospitals in general (P < .01 for all). In all types of hospitals, four- or five-star hospitals consistently had significantly lower DSH patient percentage (P < .001) and lower value of uncompensated care payment per claim (P < .05) than did other hospitals (Table).

In half of all health systems (n = 56), flagship hospitals were rated the lowest of all hospitals within that system; in approximately 20% of all health systems (n = 22), flagship hospitals were rated the highest. Flagship hospitals were more likely to have the lowest star rating in the system if the within-system difference in DSH patient percentage between flagship and nonflagship hospitals was relatively large. Within-system DSH patient percentage differences between flagship and nonflagship hospitals were 12.4%, 5.4%, and 3.5% in “flagship rated lowest,” “flagship rated middle,” and “flagship rated highest” systems, respectively (P < .05).

Average standardized deviations of star ratings for the 113 true health systems and 100 randomly generated pseudo health systems were 0.86 and 0.97, respectively (P < .05).

System-designated flagship hospitals did not generally have higher star ratings than did the other, smaller, community hospitals, either on average or within their own systems. In fact, the most common pattern observed was the system-designated flagship hospitals had the lowest star rating in their system. Flagship hospitals in multihospital systems were, however, rated higher than major teaching hospitals in general. The safety-net role of many of the system flagship hospitals, as captured by relative DSH percentage, was the most important determinant of low star ratings. A high bed number and teaching status were not as strongly associated with low star ratings.

It is already well established that the CMS star rating system does not correspond to other global hospital ratings systems like those of US News & World Report, Healthgrades, or the Leapfrog Group.¹¹ Each global rating system uses a unique set of measures and weighting systems for those measures, so discrepancies among these systems are inevitable. Multihospital systems may feel that the positive reputation for tertiary care excellence held by a flagship hospital is captured in a rating system like US News that has an explicit reputation component¹² and that the US News rankings are more prominent in the public eye than are those of CMS. To the extent that the CMS star ratings do become more widely used by the public or by payers to establish narrow provider networks, the relatively low ratings of multisystem flagship hospitals may become a cause for concern for those hospitals and systems.

System-designated flagship hospitals are typically large teaching hospitals with higher levels of technology, more highly specialized services and medical staff, more extensive research programs and active clinical trials programs, and the ability to treat cases that are difficult or complex or instances of rare conditions. They are not generally, as it turns out, the hospitals in a given system that the CMS star rating system identifies as “best.” In a number of multihospital systems, the system name is derived from the name of the flagship hospital (eg, Yale New Haven Health System and Montefiore Health System), which suggests that the system finds a marketing or branding advantage in being publicly identified with the name and positive reputation of the flagship hospital. Flagship hospitals may be designated as such because they have other attributes that patients, the community, and the system value, which may not be represented by the CMS quality metrics summarized by star ratings.

We did find a somewhat lower level of variation in star ratings in actual multihospital systems than in a set of randomly created “pseudo systems,” suggesting the presence of some mechanism for quality management in those systems leading to a more similar set of star ratings than one would find in hospitals selected at random.

Our study has a few limitations. First, we excluded multihospital health systems without any major teaching member hospital, which was based on our observation that they do not usually designate their flagship hospitals or they do not have any identifiable flagship hospitals. There may be a small number of such health systems that have designated their flagship hospitals and were excluded from the study, but we do not believe it will change our key findings. Second, it was possible that multiple hospitals in the same health system reported under the same CCN (multicampuses will often use the flagship facility’s IDs for the purposes of claims processing or cost and measure reporting), and therefore, the star ratings for the flagship hospitals reflected the performance of both the flagship hospital and the other member hospitals sharing the same CCN. We cannot fix the underlying reporting issue, and as a result, part of our analysis was probably more of a comparison of the “financial” flagship with other more loosely associated hospitals in the system. We could have in fact overestimated the flagships’ star rating performance by including data of other better performing nonflagship hospitals.

System-designated flagship hospitals tended to have lower CMS Hospital Compare overall hospital quality star ratings than did nonflagship hospitals in the same multihospital systems. The characteristics of hospitals identified as system flagships do not seem well aligned with those associated with better performance in the star rating system.

The authors declared no conflicts of interest.

For severe asthma exacerbations unresponsive to initial treatment, expert consensus guidelines from 2007 recommend consideration for adjunct treatments (magnesium or heliox) to decrease the likelihood of intubation.¹ Over the last decade, data have emerged suggesting that intravenous (IV) magnesium may be more effective for reduction of hospital admission rates.² Pooled meta-analyses have demonstrated improved pulmonary function and reduction of hospital admission by as much as 68% in children when IV magnesium is administered in the emergency department (ED), although the evidence is extremely limited because of a small number of studies (three) and small sample size (115 children).^2-5

Though these data suggest that use of IV magnesium may reduce admission rates, a study of pediatric emergency medicine (PEM) physicians in US and Canada reported reluctance regarding use for this purpose. While PEM physicians reported awareness of the literature on admission prevention, they estimated that fewer than 5% of their patients receiving IV magnesium were discharged home.⁶ Their practice was generally limited to using IV magnesium in children with impending respiratory failure for the purpose of reducing intensive care unit (ICU) admission and not hospitalization.⁶ PEM physicians’ reluctance to use IV magnesium was related to the lack of strong available evidence supporting the impact of IV magnesium on outcomes, such as admission, and gaps in the literature about its dosing and safety profile.

The goal of this study was to assess the prevailing trends in IV magnesium use across US children’s hospitals and to assess the relationship of IV magnesium use to admission rate, length of stay (LOS), readmission rate, and ICU admission rate. We hypothesized that IV magnesium use might have increased following publication of studies demonstrating an association between IV magnesium use and fewer admissions.

Study Design, Setting, and Participants

This is a retrospective cohort study of asthma (All Patient Refined Diagnosis Related Group 141) hospitalizations for patients less than 18 years old presenting to 35 tertiary care children’s hospitals from January 1, 2010, to December 31, 2017, included in the Pediatric Health Information System (PHIS; Children’s Hospital Association, Lenexa, Kansas) database. The PHIS database is an administrative database that contains demographics, International Classification of Diseases 9th and 10th Revision diagnoses and procedures, and daily billing records for all inpatient, observation, ED, and ambulatory surgery encounters. All data were deidentified prior to inclusion in the database and tracking of patients across ED and inpatient visits was achieved through an encrypted and unique patient identifier. Children transferred from other hospitals were excluded because we could not verify IV magnesium use prior to transfer. For hospitals to be included, they were required to provide continuous data throughout the study period.

Main Outcome Measure

The main outcome was exposure to IV magnesium as determined by billing information available in the PHIS database.

Patient Demographics

We assessed patients’ demographic characteristics, including age (younger than 5 years, 5-11 years, and 12-17 years), sex, race/ethnicity, and insurance status.

Healthcare Utilization and Hospital Characteristics

We assessed healthcare utilization using geometric mean LOS, proportion of patients admitted to the hospital and to the ICU, and the proportion of patients with a 7-day all-cause readmission. In addition, we divided hospitals into three equal groups based on their annual inpatient asthma volume (<300, 300-850, >850 cases per year).

Statistical Analysis

We compared demographic and clinical characteristics across patients receiving IV magnesium with those who did not receive it with use of chi-square tests for categorical variables and Wilcoxon rank sum test for continuous variables. We calculated annual IV magnesium use rates for each hospital and modeled the average annual rate with a general linear model in order to assess change over time. We used Pearson product moment correlation to compare the annual proportion of magnesium use and healthcare utilization measures, including geometric mean LOS, the proportion of patients using the inpatient wards or the ICU, and the proportion of cases with a 7-day all-cause readmission. Geometric mean LOS was used to normalize the compounding effect of non–normally distributed arithmetic mean LOS. A sensitivity analysis was performed stratifying IV magnesium use over time by hospital inpatient volume. Data were analyzed using SAS version 9.4 (SAS Institute, Cary, North Carolina), and P values < .05 were considered statistically significant.

Study Population

A total of 878,188 encounters with acute asthma exacerbation met the inclusion criteria, with 65,558 (7.5%) receiving IV magnesium (Table). Of those receiving IV magnesium, 90% were admitted to the hospital. There were statistically significant differences in IV magnesium use when compared by age, race/ethnicity, and payer type, but not gender. IV magnesium use was significantly associated with older age (more than 5 years old), non-Hispanic black race, ED visit in the year prior to admission, longer hospital LOS, and higher ICU admission rate.

Trends in Intravenous Magnesium Use

IV magnesium use among hospitalized children more than doubled from 2010 to 2017 (17% vs 36%). Low-volume hospitals had a lower frequency of IV magnesium use, compared with the moderate- and high-volume hospitals. The growth rate per year of IV magnesium use was greater in high- and moderate-volume hospitals (3.4% and 2.9% per year, respectively), compared with the low-volume hospitals (1.2% per year; P = .04).

Trends in Intravenous Magnesium Use and Hospital Outcomes

The trend in IV magnesium use was not associated with a statistically significant change in the inpatient and ICU admission rate or in the 7-day all-cause readmission rate (Figure and Appendix Figure). Although IV magnesium use increased over time, LOS decreased significantly during the same period (1.6 days in 2010 vs 1.4 days in 2017; P < .001). When analyzed by hospital volume, no significant associations were found in the inpatient admission, ICU admission, and 7-day readmission rate.

The use of IV magnesium has significantly increased in US children’s hospitals over the last 8 years, especially among those hospitalized following an ED evaluation. Over this interval, trends in inpatient and ICU admission rate, as well as 7-day all-cause readmission rate, for asthma did not change, while LOS decreased. These findings contrast with a recent Cochrane review that summarized the efficacy of IV magnesium for reducing admission rate in few small trials.²

Our study findings are more consistent with prior survey findings that IV magnesium does not reduce hospitalization and that ED physicians tend to use IV magnesium in severe asthma exacerbation for its potential therapeutic benefits because of bronchodilator and anti-inflammatory effect.^6,7 Similar to PEM physicians’ estimates, only 10% of patients receiving IV magnesium were discharged home in our study.

IV magnesium use is higher in high-volume hospitals than in moderate- and low-volume ones. One potential explanation is that high- and moderate-volume hospitals may see a higher volume of children with severe or impending respiratory failure and, therefore, are more likely to use IV magnesium than the low-volume hospitals are. Alternatively, physician adoption of magnesium use for lower-acuity asthma exacerbations could vary by hospital volume.

Trend analyses of outcomes suggest that increase in IV magnesium use was not associated with an increase in inpatient and ICU admission rate or with 7-day all-cause readmission rate, although LOS reduced. LOS might be reduced because of various quality improvement initiatives (eg, discharging patients after every 3 hours albuterol treatments or respiratory therapist–driven protocols) and might not be related to IV magnesium use.^8,9 To this point, a recent study of a respiratory assessment score–matched cohort did not find any therapeutic benefit of IV magnesium with severe asthma exacerbation when receiving continuous albuterol therapy on a pediatric ward.⁵ Perhaps future studies could explore estimating the outcome by performing comparative effectiveness studies between those with severe asthma exacerbation who did or did not receive IV magnesium. Additionally, randomized controlled trials comparing IV magnesium and standard therapy and its effects on outcomes, such as hospitalization, LOS, association with asthma chronicity, and previous oral steroid use, might provide further insight to inform clinical practice.

Certain study limitations should be noted. The study cohort included children’s hospitals only, and it is possible that care at nonchildren’s hospitals for asthma differs. PHIS dataset used in this study does not allow determination of where and when IV magnesium was given, the severity of asthma exacerbation, or the chronicity of baseline disease. Moreover, PHIS hospitals include centers in large cities, and other competing children’s hospitals may provide other tertiary care that could affect the readmission data calculation. Lastly, the temporal associations between IV magnesium use and outcomes reported in this study should not be used as evidence or lack of evidence for the effectiveness of magnesium given the limitations of the observational study design and dataset used.

In conclusion, IV magnesium use in management of asthma exacerbation in children across the United States has significantly increased. The increase occurred disproportionately in high-volume hospitals and was not associated with changes in admission rate, ICU admission rate, or 7-day all-cause readmission rate, although LOS has decreased over time.

The authors have no financial relationships relevant to this article or conflicts of interest to disclose. 

This paper was a platform presentation at annual meetings of Pediatric Academic Societies 2019; accepted for presentation at annual meeting of Pediatric Hospital Medicine, July 2019.

No funding was secured for this study. 

For severe asthma exacerbations unresponsive to initial treatment, expert consensus guidelines from 2007 recommend consideration for adjunct treatments (magnesium or heliox) to decrease the likelihood of intubation.¹ Over the last decade, data have emerged suggesting that intravenous (IV) magnesium may be more effective for reduction of hospital admission rates.² Pooled meta-analyses have demonstrated improved pulmonary function and reduction of hospital admission by as much as 68% in children when IV magnesium is administered in the emergency department (ED), although the evidence is extremely limited because of a small number of studies (three) and small sample size (115 children).^2-5

Though these data suggest that use of IV magnesium may reduce admission rates, a study of pediatric emergency medicine (PEM) physicians in US and Canada reported reluctance regarding use for this purpose. While PEM physicians reported awareness of the literature on admission prevention, they estimated that fewer than 5% of their patients receiving IV magnesium were discharged home.⁶ Their practice was generally limited to using IV magnesium in children with impending respiratory failure for the purpose of reducing intensive care unit (ICU) admission and not hospitalization.⁶ PEM physicians’ reluctance to use IV magnesium was related to the lack of strong available evidence supporting the impact of IV magnesium on outcomes, such as admission, and gaps in the literature about its dosing and safety profile.

The goal of this study was to assess the prevailing trends in IV magnesium use across US children’s hospitals and to assess the relationship of IV magnesium use to admission rate, length of stay (LOS), readmission rate, and ICU admission rate. We hypothesized that IV magnesium use might have increased following publication of studies demonstrating an association between IV magnesium use and fewer admissions.

Study Design, Setting, and Participants

This is a retrospective cohort study of asthma (All Patient Refined Diagnosis Related Group 141) hospitalizations for patients less than 18 years old presenting to 35 tertiary care children’s hospitals from January 1, 2010, to December 31, 2017, included in the Pediatric Health Information System (PHIS; Children’s Hospital Association, Lenexa, Kansas) database. The PHIS database is an administrative database that contains demographics, International Classification of Diseases 9th and 10th Revision diagnoses and procedures, and daily billing records for all inpatient, observation, ED, and ambulatory surgery encounters. All data were deidentified prior to inclusion in the database and tracking of patients across ED and inpatient visits was achieved through an encrypted and unique patient identifier. Children transferred from other hospitals were excluded because we could not verify IV magnesium use prior to transfer. For hospitals to be included, they were required to provide continuous data throughout the study period.

Main Outcome Measure

The main outcome was exposure to IV magnesium as determined by billing information available in the PHIS database.

Patient Demographics

We assessed patients’ demographic characteristics, including age (younger than 5 years, 5-11 years, and 12-17 years), sex, race/ethnicity, and insurance status.

Healthcare Utilization and Hospital Characteristics

We assessed healthcare utilization using geometric mean LOS, proportion of patients admitted to the hospital and to the ICU, and the proportion of patients with a 7-day all-cause readmission. In addition, we divided hospitals into three equal groups based on their annual inpatient asthma volume (<300, 300-850, >850 cases per year).

Statistical Analysis

We compared demographic and clinical characteristics across patients receiving IV magnesium with those who did not receive it with use of chi-square tests for categorical variables and Wilcoxon rank sum test for continuous variables. We calculated annual IV magnesium use rates for each hospital and modeled the average annual rate with a general linear model in order to assess change over time. We used Pearson product moment correlation to compare the annual proportion of magnesium use and healthcare utilization measures, including geometric mean LOS, the proportion of patients using the inpatient wards or the ICU, and the proportion of cases with a 7-day all-cause readmission. Geometric mean LOS was used to normalize the compounding effect of non–normally distributed arithmetic mean LOS. A sensitivity analysis was performed stratifying IV magnesium use over time by hospital inpatient volume. Data were analyzed using SAS version 9.4 (SAS Institute, Cary, North Carolina), and P values < .05 were considered statistically significant.

Study Population

A total of 878,188 encounters with acute asthma exacerbation met the inclusion criteria, with 65,558 (7.5%) receiving IV magnesium (Table). Of those receiving IV magnesium, 90% were admitted to the hospital. There were statistically significant differences in IV magnesium use when compared by age, race/ethnicity, and payer type, but not gender. IV magnesium use was significantly associated with older age (more than 5 years old), non-Hispanic black race, ED visit in the year prior to admission, longer hospital LOS, and higher ICU admission rate.

Trends in Intravenous Magnesium Use

IV magnesium use among hospitalized children more than doubled from 2010 to 2017 (17% vs 36%). Low-volume hospitals had a lower frequency of IV magnesium use, compared with the moderate- and high-volume hospitals. The growth rate per year of IV magnesium use was greater in high- and moderate-volume hospitals (3.4% and 2.9% per year, respectively), compared with the low-volume hospitals (1.2% per year; P = .04).

Trends in Intravenous Magnesium Use and Hospital Outcomes

The trend in IV magnesium use was not associated with a statistically significant change in the inpatient and ICU admission rate or in the 7-day all-cause readmission rate (Figure and Appendix Figure). Although IV magnesium use increased over time, LOS decreased significantly during the same period (1.6 days in 2010 vs 1.4 days in 2017; P < .001). When analyzed by hospital volume, no significant associations were found in the inpatient admission, ICU admission, and 7-day readmission rate.

The use of IV magnesium has significantly increased in US children’s hospitals over the last 8 years, especially among those hospitalized following an ED evaluation. Over this interval, trends in inpatient and ICU admission rate, as well as 7-day all-cause readmission rate, for asthma did not change, while LOS decreased. These findings contrast with a recent Cochrane review that summarized the efficacy of IV magnesium for reducing admission rate in few small trials.²

Our study findings are more consistent with prior survey findings that IV magnesium does not reduce hospitalization and that ED physicians tend to use IV magnesium in severe asthma exacerbation for its potential therapeutic benefits because of bronchodilator and anti-inflammatory effect.^6,7 Similar to PEM physicians’ estimates, only 10% of patients receiving IV magnesium were discharged home in our study.

IV magnesium use is higher in high-volume hospitals than in moderate- and low-volume ones. One potential explanation is that high- and moderate-volume hospitals may see a higher volume of children with severe or impending respiratory failure and, therefore, are more likely to use IV magnesium than the low-volume hospitals are. Alternatively, physician adoption of magnesium use for lower-acuity asthma exacerbations could vary by hospital volume.

Trend analyses of outcomes suggest that increase in IV magnesium use was not associated with an increase in inpatient and ICU admission rate or with 7-day all-cause readmission rate, although LOS reduced. LOS might be reduced because of various quality improvement initiatives (eg, discharging patients after every 3 hours albuterol treatments or respiratory therapist–driven protocols) and might not be related to IV magnesium use.^8,9 To this point, a recent study of a respiratory assessment score–matched cohort did not find any therapeutic benefit of IV magnesium with severe asthma exacerbation when receiving continuous albuterol therapy on a pediatric ward.⁵ Perhaps future studies could explore estimating the outcome by performing comparative effectiveness studies between those with severe asthma exacerbation who did or did not receive IV magnesium. Additionally, randomized controlled trials comparing IV magnesium and standard therapy and its effects on outcomes, such as hospitalization, LOS, association with asthma chronicity, and previous oral steroid use, might provide further insight to inform clinical practice.

Certain study limitations should be noted. The study cohort included children’s hospitals only, and it is possible that care at nonchildren’s hospitals for asthma differs. PHIS dataset used in this study does not allow determination of where and when IV magnesium was given, the severity of asthma exacerbation, or the chronicity of baseline disease. Moreover, PHIS hospitals include centers in large cities, and other competing children’s hospitals may provide other tertiary care that could affect the readmission data calculation. Lastly, the temporal associations between IV magnesium use and outcomes reported in this study should not be used as evidence or lack of evidence for the effectiveness of magnesium given the limitations of the observational study design and dataset used.

In conclusion, IV magnesium use in management of asthma exacerbation in children across the United States has significantly increased. The increase occurred disproportionately in high-volume hospitals and was not associated with changes in admission rate, ICU admission rate, or 7-day all-cause readmission rate, although LOS has decreased over time.

The authors have no financial relationships relevant to this article or conflicts of interest to disclose. 

This paper was a platform presentation at annual meetings of Pediatric Academic Societies 2019; accepted for presentation at annual meeting of Pediatric Hospital Medicine, July 2019.

No funding was secured for this study. 

For severe asthma exacerbations unresponsive to initial treatment, expert consensus guidelines from 2007 recommend consideration for adjunct treatments (magnesium or heliox) to decrease the likelihood of intubation.¹ Over the last decade, data have emerged suggesting that intravenous (IV) magnesium may be more effective for reduction of hospital admission rates.² Pooled meta-analyses have demonstrated improved pulmonary function and reduction of hospital admission by as much as 68% in children when IV magnesium is administered in the emergency department (ED), although the evidence is extremely limited because of a small number of studies (three) and small sample size (115 children).^2-5

Though these data suggest that use of IV magnesium may reduce admission rates, a study of pediatric emergency medicine (PEM) physicians in US and Canada reported reluctance regarding use for this purpose. While PEM physicians reported awareness of the literature on admission prevention, they estimated that fewer than 5% of their patients receiving IV magnesium were discharged home.⁶ Their practice was generally limited to using IV magnesium in children with impending respiratory failure for the purpose of reducing intensive care unit (ICU) admission and not hospitalization.⁶ PEM physicians’ reluctance to use IV magnesium was related to the lack of strong available evidence supporting the impact of IV magnesium on outcomes, such as admission, and gaps in the literature about its dosing and safety profile.

The goal of this study was to assess the prevailing trends in IV magnesium use across US children’s hospitals and to assess the relationship of IV magnesium use to admission rate, length of stay (LOS), readmission rate, and ICU admission rate. We hypothesized that IV magnesium use might have increased following publication of studies demonstrating an association between IV magnesium use and fewer admissions.

Study Design, Setting, and Participants

This is a retrospective cohort study of asthma (All Patient Refined Diagnosis Related Group 141) hospitalizations for patients less than 18 years old presenting to 35 tertiary care children’s hospitals from January 1, 2010, to December 31, 2017, included in the Pediatric Health Information System (PHIS; Children’s Hospital Association, Lenexa, Kansas) database. The PHIS database is an administrative database that contains demographics, International Classification of Diseases 9th and 10th Revision diagnoses and procedures, and daily billing records for all inpatient, observation, ED, and ambulatory surgery encounters. All data were deidentified prior to inclusion in the database and tracking of patients across ED and inpatient visits was achieved through an encrypted and unique patient identifier. Children transferred from other hospitals were excluded because we could not verify IV magnesium use prior to transfer. For hospitals to be included, they were required to provide continuous data throughout the study period.

Main Outcome Measure

The main outcome was exposure to IV magnesium as determined by billing information available in the PHIS database.

Patient Demographics

We assessed patients’ demographic characteristics, including age (younger than 5 years, 5-11 years, and 12-17 years), sex, race/ethnicity, and insurance status.

Healthcare Utilization and Hospital Characteristics

We assessed healthcare utilization using geometric mean LOS, proportion of patients admitted to the hospital and to the ICU, and the proportion of patients with a 7-day all-cause readmission. In addition, we divided hospitals into three equal groups based on their annual inpatient asthma volume (<300, 300-850, >850 cases per year).

Statistical Analysis

We compared demographic and clinical characteristics across patients receiving IV magnesium with those who did not receive it with use of chi-square tests for categorical variables and Wilcoxon rank sum test for continuous variables. We calculated annual IV magnesium use rates for each hospital and modeled the average annual rate with a general linear model in order to assess change over time. We used Pearson product moment correlation to compare the annual proportion of magnesium use and healthcare utilization measures, including geometric mean LOS, the proportion of patients using the inpatient wards or the ICU, and the proportion of cases with a 7-day all-cause readmission. Geometric mean LOS was used to normalize the compounding effect of non–normally distributed arithmetic mean LOS. A sensitivity analysis was performed stratifying IV magnesium use over time by hospital inpatient volume. Data were analyzed using SAS version 9.4 (SAS Institute, Cary, North Carolina), and P values < .05 were considered statistically significant.

Study Population

A total of 878,188 encounters with acute asthma exacerbation met the inclusion criteria, with 65,558 (7.5%) receiving IV magnesium (Table). Of those receiving IV magnesium, 90% were admitted to the hospital. There were statistically significant differences in IV magnesium use when compared by age, race/ethnicity, and payer type, but not gender. IV magnesium use was significantly associated with older age (more than 5 years old), non-Hispanic black race, ED visit in the year prior to admission, longer hospital LOS, and higher ICU admission rate.

Trends in Intravenous Magnesium Use

IV magnesium use among hospitalized children more than doubled from 2010 to 2017 (17% vs 36%). Low-volume hospitals had a lower frequency of IV magnesium use, compared with the moderate- and high-volume hospitals. The growth rate per year of IV magnesium use was greater in high- and moderate-volume hospitals (3.4% and 2.9% per year, respectively), compared with the low-volume hospitals (1.2% per year; P = .04).

Trends in Intravenous Magnesium Use and Hospital Outcomes

The trend in IV magnesium use was not associated with a statistically significant change in the inpatient and ICU admission rate or in the 7-day all-cause readmission rate (Figure and Appendix Figure). Although IV magnesium use increased over time, LOS decreased significantly during the same period (1.6 days in 2010 vs 1.4 days in 2017; P < .001). When analyzed by hospital volume, no significant associations were found in the inpatient admission, ICU admission, and 7-day readmission rate.

The use of IV magnesium has significantly increased in US children’s hospitals over the last 8 years, especially among those hospitalized following an ED evaluation. Over this interval, trends in inpatient and ICU admission rate, as well as 7-day all-cause readmission rate, for asthma did not change, while LOS decreased. These findings contrast with a recent Cochrane review that summarized the efficacy of IV magnesium for reducing admission rate in few small trials.²

Our study findings are more consistent with prior survey findings that IV magnesium does not reduce hospitalization and that ED physicians tend to use IV magnesium in severe asthma exacerbation for its potential therapeutic benefits because of bronchodilator and anti-inflammatory effect.^6,7 Similar to PEM physicians’ estimates, only 10% of patients receiving IV magnesium were discharged home in our study.

IV magnesium use is higher in high-volume hospitals than in moderate- and low-volume ones. One potential explanation is that high- and moderate-volume hospitals may see a higher volume of children with severe or impending respiratory failure and, therefore, are more likely to use IV magnesium than the low-volume hospitals are. Alternatively, physician adoption of magnesium use for lower-acuity asthma exacerbations could vary by hospital volume.

Trend analyses of outcomes suggest that increase in IV magnesium use was not associated with an increase in inpatient and ICU admission rate or with 7-day all-cause readmission rate, although LOS reduced. LOS might be reduced because of various quality improvement initiatives (eg, discharging patients after every 3 hours albuterol treatments or respiratory therapist–driven protocols) and might not be related to IV magnesium use.^8,9 To this point, a recent study of a respiratory assessment score–matched cohort did not find any therapeutic benefit of IV magnesium with severe asthma exacerbation when receiving continuous albuterol therapy on a pediatric ward.⁵ Perhaps future studies could explore estimating the outcome by performing comparative effectiveness studies between those with severe asthma exacerbation who did or did not receive IV magnesium. Additionally, randomized controlled trials comparing IV magnesium and standard therapy and its effects on outcomes, such as hospitalization, LOS, association with asthma chronicity, and previous oral steroid use, might provide further insight to inform clinical practice.

Certain study limitations should be noted. The study cohort included children’s hospitals only, and it is possible that care at nonchildren’s hospitals for asthma differs. PHIS dataset used in this study does not allow determination of where and when IV magnesium was given, the severity of asthma exacerbation, or the chronicity of baseline disease. Moreover, PHIS hospitals include centers in large cities, and other competing children’s hospitals may provide other tertiary care that could affect the readmission data calculation. Lastly, the temporal associations between IV magnesium use and outcomes reported in this study should not be used as evidence or lack of evidence for the effectiveness of magnesium given the limitations of the observational study design and dataset used.

In conclusion, IV magnesium use in management of asthma exacerbation in children across the United States has significantly increased. The increase occurred disproportionately in high-volume hospitals and was not associated with changes in admission rate, ICU admission rate, or 7-day all-cause readmission rate, although LOS has decreased over time.

The authors have no financial relationships relevant to this article or conflicts of interest to disclose. 

This paper was a platform presentation at annual meetings of Pediatric Academic Societies 2019; accepted for presentation at annual meeting of Pediatric Hospital Medicine, July 2019.

No funding was secured for this study. 

Healthcare delivery in rural America faces unique, growing challenges related to health and emergency care access.¹ Telemedicine approaches have the potential to increase rural hospitals’ ability to deliver efficient emergency care and reduce clinician shortages.² While initial evidence of telemedicine success exists, more quality research is needed to understand telemedicine patient and care team experiences,³ especially with real-time, clinician-initiated video conferencing in critical access hospital (CAH) emergency departments (ED). Some experience studies exist,⁴ but results are primarily quantitative⁵ and lack the nuanced qualitative depth needed to understand topics such as satisfaction and communication.⁶ Additionally, few explore combined patient and care team perspectives.⁵ The lack of breadth and depth makes it difficult to provide actionable recommendations for improvements and affects the feasibility of continuing this work and improving telemedicine care quality. To address these gaps, we evaluated a real-time, clinician-­initiated video conferencing program with overnight clinicians servicing ED patients in three Midwestern care system CAHs. This evaluation assessed patient and care team (nurse and clinician) experience with telemedicine using quantitative and qualitative survey data analysis.

Because this evaluation was designed to measure and improve program quality in a single healthcare system, it was deemed non-human subjects research by the organization’s institutional review board. This brief report follows telemedicine reporting guidelines.⁷

Setting and Telemedicine Program

This program, designed to reduce the need for on-call hospitalist clinicians to be onsite at CAHs overnight, was implemented in a large Midwestern nonprofit integrated healthcare system with three rural CAHs (combined capacity for 75 inpatient admissions, with full-time onsite ED clinicians and nurses, as well as on-call hospitalist clinicians) and a large metropolitan tertiary-care hospital. All adult patients presenting to CAH EDs between 6 pm and 8 am were evaluated, as usual, by an onsite ED clinician. If the admitting ED clinician and charge nurse determined that admission was appropriate, patients were signed out to remote hospitalist clinicians and roomed by onsite nurses. Nurses facilitated live audio-video telemedicine “history and physical” visits with remote clinicians via telemedicine carts (AmericanWell C750, Boston, Massachusetts, and ThinkLabs One Electronic Stethoscope, Centennial, Colorado). Already-­hospitalized patients, as well as patients admitted to a remote clinician, were cared for by the remote clinician and onsite nurse for the remainder of the night, which eliminated the need for local on-call clinicians. The onsite ED clinician responded to emergencies of already-hospitalized patients, but often consulted with remote clinicians to assist virtually with necessary orders and documentation. Remote clinicians were located at the metropolitan tertiary care hospital or home work stations.

Following a pilot period, the full-scale program was implemented in September 2017 and included 14 remote clinicians and 60 onsite nurses.

Survey Administration and Design

A postimplementation survey was designed to explore patient and care team experience with telemedicine. Patients who received a telemedicine visit between September 2017 and April 2018 were mailed a paper survey. Nonresponders were called by professional interviewers affiliated with the healthcare system. All participating clinicians (N = 14, all MDs) and nurses (N = 60, all RNs) were emailed an online care team survey with phone-in option. Care team nonresponders were sent up to two reminder emails.

Surveys captured the following five constructs: communication, workflow integration, telemedicine technology, quality of care, and general satisfaction. Existing questionnaires were used where possible; additional items were designed with clinical experts following survey design best practices.⁸ Patient-perceived communication was assessed via three Consumer Assessment of Healthcare Providers and Systems Outpatient and Ambulatory Surgery Survey items.⁹ Five additional program-developed patient survey items included satisfaction with clinician-nurse communication, satisfaction with technology, telemedicine quality of care overall and in comparison with traditional care, and whether or not patients would recommend telemedicine (Table). Four open-ended questions asked patients about improvement opportunities and general satisfaction.

Care team surveys included two items regarding ability to effectively communicate, two about satisfaction with workflow integration, one about technical problems, two about quality of care, and one about general satisfaction. Open-ended questions gathered further information and recommendations to improve communication, workflow integration, technology issues, and general satisfaction.

Analysis

Closed-ended items were dichotomized (satisfied yes/no); descriptive statistics (frequencies/percents) are presented to quantify patient and care team experience. Quantitative analyses were conducted in SAS software version 9.4 (SAS Institute, Cary, North Carolina). Open-ended responses were coded separately for patient and care team experience, following qualitative content analysis best practices.¹⁰ A lead coder read all responses, created a coding framework of identified themes, and coded individual responses. A second coder independently coded responses using the same framework. Interrater reliability was calculated for each major theme using percent agreement and prevalence- and bias-adjusted k (PABAK) statistic. A single representative quote was selected and lightly edited for each subtheme to deepen understanding and provide respondent voice.¹¹

Of eligible patients mailed a survey (N = 408), 3% self-reported as ineligible, and 54% completed the survey. This is a maximum response rate (response rate 6) according to the American Association for Public Opinion Research.¹² Patients were 67 years old on average (SD = 15), they were primarily white (97%), and 54% were female. All clinicians and 63% of nurses completed the survey.¹² Clinicians and nurses were 29% and 95% female, respectively.

Quantitative results (Table) show generally positive experience across patient and care team respondents. Over 90% were satisfied with all measures of communication. Care teams had high satisfaction with admissions processes and reported telemedicine improved cross-coverage. Patient-reported technology experience was positive but was less positive from the care team perspective. Care teams reported lower absolute quality of care than did patients but were more likely to perceive telemedicine as high quality, compared with traditional care. Most patients, clinicians, and nurses would recommend telemedicine.

Qualitatively, four major themes were identified in open-­ended responses with high interrater reliability (PABAK ranging from 0.92 to 0.98 in patient responses and 0.88 to 0.95 in care team responses) and aligned with the quantitative survey constructs: clinician-nurse communication, clinician-patient communication, workflow integration, and telemedicine technology. Patients reported satisfaction with communication with remote clinicians:

“[The clinician] was extremely attentive to me and what was going on. She was articulate and clear. I understood what was going to happen.” –Patient

Care teams suggested concrete improvement opportunities:

“I’d prefer to have some time with nursing staff both before and (sometimes) after the patient encounter.” –Clinician

“Since we cannot hear what [the clinicians] are hearing with the stethoscope, it’s nice when they tell us when to move it to the next spot.” –Nurse

Clinicians and nurses gave favorable responses regarding workflow integration, though time (both admissions wait time and session duration) was a reported opportunity:

“It would be helpful if we could speed up the time from admit request to screen time.” –Clinician

“When the [clinicians] get swamped, they’re hard to get a hold of, and admissions can take a long time. They may have too much on their plates dealing with several locations.” –Nurse

Technology issues—internet connection, stethoscope, sound, and screen or camera—were mentioned by patients and care teams, though technology was reviewed favorably overall by most patients:

“I was fascinated by the technology. Visiting someone over a television was impressive. ... The picture, the sound clarity, and the connection itself was flawless.” –Patient

Some patients commented that telemedicine was the best option given the situation, but still preferred an in-person doctor:

“If a doctor wasn’t available, telemedicine is better than nothing.” –Patient

Nurses who would not recommend telemedicine noted the need for personal connection:

“[I] still prefer [an] in-person MD for more personal contact. The older patients often state they wish the doctor would come and see them.” –Nurse

Patients who would not recommend telemedicine also desired personal connection:

“I would sooner talk to a person than a machine.” –Patient

A few clinicians noted the connection with patients would be improved if they knew about others in the room:

“It’d be nice if everyone in the room was introduced. Sometimes people are sitting out of view of the camera and I don’t realize they’re there until later.” –Clinician

These results make important contributions to understanding and improving the telemedicine experience in rural emergency hospital medicine. While the predominantly white patient respondent population limits generalizability, these demographics are representative of the overall population of the participating hospitals. A strength of this evaluation is its contemporaneous consideration of patient and care team experience with both quantitative and rich, qualitative analysis. Patients and care teams alike thought overnight telemedicine was better than the status quo. While our quality of care findings align with some previous literature,¹³ care teams in the current analysis overwhelmingly would recommend telemedicine, whereas some clinicians in prior work would not recommend telemedicine.¹⁴

In terms of communication, in line with existing literature, some patients still preferred in-person visits,¹⁵ a view also shared by some care team members. Workflow and technology barriers were raised, corroborating existing work,¹³ but actionable solutions (eg, adding care team–only time before visits or verbalizing when to move stethoscopes) were also identified.

Embedding patient and care team experience surveys and sharing results is critical in advancing telemedicine. Findings from this evaluation strengthen the case for payer reimbursement of telemedicine in rural acute care. Continued work to improve, test, and publish findings on patient and care team experience with telemedicine is critical to providing quality services in often-underserved communities.

The authors would like to acknowledge the contributions of Ann Werner in identifying the patient survey sample, Brian Barklind in identifying source data for the analysis, and both Brian Barklind and Rachael Rivard for conducting the analyses and summarizing results. We would also like to thank Kelly Logue for her involvement in conceptualizing the telemedicine evaluation described here, as well as Larisa Polynskaya for her help preparing the manuscript for publication, and the care teams and patients who provided valuable input.

Healthcare delivery in rural America faces unique, growing challenges related to health and emergency care access.¹ Telemedicine approaches have the potential to increase rural hospitals’ ability to deliver efficient emergency care and reduce clinician shortages.² While initial evidence of telemedicine success exists, more quality research is needed to understand telemedicine patient and care team experiences,³ especially with real-time, clinician-initiated video conferencing in critical access hospital (CAH) emergency departments (ED). Some experience studies exist,⁴ but results are primarily quantitative⁵ and lack the nuanced qualitative depth needed to understand topics such as satisfaction and communication.⁶ Additionally, few explore combined patient and care team perspectives.⁵ The lack of breadth and depth makes it difficult to provide actionable recommendations for improvements and affects the feasibility of continuing this work and improving telemedicine care quality. To address these gaps, we evaluated a real-time, clinician-­initiated video conferencing program with overnight clinicians servicing ED patients in three Midwestern care system CAHs. This evaluation assessed patient and care team (nurse and clinician) experience with telemedicine using quantitative and qualitative survey data analysis.

Because this evaluation was designed to measure and improve program quality in a single healthcare system, it was deemed non-human subjects research by the organization’s institutional review board. This brief report follows telemedicine reporting guidelines.⁷

Setting and Telemedicine Program

This program, designed to reduce the need for on-call hospitalist clinicians to be onsite at CAHs overnight, was implemented in a large Midwestern nonprofit integrated healthcare system with three rural CAHs (combined capacity for 75 inpatient admissions, with full-time onsite ED clinicians and nurses, as well as on-call hospitalist clinicians) and a large metropolitan tertiary-care hospital. All adult patients presenting to CAH EDs between 6 pm and 8 am were evaluated, as usual, by an onsite ED clinician. If the admitting ED clinician and charge nurse determined that admission was appropriate, patients were signed out to remote hospitalist clinicians and roomed by onsite nurses. Nurses facilitated live audio-video telemedicine “history and physical” visits with remote clinicians via telemedicine carts (AmericanWell C750, Boston, Massachusetts, and ThinkLabs One Electronic Stethoscope, Centennial, Colorado). Already-­hospitalized patients, as well as patients admitted to a remote clinician, were cared for by the remote clinician and onsite nurse for the remainder of the night, which eliminated the need for local on-call clinicians. The onsite ED clinician responded to emergencies of already-hospitalized patients, but often consulted with remote clinicians to assist virtually with necessary orders and documentation. Remote clinicians were located at the metropolitan tertiary care hospital or home work stations.

Following a pilot period, the full-scale program was implemented in September 2017 and included 14 remote clinicians and 60 onsite nurses.

Survey Administration and Design

A postimplementation survey was designed to explore patient and care team experience with telemedicine. Patients who received a telemedicine visit between September 2017 and April 2018 were mailed a paper survey. Nonresponders were called by professional interviewers affiliated with the healthcare system. All participating clinicians (N = 14, all MDs) and nurses (N = 60, all RNs) were emailed an online care team survey with phone-in option. Care team nonresponders were sent up to two reminder emails.

Surveys captured the following five constructs: communication, workflow integration, telemedicine technology, quality of care, and general satisfaction. Existing questionnaires were used where possible; additional items were designed with clinical experts following survey design best practices.⁸ Patient-perceived communication was assessed via three Consumer Assessment of Healthcare Providers and Systems Outpatient and Ambulatory Surgery Survey items.⁹ Five additional program-developed patient survey items included satisfaction with clinician-nurse communication, satisfaction with technology, telemedicine quality of care overall and in comparison with traditional care, and whether or not patients would recommend telemedicine (Table). Four open-ended questions asked patients about improvement opportunities and general satisfaction.

Care team surveys included two items regarding ability to effectively communicate, two about satisfaction with workflow integration, one about technical problems, two about quality of care, and one about general satisfaction. Open-ended questions gathered further information and recommendations to improve communication, workflow integration, technology issues, and general satisfaction.

Analysis

Closed-ended items were dichotomized (satisfied yes/no); descriptive statistics (frequencies/percents) are presented to quantify patient and care team experience. Quantitative analyses were conducted in SAS software version 9.4 (SAS Institute, Cary, North Carolina). Open-ended responses were coded separately for patient and care team experience, following qualitative content analysis best practices.¹⁰ A lead coder read all responses, created a coding framework of identified themes, and coded individual responses. A second coder independently coded responses using the same framework. Interrater reliability was calculated for each major theme using percent agreement and prevalence- and bias-adjusted k (PABAK) statistic. A single representative quote was selected and lightly edited for each subtheme to deepen understanding and provide respondent voice.¹¹

Of eligible patients mailed a survey (N = 408), 3% self-reported as ineligible, and 54% completed the survey. This is a maximum response rate (response rate 6) according to the American Association for Public Opinion Research.¹² Patients were 67 years old on average (SD = 15), they were primarily white (97%), and 54% were female. All clinicians and 63% of nurses completed the survey.¹² Clinicians and nurses were 29% and 95% female, respectively.

Quantitative results (Table) show generally positive experience across patient and care team respondents. Over 90% were satisfied with all measures of communication. Care teams had high satisfaction with admissions processes and reported telemedicine improved cross-coverage. Patient-reported technology experience was positive but was less positive from the care team perspective. Care teams reported lower absolute quality of care than did patients but were more likely to perceive telemedicine as high quality, compared with traditional care. Most patients, clinicians, and nurses would recommend telemedicine.

Qualitatively, four major themes were identified in open-­ended responses with high interrater reliability (PABAK ranging from 0.92 to 0.98 in patient responses and 0.88 to 0.95 in care team responses) and aligned with the quantitative survey constructs: clinician-nurse communication, clinician-patient communication, workflow integration, and telemedicine technology. Patients reported satisfaction with communication with remote clinicians:

“[The clinician] was extremely attentive to me and what was going on. She was articulate and clear. I understood what was going to happen.” –Patient

Care teams suggested concrete improvement opportunities:

“I’d prefer to have some time with nursing staff both before and (sometimes) after the patient encounter.” –Clinician

“Since we cannot hear what [the clinicians] are hearing with the stethoscope, it’s nice when they tell us when to move it to the next spot.” –Nurse

Clinicians and nurses gave favorable responses regarding workflow integration, though time (both admissions wait time and session duration) was a reported opportunity:

“It would be helpful if we could speed up the time from admit request to screen time.” –Clinician

“When the [clinicians] get swamped, they’re hard to get a hold of, and admissions can take a long time. They may have too much on their plates dealing with several locations.” –Nurse

Technology issues—internet connection, stethoscope, sound, and screen or camera—were mentioned by patients and care teams, though technology was reviewed favorably overall by most patients:

“I was fascinated by the technology. Visiting someone over a television was impressive. ... The picture, the sound clarity, and the connection itself was flawless.” –Patient

Some patients commented that telemedicine was the best option given the situation, but still preferred an in-person doctor:

“If a doctor wasn’t available, telemedicine is better than nothing.” –Patient

Nurses who would not recommend telemedicine noted the need for personal connection:

“[I] still prefer [an] in-person MD for more personal contact. The older patients often state they wish the doctor would come and see them.” –Nurse

Patients who would not recommend telemedicine also desired personal connection:

“I would sooner talk to a person than a machine.” –Patient

A few clinicians noted the connection with patients would be improved if they knew about others in the room:

“It’d be nice if everyone in the room was introduced. Sometimes people are sitting out of view of the camera and I don’t realize they’re there until later.” –Clinician

These results make important contributions to understanding and improving the telemedicine experience in rural emergency hospital medicine. While the predominantly white patient respondent population limits generalizability, these demographics are representative of the overall population of the participating hospitals. A strength of this evaluation is its contemporaneous consideration of patient and care team experience with both quantitative and rich, qualitative analysis. Patients and care teams alike thought overnight telemedicine was better than the status quo. While our quality of care findings align with some previous literature,¹³ care teams in the current analysis overwhelmingly would recommend telemedicine, whereas some clinicians in prior work would not recommend telemedicine.¹⁴

In terms of communication, in line with existing literature, some patients still preferred in-person visits,¹⁵ a view also shared by some care team members. Workflow and technology barriers were raised, corroborating existing work,¹³ but actionable solutions (eg, adding care team–only time before visits or verbalizing when to move stethoscopes) were also identified.

Embedding patient and care team experience surveys and sharing results is critical in advancing telemedicine. Findings from this evaluation strengthen the case for payer reimbursement of telemedicine in rural acute care. Continued work to improve, test, and publish findings on patient and care team experience with telemedicine is critical to providing quality services in often-underserved communities.

The authors would like to acknowledge the contributions of Ann Werner in identifying the patient survey sample, Brian Barklind in identifying source data for the analysis, and both Brian Barklind and Rachael Rivard for conducting the analyses and summarizing results. We would also like to thank Kelly Logue for her involvement in conceptualizing the telemedicine evaluation described here, as well as Larisa Polynskaya for her help preparing the manuscript for publication, and the care teams and patients who provided valuable input.

Healthcare delivery in rural America faces unique, growing challenges related to health and emergency care access.¹ Telemedicine approaches have the potential to increase rural hospitals’ ability to deliver efficient emergency care and reduce clinician shortages.² While initial evidence of telemedicine success exists, more quality research is needed to understand telemedicine patient and care team experiences,³ especially with real-time, clinician-initiated video conferencing in critical access hospital (CAH) emergency departments (ED). Some experience studies exist,⁴ but results are primarily quantitative⁵ and lack the nuanced qualitative depth needed to understand topics such as satisfaction and communication.⁶ Additionally, few explore combined patient and care team perspectives.⁵ The lack of breadth and depth makes it difficult to provide actionable recommendations for improvements and affects the feasibility of continuing this work and improving telemedicine care quality. To address these gaps, we evaluated a real-time, clinician-­initiated video conferencing program with overnight clinicians servicing ED patients in three Midwestern care system CAHs. This evaluation assessed patient and care team (nurse and clinician) experience with telemedicine using quantitative and qualitative survey data analysis.

Because this evaluation was designed to measure and improve program quality in a single healthcare system, it was deemed non-human subjects research by the organization’s institutional review board. This brief report follows telemedicine reporting guidelines.⁷

Setting and Telemedicine Program

This program, designed to reduce the need for on-call hospitalist clinicians to be onsite at CAHs overnight, was implemented in a large Midwestern nonprofit integrated healthcare system with three rural CAHs (combined capacity for 75 inpatient admissions, with full-time onsite ED clinicians and nurses, as well as on-call hospitalist clinicians) and a large metropolitan tertiary-care hospital. All adult patients presenting to CAH EDs between 6 pm and 8 am were evaluated, as usual, by an onsite ED clinician. If the admitting ED clinician and charge nurse determined that admission was appropriate, patients were signed out to remote hospitalist clinicians and roomed by onsite nurses. Nurses facilitated live audio-video telemedicine “history and physical” visits with remote clinicians via telemedicine carts (AmericanWell C750, Boston, Massachusetts, and ThinkLabs One Electronic Stethoscope, Centennial, Colorado). Already-­hospitalized patients, as well as patients admitted to a remote clinician, were cared for by the remote clinician and onsite nurse for the remainder of the night, which eliminated the need for local on-call clinicians. The onsite ED clinician responded to emergencies of already-hospitalized patients, but often consulted with remote clinicians to assist virtually with necessary orders and documentation. Remote clinicians were located at the metropolitan tertiary care hospital or home work stations.

Following a pilot period, the full-scale program was implemented in September 2017 and included 14 remote clinicians and 60 onsite nurses.

Survey Administration and Design

A postimplementation survey was designed to explore patient and care team experience with telemedicine. Patients who received a telemedicine visit between September 2017 and April 2018 were mailed a paper survey. Nonresponders were called by professional interviewers affiliated with the healthcare system. All participating clinicians (N = 14, all MDs) and nurses (N = 60, all RNs) were emailed an online care team survey with phone-in option. Care team nonresponders were sent up to two reminder emails.

Surveys captured the following five constructs: communication, workflow integration, telemedicine technology, quality of care, and general satisfaction. Existing questionnaires were used where possible; additional items were designed with clinical experts following survey design best practices.⁸ Patient-perceived communication was assessed via three Consumer Assessment of Healthcare Providers and Systems Outpatient and Ambulatory Surgery Survey items.⁹ Five additional program-developed patient survey items included satisfaction with clinician-nurse communication, satisfaction with technology, telemedicine quality of care overall and in comparison with traditional care, and whether or not patients would recommend telemedicine (Table). Four open-ended questions asked patients about improvement opportunities and general satisfaction.

Care team surveys included two items regarding ability to effectively communicate, two about satisfaction with workflow integration, one about technical problems, two about quality of care, and one about general satisfaction. Open-ended questions gathered further information and recommendations to improve communication, workflow integration, technology issues, and general satisfaction.

Analysis

Closed-ended items were dichotomized (satisfied yes/no); descriptive statistics (frequencies/percents) are presented to quantify patient and care team experience. Quantitative analyses were conducted in SAS software version 9.4 (SAS Institute, Cary, North Carolina). Open-ended responses were coded separately for patient and care team experience, following qualitative content analysis best practices.¹⁰ A lead coder read all responses, created a coding framework of identified themes, and coded individual responses. A second coder independently coded responses using the same framework. Interrater reliability was calculated for each major theme using percent agreement and prevalence- and bias-adjusted k (PABAK) statistic. A single representative quote was selected and lightly edited for each subtheme to deepen understanding and provide respondent voice.¹¹

Of eligible patients mailed a survey (N = 408), 3% self-reported as ineligible, and 54% completed the survey. This is a maximum response rate (response rate 6) according to the American Association for Public Opinion Research.¹² Patients were 67 years old on average (SD = 15), they were primarily white (97%), and 54% were female. All clinicians and 63% of nurses completed the survey.¹² Clinicians and nurses were 29% and 95% female, respectively.

Quantitative results (Table) show generally positive experience across patient and care team respondents. Over 90% were satisfied with all measures of communication. Care teams had high satisfaction with admissions processes and reported telemedicine improved cross-coverage. Patient-reported technology experience was positive but was less positive from the care team perspective. Care teams reported lower absolute quality of care than did patients but were more likely to perceive telemedicine as high quality, compared with traditional care. Most patients, clinicians, and nurses would recommend telemedicine.

Qualitatively, four major themes were identified in open-­ended responses with high interrater reliability (PABAK ranging from 0.92 to 0.98 in patient responses and 0.88 to 0.95 in care team responses) and aligned with the quantitative survey constructs: clinician-nurse communication, clinician-patient communication, workflow integration, and telemedicine technology. Patients reported satisfaction with communication with remote clinicians:

“[The clinician] was extremely attentive to me and what was going on. She was articulate and clear. I understood what was going to happen.” –Patient

Care teams suggested concrete improvement opportunities:

“I’d prefer to have some time with nursing staff both before and (sometimes) after the patient encounter.” –Clinician

“Since we cannot hear what [the clinicians] are hearing with the stethoscope, it’s nice when they tell us when to move it to the next spot.” –Nurse

Clinicians and nurses gave favorable responses regarding workflow integration, though time (both admissions wait time and session duration) was a reported opportunity:

“It would be helpful if we could speed up the time from admit request to screen time.” –Clinician

“When the [clinicians] get swamped, they’re hard to get a hold of, and admissions can take a long time. They may have too much on their plates dealing with several locations.” –Nurse

Technology issues—internet connection, stethoscope, sound, and screen or camera—were mentioned by patients and care teams, though technology was reviewed favorably overall by most patients:

“I was fascinated by the technology. Visiting someone over a television was impressive. ... The picture, the sound clarity, and the connection itself was flawless.” –Patient

Some patients commented that telemedicine was the best option given the situation, but still preferred an in-person doctor:

“If a doctor wasn’t available, telemedicine is better than nothing.” –Patient

Nurses who would not recommend telemedicine noted the need for personal connection:

“[I] still prefer [an] in-person MD for more personal contact. The older patients often state they wish the doctor would come and see them.” –Nurse

Patients who would not recommend telemedicine also desired personal connection:

“I would sooner talk to a person than a machine.” –Patient

A few clinicians noted the connection with patients would be improved if they knew about others in the room:

“It’d be nice if everyone in the room was introduced. Sometimes people are sitting out of view of the camera and I don’t realize they’re there until later.” –Clinician

These results make important contributions to understanding and improving the telemedicine experience in rural emergency hospital medicine. While the predominantly white patient respondent population limits generalizability, these demographics are representative of the overall population of the participating hospitals. A strength of this evaluation is its contemporaneous consideration of patient and care team experience with both quantitative and rich, qualitative analysis. Patients and care teams alike thought overnight telemedicine was better than the status quo. While our quality of care findings align with some previous literature,¹³ care teams in the current analysis overwhelmingly would recommend telemedicine, whereas some clinicians in prior work would not recommend telemedicine.¹⁴

In terms of communication, in line with existing literature, some patients still preferred in-person visits,¹⁵ a view also shared by some care team members. Workflow and technology barriers were raised, corroborating existing work,¹³ but actionable solutions (eg, adding care team–only time before visits or verbalizing when to move stethoscopes) were also identified.

Embedding patient and care team experience surveys and sharing results is critical in advancing telemedicine. Findings from this evaluation strengthen the case for payer reimbursement of telemedicine in rural acute care. Continued work to improve, test, and publish findings on patient and care team experience with telemedicine is critical to providing quality services in often-underserved communities.

The authors would like to acknowledge the contributions of Ann Werner in identifying the patient survey sample, Brian Barklind in identifying source data for the analysis, and both Brian Barklind and Rachael Rivard for conducting the analyses and summarizing results. We would also like to thank Kelly Logue for her involvement in conceptualizing the telemedicine evaluation described here, as well as Larisa Polynskaya for her help preparing the manuscript for publication, and the care teams and patients who provided valuable input.

Sleep disturbance is common in hospitals, and both the quality and quantity of sleep are negatively affected in hospitalized patients.¹ Sleep disturbances in hospitals are associated with hyperglycemia,² delirium,³ lower patient satisfaction,⁴ and increased risk of readmission.⁵

A significant proportion of hospitalized patients receive sleep medications (ie, hypnotic medication) despite limited evidence.^6,7 Sleep medications have adverse effects including falls, fractures, cognitive impairment, and delirium.⁸ Commonly used nonbenzodiazepine sleep medications (eg, zopiclone) are perceived as safer but may have similar risks.⁹

Melatonin is increasingly used to treat insomnia, although evidence for its efficacy and safety is lacking.¹⁰ While melatonin use doubled between 2007 and 2012 in the United States,¹¹ previous hospital-based studies have not included melatonin.⁷ It is not known if increased melatonin use in the hospital mitigates use of higher-risk medications. Melatonin preparations can also have quality issues, including deviations from labelled dosage and contamination with compounds such as serotonin,¹² and patients continuing melatonin after discharge could have adverse effects if switched to different preparations.

In this study, we aimed to determine temporal trends in melatonin use in hospitalized patients, and compare them with trends in use of other sleep medications.

We conducted the study at two urban academic hospitals with a total of 706 acute care beds in the same network in Toronto, Canada. This study was approved by the University Health Network’s research ethics board.

We abstracted pharmacy dispensing data on melatonin, zopiclone, and lorazepam from January 1, 2013, to December 31, 2018. We included oral medications dispensed to inpatient units or admitted patients in the emergency department (ED). Zopiclone is the most commonly used nonbenzodiazepine sleep medication in Canada.¹³ Lorazepam is the most commonly used benzodiazepine at our institution (97% of all oral benzodiazepine doses). While lorazepam is prescribed for many nonsleep indications, we included it to assess the impact of melatonin. We did not include antipsychotics or trazodone, which are rarely newly initiated for insomnia at our institution.

We abstracted the monthly number of doses dispensed by unit and hospital. We categorized units based on the primary patient population as either internal medicine, critical care, or other. Admitted patients in the ED were counted as “other” regardless of service. As the focus of our study was on internal medicine and critical care, we did not analyze by type of unit in the “other” group, which is heterogeneous.

Each medication-dispensing event was counted as one dose, regardless of the number or strength of tablets (eg, a patient dispensed two 3-mg tablets of melatonin, for a total of 6 mg, would be counted as a single dose). Most unused doses are credited back (ie, if a medication was refused and returned to pharmacy, it was not counted). Lorazepam and zopiclone are on hospital formulary, while melatonin is not. To order melatonin, clinicians must select “Nonformulary medication” in the electronic health record and manually enter medication name, dose, route, and frequency, as well as select a justification for use. The hospital supplies nonformulary medications such as melatonin to patients.

To account for changes in patient volumes, we standardized medication dispensing rates per 1,000 inpatient days. We discovered rare instances in which the monthly number of doses was a negative number because of pharmacy inventory accounting. This issue affected only 0.13% of observations and the magnitude was small (8 doses or fewer); in these cases, we assumed the number of doses was zero.

We used line charts to visualize changes in medication dispensing over time by medication and hospital. We compared rates of medications use between unit type and hospital with use of relative difference and rate difference. Statistical analysis was performed with R (The R Foundation for Statistical Computing, 2018) using lubridate (2011), dplyr (2018), ggplot2 (2016), fmsb (2019), and forcats (2018).

A total of 1,542,225 inpatient days were analyzed, of which 60.4% were at hospital A. Internal medicine accounted for 23.5% of inpatient days, critical care for 11.7%, and other units for 64.8%.

Overall Trends in Sleep Medication Use

There were 351,131 dispensed doses of study medications (13% melatonin, 43% lorazepam, and 44% zopiclone). Overall use of the three study medications per 1,000 inpatient days increased by 25.7% during the study.

Melatonin use increased by 71.3 doses per 1,000 inpatient days during 2013-2018, while zopiclone use decreased by 20.4 doses per 1,000 inpatient days (Table). Lorazepam use increased slightly by 3.5 doses per 1,000 inpatient days. All rate differences reported in the results are statistically significant (Appendix Table).

Unit Type Comparison

Melatonin use was highest in critical care and internal medicine (50.9 and 48.4 doses per 1,000 inpatient days, respectively), compared with that in other units (19.3 doses per 1,000 inpatient days). Among critical care units, melatonin use was highest in medical-surgical units (67.4 doses per 1,000 inpatient days) and lower in cardiac and cardiovascular surgery units (24.6 and 18.3 doses per 1,000 inpatient days respectively). Zopiclone use was highest in critical care and other units (117.9 and 112.2 doses per 1,000 inpatient days, respectively) and lowest in internal medicine (57.0 doses per 1,000 inpatient days).

Hospital Site Comparison

Overall melatonin use was 65.4% higher at hospital B than at hospital A (42.4 vs 21.5 doses per 1,000 inpatient days; Figure). Zopiclone use was 81.7% lower at hospital B (54.6 vs 130.0 doses per 1,000 inpatient days).

When similar units were compared between hospitals, the trends were similar. For example, among internal medicine units, melatonin use was 66.7% higher at hospital B than at hospital A (64.4 vs 32.3 doses per 1,000 inpatient days).

During this 6-year study period of sleep medication use at two academic hospitals, overall use of melatonin, zopiclone, and lorazepam increased by 25.7%. Melatonin increased from almost no use to more than 70 doses per 1,000 inpatient days. The increase in melatonin was not accompanied by a proportional decline in zopiclone, which only decreased by 20.4 doses per 1,000 inpatient days. Lorazepam use increased slightly. This suggests that melatonin is not simply being substituted for higher-risk sleep medications and is instead being given to patients who might not have received sleep medications otherwise.

There are a few potential explanations for the disproportionate increase in melatonin. Providers may be more liberal in prescribing melatonin for insomnia because of perceived greater safety, compared with other medications. Melatonin may also be prescribed for delirium, despite a lack of high-quality evidence.¹⁴ Interestingly, melatonin use has increased despite a paucity of evidence for its efficacy or safety in hospital.⁶ Considering the additional barriers that exist to ordering melatonin, a nonformulary medication at our institution, the magnitude of increase is even more striking.

Melatonin use was highest on internal medicine and critical care units. This may reflect patient differences (eg, older patients with more comorbid conditions might leave prescribers reluctant to use benzodiazepines), differences in the physical environment (eg, noise/lighting), differences in nursing practices (eg, intensity of monitoring or medication administration), or differences in prescribing.

Melatonin use was almost twice as high at hospital B as it was at hospital A. While the services differ at each hospital, the results were similar when comparing the same unit type (eg, internal medicine). Internal medicine units have similar (though not identical) patient populations and team structures at both hospitals, and residents rotate between hospitals. Attendings and nurses are based primarily at one hospital and their practice patterns might differ. Geriatricians have a stronger presence at hospital B. Higher zopiclone use at hospital A could explain lower melatonin use. Lastly, improvement initiatives may have contributed (eg, one unit at hospital B promoted melatonin in 2017).

Our study has potential limitations. We studied dispensed rather than administered medications; however, numbers of doses dispensed but not administered are expected to be low because most unused doses are accounted for. By studying dispensing data, we might have underestimated the number of prescriptions (eg, if a patient was prescribed but refused a medication, this would not be captured). Our study did not examine medications after hospital discharge, although medications started in a hospital are often continued at discharge.^7,15 Our study could not determine indications for medication prescribing, and melatonin and lorazepam are both used for nonsleep indications. Our study could not differentiate between continuation of home medications and new prescriptions. Finally, the results may not be generalizable to other settings.

In this 6-year study of sleep medication use at two academic hospitals, we found that overall use of melatonin, zopiclone, and lorazepam increased by 25%, predominantly because of markedly increased melatonin use. Given the current lack of high-quality evidence, further research on the use of melatonin in hospitalized patients is needed.

Sleep disturbance is common in hospitals, and both the quality and quantity of sleep are negatively affected in hospitalized patients.¹ Sleep disturbances in hospitals are associated with hyperglycemia,² delirium,³ lower patient satisfaction,⁴ and increased risk of readmission.⁵

A significant proportion of hospitalized patients receive sleep medications (ie, hypnotic medication) despite limited evidence.^6,7 Sleep medications have adverse effects including falls, fractures, cognitive impairment, and delirium.⁸ Commonly used nonbenzodiazepine sleep medications (eg, zopiclone) are perceived as safer but may have similar risks.⁹

Melatonin is increasingly used to treat insomnia, although evidence for its efficacy and safety is lacking.¹⁰ While melatonin use doubled between 2007 and 2012 in the United States,¹¹ previous hospital-based studies have not included melatonin.⁷ It is not known if increased melatonin use in the hospital mitigates use of higher-risk medications. Melatonin preparations can also have quality issues, including deviations from labelled dosage and contamination with compounds such as serotonin,¹² and patients continuing melatonin after discharge could have adverse effects if switched to different preparations.

In this study, we aimed to determine temporal trends in melatonin use in hospitalized patients, and compare them with trends in use of other sleep medications.

We conducted the study at two urban academic hospitals with a total of 706 acute care beds in the same network in Toronto, Canada. This study was approved by the University Health Network’s research ethics board.

We abstracted pharmacy dispensing data on melatonin, zopiclone, and lorazepam from January 1, 2013, to December 31, 2018. We included oral medications dispensed to inpatient units or admitted patients in the emergency department (ED). Zopiclone is the most commonly used nonbenzodiazepine sleep medication in Canada.¹³ Lorazepam is the most commonly used benzodiazepine at our institution (97% of all oral benzodiazepine doses). While lorazepam is prescribed for many nonsleep indications, we included it to assess the impact of melatonin. We did not include antipsychotics or trazodone, which are rarely newly initiated for insomnia at our institution.

We abstracted the monthly number of doses dispensed by unit and hospital. We categorized units based on the primary patient population as either internal medicine, critical care, or other. Admitted patients in the ED were counted as “other” regardless of service. As the focus of our study was on internal medicine and critical care, we did not analyze by type of unit in the “other” group, which is heterogeneous.

Each medication-dispensing event was counted as one dose, regardless of the number or strength of tablets (eg, a patient dispensed two 3-mg tablets of melatonin, for a total of 6 mg, would be counted as a single dose). Most unused doses are credited back (ie, if a medication was refused and returned to pharmacy, it was not counted). Lorazepam and zopiclone are on hospital formulary, while melatonin is not. To order melatonin, clinicians must select “Nonformulary medication” in the electronic health record and manually enter medication name, dose, route, and frequency, as well as select a justification for use. The hospital supplies nonformulary medications such as melatonin to patients.

To account for changes in patient volumes, we standardized medication dispensing rates per 1,000 inpatient days. We discovered rare instances in which the monthly number of doses was a negative number because of pharmacy inventory accounting. This issue affected only 0.13% of observations and the magnitude was small (8 doses or fewer); in these cases, we assumed the number of doses was zero.

We used line charts to visualize changes in medication dispensing over time by medication and hospital. We compared rates of medications use between unit type and hospital with use of relative difference and rate difference. Statistical analysis was performed with R (The R Foundation for Statistical Computing, 2018) using lubridate (2011), dplyr (2018), ggplot2 (2016), fmsb (2019), and forcats (2018).

A total of 1,542,225 inpatient days were analyzed, of which 60.4% were at hospital A. Internal medicine accounted for 23.5% of inpatient days, critical care for 11.7%, and other units for 64.8%.

Overall Trends in Sleep Medication Use

There were 351,131 dispensed doses of study medications (13% melatonin, 43% lorazepam, and 44% zopiclone). Overall use of the three study medications per 1,000 inpatient days increased by 25.7% during the study.

Melatonin use increased by 71.3 doses per 1,000 inpatient days during 2013-2018, while zopiclone use decreased by 20.4 doses per 1,000 inpatient days (Table). Lorazepam use increased slightly by 3.5 doses per 1,000 inpatient days. All rate differences reported in the results are statistically significant (Appendix Table).

Unit Type Comparison

Melatonin use was highest in critical care and internal medicine (50.9 and 48.4 doses per 1,000 inpatient days, respectively), compared with that in other units (19.3 doses per 1,000 inpatient days). Among critical care units, melatonin use was highest in medical-surgical units (67.4 doses per 1,000 inpatient days) and lower in cardiac and cardiovascular surgery units (24.6 and 18.3 doses per 1,000 inpatient days respectively). Zopiclone use was highest in critical care and other units (117.9 and 112.2 doses per 1,000 inpatient days, respectively) and lowest in internal medicine (57.0 doses per 1,000 inpatient days).

Hospital Site Comparison

Overall melatonin use was 65.4% higher at hospital B than at hospital A (42.4 vs 21.5 doses per 1,000 inpatient days; Figure). Zopiclone use was 81.7% lower at hospital B (54.6 vs 130.0 doses per 1,000 inpatient days).

When similar units were compared between hospitals, the trends were similar. For example, among internal medicine units, melatonin use was 66.7% higher at hospital B than at hospital A (64.4 vs 32.3 doses per 1,000 inpatient days).

During this 6-year study period of sleep medication use at two academic hospitals, overall use of melatonin, zopiclone, and lorazepam increased by 25.7%. Melatonin increased from almost no use to more than 70 doses per 1,000 inpatient days. The increase in melatonin was not accompanied by a proportional decline in zopiclone, which only decreased by 20.4 doses per 1,000 inpatient days. Lorazepam use increased slightly. This suggests that melatonin is not simply being substituted for higher-risk sleep medications and is instead being given to patients who might not have received sleep medications otherwise.

There are a few potential explanations for the disproportionate increase in melatonin. Providers may be more liberal in prescribing melatonin for insomnia because of perceived greater safety, compared with other medications. Melatonin may also be prescribed for delirium, despite a lack of high-quality evidence.¹⁴ Interestingly, melatonin use has increased despite a paucity of evidence for its efficacy or safety in hospital.⁶ Considering the additional barriers that exist to ordering melatonin, a nonformulary medication at our institution, the magnitude of increase is even more striking.

Melatonin use was highest on internal medicine and critical care units. This may reflect patient differences (eg, older patients with more comorbid conditions might leave prescribers reluctant to use benzodiazepines), differences in the physical environment (eg, noise/lighting), differences in nursing practices (eg, intensity of monitoring or medication administration), or differences in prescribing.

Melatonin use was almost twice as high at hospital B as it was at hospital A. While the services differ at each hospital, the results were similar when comparing the same unit type (eg, internal medicine). Internal medicine units have similar (though not identical) patient populations and team structures at both hospitals, and residents rotate between hospitals. Attendings and nurses are based primarily at one hospital and their practice patterns might differ. Geriatricians have a stronger presence at hospital B. Higher zopiclone use at hospital A could explain lower melatonin use. Lastly, improvement initiatives may have contributed (eg, one unit at hospital B promoted melatonin in 2017).

Our study has potential limitations. We studied dispensed rather than administered medications; however, numbers of doses dispensed but not administered are expected to be low because most unused doses are accounted for. By studying dispensing data, we might have underestimated the number of prescriptions (eg, if a patient was prescribed but refused a medication, this would not be captured). Our study did not examine medications after hospital discharge, although medications started in a hospital are often continued at discharge.^7,15 Our study could not determine indications for medication prescribing, and melatonin and lorazepam are both used for nonsleep indications. Our study could not differentiate between continuation of home medications and new prescriptions. Finally, the results may not be generalizable to other settings.

In this 6-year study of sleep medication use at two academic hospitals, we found that overall use of melatonin, zopiclone, and lorazepam increased by 25%, predominantly because of markedly increased melatonin use. Given the current lack of high-quality evidence, further research on the use of melatonin in hospitalized patients is needed.

Sleep disturbance is common in hospitals, and both the quality and quantity of sleep are negatively affected in hospitalized patients.¹ Sleep disturbances in hospitals are associated with hyperglycemia,² delirium,³ lower patient satisfaction,⁴ and increased risk of readmission.⁵

A significant proportion of hospitalized patients receive sleep medications (ie, hypnotic medication) despite limited evidence.^6,7 Sleep medications have adverse effects including falls, fractures, cognitive impairment, and delirium.⁸ Commonly used nonbenzodiazepine sleep medications (eg, zopiclone) are perceived as safer but may have similar risks.⁹

Melatonin is increasingly used to treat insomnia, although evidence for its efficacy and safety is lacking.¹⁰ While melatonin use doubled between 2007 and 2012 in the United States,¹¹ previous hospital-based studies have not included melatonin.⁷ It is not known if increased melatonin use in the hospital mitigates use of higher-risk medications. Melatonin preparations can also have quality issues, including deviations from labelled dosage and contamination with compounds such as serotonin,¹² and patients continuing melatonin after discharge could have adverse effects if switched to different preparations.

In this study, we aimed to determine temporal trends in melatonin use in hospitalized patients, and compare them with trends in use of other sleep medications.

We conducted the study at two urban academic hospitals with a total of 706 acute care beds in the same network in Toronto, Canada. This study was approved by the University Health Network’s research ethics board.

We abstracted pharmacy dispensing data on melatonin, zopiclone, and lorazepam from January 1, 2013, to December 31, 2018. We included oral medications dispensed to inpatient units or admitted patients in the emergency department (ED). Zopiclone is the most commonly used nonbenzodiazepine sleep medication in Canada.¹³ Lorazepam is the most commonly used benzodiazepine at our institution (97% of all oral benzodiazepine doses). While lorazepam is prescribed for many nonsleep indications, we included it to assess the impact of melatonin. We did not include antipsychotics or trazodone, which are rarely newly initiated for insomnia at our institution.

We abstracted the monthly number of doses dispensed by unit and hospital. We categorized units based on the primary patient population as either internal medicine, critical care, or other. Admitted patients in the ED were counted as “other” regardless of service. As the focus of our study was on internal medicine and critical care, we did not analyze by type of unit in the “other” group, which is heterogeneous.

Each medication-dispensing event was counted as one dose, regardless of the number or strength of tablets (eg, a patient dispensed two 3-mg tablets of melatonin, for a total of 6 mg, would be counted as a single dose). Most unused doses are credited back (ie, if a medication was refused and returned to pharmacy, it was not counted). Lorazepam and zopiclone are on hospital formulary, while melatonin is not. To order melatonin, clinicians must select “Nonformulary medication” in the electronic health record and manually enter medication name, dose, route, and frequency, as well as select a justification for use. The hospital supplies nonformulary medications such as melatonin to patients.

To account for changes in patient volumes, we standardized medication dispensing rates per 1,000 inpatient days. We discovered rare instances in which the monthly number of doses was a negative number because of pharmacy inventory accounting. This issue affected only 0.13% of observations and the magnitude was small (8 doses or fewer); in these cases, we assumed the number of doses was zero.

We used line charts to visualize changes in medication dispensing over time by medication and hospital. We compared rates of medications use between unit type and hospital with use of relative difference and rate difference. Statistical analysis was performed with R (The R Foundation for Statistical Computing, 2018) using lubridate (2011), dplyr (2018), ggplot2 (2016), fmsb (2019), and forcats (2018).

A total of 1,542,225 inpatient days were analyzed, of which 60.4% were at hospital A. Internal medicine accounted for 23.5% of inpatient days, critical care for 11.7%, and other units for 64.8%.

Overall Trends in Sleep Medication Use

There were 351,131 dispensed doses of study medications (13% melatonin, 43% lorazepam, and 44% zopiclone). Overall use of the three study medications per 1,000 inpatient days increased by 25.7% during the study.

Melatonin use increased by 71.3 doses per 1,000 inpatient days during 2013-2018, while zopiclone use decreased by 20.4 doses per 1,000 inpatient days (Table). Lorazepam use increased slightly by 3.5 doses per 1,000 inpatient days. All rate differences reported in the results are statistically significant (Appendix Table).

Unit Type Comparison

Melatonin use was highest in critical care and internal medicine (50.9 and 48.4 doses per 1,000 inpatient days, respectively), compared with that in other units (19.3 doses per 1,000 inpatient days). Among critical care units, melatonin use was highest in medical-surgical units (67.4 doses per 1,000 inpatient days) and lower in cardiac and cardiovascular surgery units (24.6 and 18.3 doses per 1,000 inpatient days respectively). Zopiclone use was highest in critical care and other units (117.9 and 112.2 doses per 1,000 inpatient days, respectively) and lowest in internal medicine (57.0 doses per 1,000 inpatient days).

Hospital Site Comparison

Overall melatonin use was 65.4% higher at hospital B than at hospital A (42.4 vs 21.5 doses per 1,000 inpatient days; Figure). Zopiclone use was 81.7% lower at hospital B (54.6 vs 130.0 doses per 1,000 inpatient days).

When similar units were compared between hospitals, the trends were similar. For example, among internal medicine units, melatonin use was 66.7% higher at hospital B than at hospital A (64.4 vs 32.3 doses per 1,000 inpatient days).

During this 6-year study period of sleep medication use at two academic hospitals, overall use of melatonin, zopiclone, and lorazepam increased by 25.7%. Melatonin increased from almost no use to more than 70 doses per 1,000 inpatient days. The increase in melatonin was not accompanied by a proportional decline in zopiclone, which only decreased by 20.4 doses per 1,000 inpatient days. Lorazepam use increased slightly. This suggests that melatonin is not simply being substituted for higher-risk sleep medications and is instead being given to patients who might not have received sleep medications otherwise.

There are a few potential explanations for the disproportionate increase in melatonin. Providers may be more liberal in prescribing melatonin for insomnia because of perceived greater safety, compared with other medications. Melatonin may also be prescribed for delirium, despite a lack of high-quality evidence.¹⁴ Interestingly, melatonin use has increased despite a paucity of evidence for its efficacy or safety in hospital.⁶ Considering the additional barriers that exist to ordering melatonin, a nonformulary medication at our institution, the magnitude of increase is even more striking.

Melatonin use was highest on internal medicine and critical care units. This may reflect patient differences (eg, older patients with more comorbid conditions might leave prescribers reluctant to use benzodiazepines), differences in the physical environment (eg, noise/lighting), differences in nursing practices (eg, intensity of monitoring or medication administration), or differences in prescribing.

Melatonin use was almost twice as high at hospital B as it was at hospital A. While the services differ at each hospital, the results were similar when comparing the same unit type (eg, internal medicine). Internal medicine units have similar (though not identical) patient populations and team structures at both hospitals, and residents rotate between hospitals. Attendings and nurses are based primarily at one hospital and their practice patterns might differ. Geriatricians have a stronger presence at hospital B. Higher zopiclone use at hospital A could explain lower melatonin use. Lastly, improvement initiatives may have contributed (eg, one unit at hospital B promoted melatonin in 2017).

Our study has potential limitations. We studied dispensed rather than administered medications; however, numbers of doses dispensed but not administered are expected to be low because most unused doses are accounted for. By studying dispensing data, we might have underestimated the number of prescriptions (eg, if a patient was prescribed but refused a medication, this would not be captured). Our study did not examine medications after hospital discharge, although medications started in a hospital are often continued at discharge.^7,15 Our study could not determine indications for medication prescribing, and melatonin and lorazepam are both used for nonsleep indications. Our study could not differentiate between continuation of home medications and new prescriptions. Finally, the results may not be generalizable to other settings.

In this 6-year study of sleep medication use at two academic hospitals, we found that overall use of melatonin, zopiclone, and lorazepam increased by 25%, predominantly because of markedly increased melatonin use. Given the current lack of high-quality evidence, further research on the use of melatonin in hospitalized patients is needed.

Increasing numbers of children and adolescents are presenting to children’s hospitals with acute mental health crises requiring emergent or inpatient treatment.^1-5 As a result, children’s hospitals are experiencing additional financial challenges because specialty mental health services are often reimbursed at lower rates than other medical services.^6-9 Poor reimbursement has also been cited as a deterrent to the provision of mental health specialty care, including emergency mental health crisis services.¹⁰ The cumulative financial impact of recent trends in the provision of mental health crisis services at children’s hospitals, however, is unknown. We conducted this study to assess children’s hospitals’ costs, reimbursement, and net profits or losses when delivering inpatient mental health care.

We conducted a retrospective cohort study of the Children’s Hospital Association’s Pediatric Health Information System (PHIS) and Revenue Management Program (RMP) databases. PHIS is an administrative and billing database that collects International Classification of Disease, 10th Revision (ICD-10) diagnoses, procedure codes, and hospital charges from encounters at 52 US children’s hospitals. Costs are estimated from charges using hospital-, year-, and department-specific cost-to-charge ratios. The RMP database is an add-on module to the PHIS database that captures reimbursement data submitted quarterly from 17 participating hospitals based on actual reimbursement amounts collected for each encounter.

Among the 17 participating hospitals, we included all medical (ie, not surgical or intensive care) encounters during calendar year 2017 for children older than 6 years. We stratified encounters into three diagnosis types: primary mental health diagnosis,⁵ suicide attempt,¹¹ or other medical hospitalizations. We separated suicide attempts since these encounters often require care for both mental health concerns and medical complications. Eating disorders were excluded because these programs at children’s hospitals primarily focus on medical complications, require complex multispecialty support, have significantly longer hospitalizations and made up a small volume of overall mental health hospitalizations.

We stratified all analyses by inpatient or observation encounter and determined the proportion of encounters and hospital days attributed to primary mental health, suicide attempt, and other medical conditions at each hospital. One of the 17 children’s hospitals does not use observation status billing, so the observation encounters dataset includes 16 hospitals.

We summarized patients’ demographic and clinical characteristics using frequencies and percentages, comparing across diagnosis groups using chi-square tests. We calculated mean cost per day as (total cost) ÷ (total length of stay [LOS]), reimbursement per day as (total reimbursement) ÷ (total LOS) for each hospital and patient group, and margin per day as (reimbursement per day) – (cost per day). We then determined the total margin difference of caring for mental health vs caring for other medical encounters as ([margin per day for mental health] – [margin per day other medical]) × (number of mental health days). Similarly, we calculated the total margin loss for suicide attempts vs other medical encounters. After calculating profits and losses at individual hospitals, we summed total annual profits and losses to calculate cumulative annual differences. We summarized these profits and losses across all hospitals with medians and interquartile ranges (IQR).

This study of deidentified administrative data was approved by the Internal Review Board at Vanderbilt University as non-human subjects research. All statistical analyses were performed using SAS v.9.4 (SAS Institute, Cary, North Carolina), and P values < .05 were considered statistically significant.

Study Population

Across the 17 included children’s hospitals, there were 8,521 (7.6%) mental health encounters, 3,247 (2.9%) suicide attempt encounters, and 99,937 (89.5%) other medical encounters. LOS was significantly longer for mental health hospitalizations than for suicide attempts and for other medical hospitalizations.

Hospital Characteristics

All 17 free-standing children’s hospitals in the study had an inpatient behavioral health/psychiatric consultation service, and 7 of the 17 had an inpatient behavioral health/psychiatric unit. The total number of discharges for mental health, suicide attempt, and other medical conditions per year varied (range, 2,868-13,214) across the hospitals.

Hospital Daily Profits and Losses for Mental Health, Suicide Attempt, and Other Medical Admissions

For inpatient status mental health hospitalizations, the median margin was $376/day (IQR, $23-$618). For inpatient status suicide attempt hospitalizations, the median margin was $685/day (IQR, $3-$1,117), and for other medical hospitalizations the median margin was $603/day (IQR, $240-$991). With regard to observation status admissions, mental health hospitalizations had a median margin of –$453/day (IQR, –$806 to $362), suicide attempts of –$103/day (IQR, –$639 to $264), and other medical conditions of $353/day (IQR, –$616 to $658; Figure).

Hospital Annual Profits and Losses for Mental Health and Suicide Attempt Admissions, Compared With Other Medical Admissions

The Table shows daily and annual profits and losses for inpatient and observation status. The total annual loss across all hospitals for mental health and suicide attempt hospitalizations, compared with other medical hospitalizations, including both inpatient and observation status, was –$26,658,255 when taking both profits and losses into account. For the seven hospitals with net profits for mental health and suicide attempt hospitalizations, compared with other medical hospitalizations, the median net profit for combined inpatient and observation status encounters was $119,361 (IQR, $82,818-$195,543), and the total net profit was $5,872,665. For the 10 hospitals with net losses for mental health and suicide attempt hospitalizations, compared with other medical hospitalizations, the median net loss for combined inpatient and observation status was –$2,169,357 (IQR, –$4,034,085 to –$511,755), and the total net loss was –$27,419,379.

Hospitalizations for mental health disorders and suicide attempts accounted for 10.5% of hospitalizations at 17 US children’s hospitals in 2017. Overall, mental health and suicide attempt hospitalizations had lower financial margins than did other medical hospitalizations, and they accounted for a total margin loss of more than $26 million across 17 hospitals. Seven hospitals generated a profit for mental health and suicide attempt admissions; 10 hospitals reported losses. Only three hospitals generated a higher net profit for mental health admissions than for other medical admissions. More hospitals had net profits for inpatient status mental health and suicide attempt admissions than for observation status mental health and suicide attempt admissions.

For a minority of children’s hospitals, mental health hospitalizations had higher profit margins than for other medical hospitalizations. This raises questions about patient outcomes and the type of care models employed. One potential explanation is that these hospitals have negotiated favorable agreements with payers. Another possibility could be variations in case-mix and payer mix. Certain mental health services, such as crisis response teams, social workers, and child life specialists, may also be funded from nonpayer sources, so estimates may not fully reflect the cost of providing mental health services. A worst-case view is that hospitals with higher profit margins are providing less or poorer care because of lower reimbursement.

Mental health and suicide attempt hospitalizations were associated with smaller margins but counterintuitively generally wider IQRs for cost. This might be related to variation in care models, but our study was not positioned to examine reasons for this variation. The relationship between reimbursement or margins and patient outcomes, as well as specific mechanisms which may drive costs and outcomes, are areas for future research.

Health insurance plays a crucial role in mental health care. In our study, hospitals were more likely to report positive margins from inpatient status mental health hospitalizations rather than from observation status ones. This is unsurprising because payments for observation status are generally lower than for inpatient status.¹² Less is known about what influences billing and payment for inpatient versus observation at individual hospitals, particularly for mental health hospitalizations. In many cases, billing status is not strictly under the hospital’s control and may be determined by payers during or after the hospitalization. Significant variability in the percentage of patients billed as observation status and the impact of lower, often negative, margins for observation mental health encounters, will have a disproportionate effect on some hospitals. Future work could investigate how these differences may influence overall costs and delivery of care.

This study has several limitations that deserve attention. Costs reported are based on cost to charge ratios, which may generate imperfect estimates. Data was limited to 17 freestanding children’s hospitals, and our findings may not generalize to other hospitals. We also compared mental health and suicide attempt hospitalizations with “other medical” hospitalizations. This broad group contains certain medical conditions that may have higher or lower profit margins than average, and estimates of the margins could be over- or underestimated. We assumed that mental health and suicide attempt admissions were displacing admissions with non–mental health medical conditions (ie, not an empty bed). If those beds would otherwise be unoccupied, raw margins are better estimates of the financial impact than margin differences between mental health/suicide attempt and other medical hospitalizations.

Children’s hospitals are more likely to have significantly lower financial margins for mental health and suicide attempt hospitalizations than for other medical hospitalizations. Future work to investigate how quality of care is associated with reimbursement can help ensure that funding for children’s acute mental health care services is commensurate with resources required to provide high quality services.

The authors had no financial relationships relevant to this article to disclose.

Research reported in this publication was supported by the National Institute of Mental Health of the National Institutes of Health under Award Number K23MH115162 (Doupnik).

The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Increasing numbers of children and adolescents are presenting to children’s hospitals with acute mental health crises requiring emergent or inpatient treatment.^1-5 As a result, children’s hospitals are experiencing additional financial challenges because specialty mental health services are often reimbursed at lower rates than other medical services.^6-9 Poor reimbursement has also been cited as a deterrent to the provision of mental health specialty care, including emergency mental health crisis services.¹⁰ The cumulative financial impact of recent trends in the provision of mental health crisis services at children’s hospitals, however, is unknown. We conducted this study to assess children’s hospitals’ costs, reimbursement, and net profits or losses when delivering inpatient mental health care.

We conducted a retrospective cohort study of the Children’s Hospital Association’s Pediatric Health Information System (PHIS) and Revenue Management Program (RMP) databases. PHIS is an administrative and billing database that collects International Classification of Disease, 10th Revision (ICD-10) diagnoses, procedure codes, and hospital charges from encounters at 52 US children’s hospitals. Costs are estimated from charges using hospital-, year-, and department-specific cost-to-charge ratios. The RMP database is an add-on module to the PHIS database that captures reimbursement data submitted quarterly from 17 participating hospitals based on actual reimbursement amounts collected for each encounter.

Among the 17 participating hospitals, we included all medical (ie, not surgical or intensive care) encounters during calendar year 2017 for children older than 6 years. We stratified encounters into three diagnosis types: primary mental health diagnosis,⁵ suicide attempt,¹¹ or other medical hospitalizations. We separated suicide attempts since these encounters often require care for both mental health concerns and medical complications. Eating disorders were excluded because these programs at children’s hospitals primarily focus on medical complications, require complex multispecialty support, have significantly longer hospitalizations and made up a small volume of overall mental health hospitalizations.

We stratified all analyses by inpatient or observation encounter and determined the proportion of encounters and hospital days attributed to primary mental health, suicide attempt, and other medical conditions at each hospital. One of the 17 children’s hospitals does not use observation status billing, so the observation encounters dataset includes 16 hospitals.

We summarized patients’ demographic and clinical characteristics using frequencies and percentages, comparing across diagnosis groups using chi-square tests. We calculated mean cost per day as (total cost) ÷ (total length of stay [LOS]), reimbursement per day as (total reimbursement) ÷ (total LOS) for each hospital and patient group, and margin per day as (reimbursement per day) – (cost per day). We then determined the total margin difference of caring for mental health vs caring for other medical encounters as ([margin per day for mental health] – [margin per day other medical]) × (number of mental health days). Similarly, we calculated the total margin loss for suicide attempts vs other medical encounters. After calculating profits and losses at individual hospitals, we summed total annual profits and losses to calculate cumulative annual differences. We summarized these profits and losses across all hospitals with medians and interquartile ranges (IQR).

This study of deidentified administrative data was approved by the Internal Review Board at Vanderbilt University as non-human subjects research. All statistical analyses were performed using SAS v.9.4 (SAS Institute, Cary, North Carolina), and P values < .05 were considered statistically significant.

Study Population

Across the 17 included children’s hospitals, there were 8,521 (7.6%) mental health encounters, 3,247 (2.9%) suicide attempt encounters, and 99,937 (89.5%) other medical encounters. LOS was significantly longer for mental health hospitalizations than for suicide attempts and for other medical hospitalizations.

Hospital Characteristics

All 17 free-standing children’s hospitals in the study had an inpatient behavioral health/psychiatric consultation service, and 7 of the 17 had an inpatient behavioral health/psychiatric unit. The total number of discharges for mental health, suicide attempt, and other medical conditions per year varied (range, 2,868-13,214) across the hospitals.

Hospital Daily Profits and Losses for Mental Health, Suicide Attempt, and Other Medical Admissions

For inpatient status mental health hospitalizations, the median margin was $376/day (IQR, $23-$618). For inpatient status suicide attempt hospitalizations, the median margin was $685/day (IQR, $3-$1,117), and for other medical hospitalizations the median margin was $603/day (IQR, $240-$991). With regard to observation status admissions, mental health hospitalizations had a median margin of –$453/day (IQR, –$806 to $362), suicide attempts of –$103/day (IQR, –$639 to $264), and other medical conditions of $353/day (IQR, –$616 to $658; Figure).

Hospital Annual Profits and Losses for Mental Health and Suicide Attempt Admissions, Compared With Other Medical Admissions

The Table shows daily and annual profits and losses for inpatient and observation status. The total annual loss across all hospitals for mental health and suicide attempt hospitalizations, compared with other medical hospitalizations, including both inpatient and observation status, was –$26,658,255 when taking both profits and losses into account. For the seven hospitals with net profits for mental health and suicide attempt hospitalizations, compared with other medical hospitalizations, the median net profit for combined inpatient and observation status encounters was $119,361 (IQR, $82,818-$195,543), and the total net profit was $5,872,665. For the 10 hospitals with net losses for mental health and suicide attempt hospitalizations, compared with other medical hospitalizations, the median net loss for combined inpatient and observation status was –$2,169,357 (IQR, –$4,034,085 to –$511,755), and the total net loss was –$27,419,379.

Hospitalizations for mental health disorders and suicide attempts accounted for 10.5% of hospitalizations at 17 US children’s hospitals in 2017. Overall, mental health and suicide attempt hospitalizations had lower financial margins than did other medical hospitalizations, and they accounted for a total margin loss of more than $26 million across 17 hospitals. Seven hospitals generated a profit for mental health and suicide attempt admissions; 10 hospitals reported losses. Only three hospitals generated a higher net profit for mental health admissions than for other medical admissions. More hospitals had net profits for inpatient status mental health and suicide attempt admissions than for observation status mental health and suicide attempt admissions.

For a minority of children’s hospitals, mental health hospitalizations had higher profit margins than for other medical hospitalizations. This raises questions about patient outcomes and the type of care models employed. One potential explanation is that these hospitals have negotiated favorable agreements with payers. Another possibility could be variations in case-mix and payer mix. Certain mental health services, such as crisis response teams, social workers, and child life specialists, may also be funded from nonpayer sources, so estimates may not fully reflect the cost of providing mental health services. A worst-case view is that hospitals with higher profit margins are providing less or poorer care because of lower reimbursement.

Mental health and suicide attempt hospitalizations were associated with smaller margins but counterintuitively generally wider IQRs for cost. This might be related to variation in care models, but our study was not positioned to examine reasons for this variation. The relationship between reimbursement or margins and patient outcomes, as well as specific mechanisms which may drive costs and outcomes, are areas for future research.

Health insurance plays a crucial role in mental health care. In our study, hospitals were more likely to report positive margins from inpatient status mental health hospitalizations rather than from observation status ones. This is unsurprising because payments for observation status are generally lower than for inpatient status.¹² Less is known about what influences billing and payment for inpatient versus observation at individual hospitals, particularly for mental health hospitalizations. In many cases, billing status is not strictly under the hospital’s control and may be determined by payers during or after the hospitalization. Significant variability in the percentage of patients billed as observation status and the impact of lower, often negative, margins for observation mental health encounters, will have a disproportionate effect on some hospitals. Future work could investigate how these differences may influence overall costs and delivery of care.

This study has several limitations that deserve attention. Costs reported are based on cost to charge ratios, which may generate imperfect estimates. Data was limited to 17 freestanding children’s hospitals, and our findings may not generalize to other hospitals. We also compared mental health and suicide attempt hospitalizations with “other medical” hospitalizations. This broad group contains certain medical conditions that may have higher or lower profit margins than average, and estimates of the margins could be over- or underestimated. We assumed that mental health and suicide attempt admissions were displacing admissions with non–mental health medical conditions (ie, not an empty bed). If those beds would otherwise be unoccupied, raw margins are better estimates of the financial impact than margin differences between mental health/suicide attempt and other medical hospitalizations.

Children’s hospitals are more likely to have significantly lower financial margins for mental health and suicide attempt hospitalizations than for other medical hospitalizations. Future work to investigate how quality of care is associated with reimbursement can help ensure that funding for children’s acute mental health care services is commensurate with resources required to provide high quality services.

The authors had no financial relationships relevant to this article to disclose.

Research reported in this publication was supported by the National Institute of Mental Health of the National Institutes of Health under Award Number K23MH115162 (Doupnik).

The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Increasing numbers of children and adolescents are presenting to children’s hospitals with acute mental health crises requiring emergent or inpatient treatment.^1-5 As a result, children’s hospitals are experiencing additional financial challenges because specialty mental health services are often reimbursed at lower rates than other medical services.^6-9 Poor reimbursement has also been cited as a deterrent to the provision of mental health specialty care, including emergency mental health crisis services.¹⁰ The cumulative financial impact of recent trends in the provision of mental health crisis services at children’s hospitals, however, is unknown. We conducted this study to assess children’s hospitals’ costs, reimbursement, and net profits or losses when delivering inpatient mental health care.

We conducted a retrospective cohort study of the Children’s Hospital Association’s Pediatric Health Information System (PHIS) and Revenue Management Program (RMP) databases. PHIS is an administrative and billing database that collects International Classification of Disease, 10th Revision (ICD-10) diagnoses, procedure codes, and hospital charges from encounters at 52 US children’s hospitals. Costs are estimated from charges using hospital-, year-, and department-specific cost-to-charge ratios. The RMP database is an add-on module to the PHIS database that captures reimbursement data submitted quarterly from 17 participating hospitals based on actual reimbursement amounts collected for each encounter.

Among the 17 participating hospitals, we included all medical (ie, not surgical or intensive care) encounters during calendar year 2017 for children older than 6 years. We stratified encounters into three diagnosis types: primary mental health diagnosis,⁵ suicide attempt,¹¹ or other medical hospitalizations. We separated suicide attempts since these encounters often require care for both mental health concerns and medical complications. Eating disorders were excluded because these programs at children’s hospitals primarily focus on medical complications, require complex multispecialty support, have significantly longer hospitalizations and made up a small volume of overall mental health hospitalizations.

We stratified all analyses by inpatient or observation encounter and determined the proportion of encounters and hospital days attributed to primary mental health, suicide attempt, and other medical conditions at each hospital. One of the 17 children’s hospitals does not use observation status billing, so the observation encounters dataset includes 16 hospitals.

We summarized patients’ demographic and clinical characteristics using frequencies and percentages, comparing across diagnosis groups using chi-square tests. We calculated mean cost per day as (total cost) ÷ (total length of stay [LOS]), reimbursement per day as (total reimbursement) ÷ (total LOS) for each hospital and patient group, and margin per day as (reimbursement per day) – (cost per day). We then determined the total margin difference of caring for mental health vs caring for other medical encounters as ([margin per day for mental health] – [margin per day other medical]) × (number of mental health days). Similarly, we calculated the total margin loss for suicide attempts vs other medical encounters. After calculating profits and losses at individual hospitals, we summed total annual profits and losses to calculate cumulative annual differences. We summarized these profits and losses across all hospitals with medians and interquartile ranges (IQR).

This study of deidentified administrative data was approved by the Internal Review Board at Vanderbilt University as non-human subjects research. All statistical analyses were performed using SAS v.9.4 (SAS Institute, Cary, North Carolina), and P values < .05 were considered statistically significant.

Study Population

Across the 17 included children’s hospitals, there were 8,521 (7.6%) mental health encounters, 3,247 (2.9%) suicide attempt encounters, and 99,937 (89.5%) other medical encounters. LOS was significantly longer for mental health hospitalizations than for suicide attempts and for other medical hospitalizations.

Hospital Characteristics

All 17 free-standing children’s hospitals in the study had an inpatient behavioral health/psychiatric consultation service, and 7 of the 17 had an inpatient behavioral health/psychiatric unit. The total number of discharges for mental health, suicide attempt, and other medical conditions per year varied (range, 2,868-13,214) across the hospitals.

Hospital Daily Profits and Losses for Mental Health, Suicide Attempt, and Other Medical Admissions

For inpatient status mental health hospitalizations, the median margin was $376/day (IQR, $23-$618). For inpatient status suicide attempt hospitalizations, the median margin was $685/day (IQR, $3-$1,117), and for other medical hospitalizations the median margin was $603/day (IQR, $240-$991). With regard to observation status admissions, mental health hospitalizations had a median margin of –$453/day (IQR, –$806 to $362), suicide attempts of –$103/day (IQR, –$639 to $264), and other medical conditions of $353/day (IQR, –$616 to $658; Figure).

Hospital Annual Profits and Losses for Mental Health and Suicide Attempt Admissions, Compared With Other Medical Admissions

The Table shows daily and annual profits and losses for inpatient and observation status. The total annual loss across all hospitals for mental health and suicide attempt hospitalizations, compared with other medical hospitalizations, including both inpatient and observation status, was –$26,658,255 when taking both profits and losses into account. For the seven hospitals with net profits for mental health and suicide attempt hospitalizations, compared with other medical hospitalizations, the median net profit for combined inpatient and observation status encounters was $119,361 (IQR, $82,818-$195,543), and the total net profit was $5,872,665. For the 10 hospitals with net losses for mental health and suicide attempt hospitalizations, compared with other medical hospitalizations, the median net loss for combined inpatient and observation status was –$2,169,357 (IQR, –$4,034,085 to –$511,755), and the total net loss was –$27,419,379.

Hospitalizations for mental health disorders and suicide attempts accounted for 10.5% of hospitalizations at 17 US children’s hospitals in 2017. Overall, mental health and suicide attempt hospitalizations had lower financial margins than did other medical hospitalizations, and they accounted for a total margin loss of more than $26 million across 17 hospitals. Seven hospitals generated a profit for mental health and suicide attempt admissions; 10 hospitals reported losses. Only three hospitals generated a higher net profit for mental health admissions than for other medical admissions. More hospitals had net profits for inpatient status mental health and suicide attempt admissions than for observation status mental health and suicide attempt admissions.

For a minority of children’s hospitals, mental health hospitalizations had higher profit margins than for other medical hospitalizations. This raises questions about patient outcomes and the type of care models employed. One potential explanation is that these hospitals have negotiated favorable agreements with payers. Another possibility could be variations in case-mix and payer mix. Certain mental health services, such as crisis response teams, social workers, and child life specialists, may also be funded from nonpayer sources, so estimates may not fully reflect the cost of providing mental health services. A worst-case view is that hospitals with higher profit margins are providing less or poorer care because of lower reimbursement.

Mental health and suicide attempt hospitalizations were associated with smaller margins but counterintuitively generally wider IQRs for cost. This might be related to variation in care models, but our study was not positioned to examine reasons for this variation. The relationship between reimbursement or margins and patient outcomes, as well as specific mechanisms which may drive costs and outcomes, are areas for future research.

Health insurance plays a crucial role in mental health care. In our study, hospitals were more likely to report positive margins from inpatient status mental health hospitalizations rather than from observation status ones. This is unsurprising because payments for observation status are generally lower than for inpatient status.¹² Less is known about what influences billing and payment for inpatient versus observation at individual hospitals, particularly for mental health hospitalizations. In many cases, billing status is not strictly under the hospital’s control and may be determined by payers during or after the hospitalization. Significant variability in the percentage of patients billed as observation status and the impact of lower, often negative, margins for observation mental health encounters, will have a disproportionate effect on some hospitals. Future work could investigate how these differences may influence overall costs and delivery of care.

This study has several limitations that deserve attention. Costs reported are based on cost to charge ratios, which may generate imperfect estimates. Data was limited to 17 freestanding children’s hospitals, and our findings may not generalize to other hospitals. We also compared mental health and suicide attempt hospitalizations with “other medical” hospitalizations. This broad group contains certain medical conditions that may have higher or lower profit margins than average, and estimates of the margins could be over- or underestimated. We assumed that mental health and suicide attempt admissions were displacing admissions with non–mental health medical conditions (ie, not an empty bed). If those beds would otherwise be unoccupied, raw margins are better estimates of the financial impact than margin differences between mental health/suicide attempt and other medical hospitalizations.

Children’s hospitals are more likely to have significantly lower financial margins for mental health and suicide attempt hospitalizations than for other medical hospitalizations. Future work to investigate how quality of care is associated with reimbursement can help ensure that funding for children’s acute mental health care services is commensurate with resources required to provide high quality services.

The authors had no financial relationships relevant to this article to disclose.

Research reported in this publication was supported by the National Institute of Mental Health of the National Institutes of Health under Award Number K23MH115162 (Doupnik).

The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.