User login
Fever is one of the most common reasons for emergency department (ED) evaluation of infants under 90 days of age.[1] Up to 10% to 20% of febrile young infants will have a serious bacterial infection (SBI),[2, 3, 4] but infants with SBI are difficult to distinguish from those without SBI based upon symptoms and physical examination findings alone.[5] Previously developed clinical prediction algorithms can help to identify febrile infants at low risk for SBI, but differ in age range as well as recommendations for testing and empiric treatment.[6, 7, 8] Consequently, there is widespread variation in management of febrile young infants at US children's hospitals,[9, 10, 11] and defining optimal management strategies remains an important issue in pediatric healthcare.[12] Administrative datasets are convenient and inexpensive, and can be used to evaluate practice variation, trends, and outcomes of a large, diverse group of patients within and across institutions.[9, 10] Accurately identifying febrile infants evaluated for suspected SBI in administrative databases would facilitate comparative effectiveness research, quality improvement initiatives, and institutional benchmarking.
Prior studies have validated the accuracy of administrative billing codes for identification of other common childhood illnesses, including urinary tract infection (UTI)[13] and pneumonia.[14] The accuracy of International Classification of Diseases, Ninth Revision (ICD‐9) diagnosis codes in identifying febrile young infants evaluated for SBI is not known. Reliance on administrative ICD‐9 diagnosis codes for patient identification can lead to misclassification of patients due to variable database quality, the validity of the diagnosis codes being utilized, and hospital coding practices.[15] Additionally, fever is a symptom and not a specific diagnosis. If a particular bacterial or viral diagnosis is established (eg, enterovirus meningitis), a discharge diagnosis of fever may not be attributed to the patient encounter. Thus, evaluating the performance characteristics and capture of clinical outcomes of different combinations of ICD‐9 diagnosis codes for identifying febrile infants is necessary for both the conduct and interpretation of studies that utilize administrative databases. The primary objective of this investigation was to identify the most accurate ICD‐9 coding strategies for the identification of febrile infants aged <90 days using administrative data. We also sought to evaluate capture of clinically important outcomes across identification strategies.
METHODS
Study Design and Setting
For this multicenter retrospective study, we used the Pediatric Health Information System (PHIS) database to identify infants <90 days of age[16] who presented between July 1, 2012 and June 30, 2013 to 1 of 8 EDs. We assessed performance characteristics of ICD‐9 diagnosis code case‐identification algorithms by comparing ICD‐9 code combinations to a fever reference standard determined by medical record review. The institutional review board at each participating site approved the study protocol.
Data Source
Data were obtained from 2 sources: the PHIS database and medical record review. We used the PHIS database to identify eligible patients by ICD‐9 diagnosis codes; patient encounters were randomly selected using a random number generator. The PHIS database contains demographic, diagnosis, and billing data from 44 hospitals affiliated with the Children's Hospital Association (Overland Park, Kansas) and represents 85% of freestanding children's hospitals in the United States.[17] Data are deidentified; encrypted unique patient identifiers permit tracking of patients across visits within a site.[18] The Children's Hospital Association and participating hospitals jointly assure the quality and integrity of the data.[19]
For each patient encounter identified in the PHIS database, detailed medical record review was performed by trained investigators at each of the 8 study sites (see Supporting Information, Appendix, in the online version of this article). A standardized data collection instrument was pilot tested by all investigators prior to use. Data were collected and managed using the Research Electronic Data Capture (REDCap) tool hosted at Boston Children's Hospital.[20]
Exclusions
Using PHIS data, prior to medical record review we excluded infants with a complex chronic condition as defined previously[21] and those transferred from another institution, as these infants may warrant a nonstandard evaluation and/or may have incomplete data.
ICD‐9 Diagnosis Code Groups
In the PHIS database, all patients discharged from the hospital (including hospitalized patients as well as patients discharged from the ED) receive 1 or more ICD‐9 discharge diagnosis codes. These diagnosis codes are ascribed after discharge from the hospital, or for ED patients, after ED discharge. Additionally, patients may receive an admission diagnosis, which reflects the diagnosis ascribed at the time of ED discharge or transfer to the inpatient unit.
We reviewed medical records of infants selected from the following ICD‐9 diagnosis code groups (Figure 1): (1) discharge diagnosis code of fever (780.6 [fever and other physiologic disturbances of temperature regulation], 778.4 [other disturbances of temperature regulation of newborn], 780.60 [fever, unspecified], or 780.61 [fever presenting with conditions classified elsewhere])[9, 10] regardless of the presence of admission diagnosis of fever or diagnosis of serious infection, (2) admission diagnosis code of fever without associated discharge diagnosis code of fever,[10] (3) discharge diagnosis code of serious infection determined a priori (see Supporting Information, Appendix, in the online version of this article) without discharge or admission diagnosis code of fever, and (4) infants without any diagnosis code of fever or serious infection.
Medical records reviewed in each of the 4 ICD‐9 diagnosis code groups were randomly selected from the overall set of ED encounters in the population of infants <90 days of age evaluated during the study period. Twenty‐five percent population sampling was used for 3 of the ICD‐9 diagnosis code groups, whereas 5% sampling was used for the no fever/no serious infection code group. The number of medical records reviewed in each ICD‐9 diagnosis code group was proportional to the distribution of ICD‐9 codes across the entire population of infants <90 days of age. These records were distributed equally across sites (228 records per site), except for 1 site that does not assign admission diagnoses (201 records).
Investigators were blinded to ICD‐9 diagnosis code groups during medical record review. Infants with multiple visits during the study period were eligible to be included more than once if the visits occurred more than 3 days apart. For infants with more than 1 ED visit on a particular calendar day, investigators were instructed to review the initial visit.
For each encounter, we also abstracted demographic characteristics (gender, race/ethnicity), insurance status, hospital region (using US Census categories[22]), and season from the PHIS database.
Reference Standard
The presence of fever was determined by medical record review. We defined fever as any documented temperature 100.4F (38.0C) at home or in the ED.[16]
ICD‐9 Code Case‐Identification Algorithms
Using the aforementioned ICD‐9 diagnosis code groups individually and in combination, the following 4 case‐identification algorithms, determined from prior study or group consensus, were compared to the reference standard: (1) ICD‐9 discharge diagnosis code of fever,[9] (2) ICD‐9 admission or discharge diagnosis code of fever,[10, 11] (3) ICD‐9 discharge diagnosis code of fever or serious infection, and (4) ICD‐9 discharge or admission diagnosis code of fever or serious infection. Algorithms were compared overall, separately for discharged and hospitalized infants, and across 3 distinct age groups (28 days, 2956 days, and 5789 days).
Patient‐Level Outcomes
To compare differences in outcomes by case‐identification algorithm, from the PHIS database we abstracted hospitalization rates, rates of UTI/pyelonephritis,[13] bacteremia/sepsis, and bacterial meningitis.[19] Severe outcomes were defined as intensive care unit admission, mechanical ventilation, central line placement, receipt of extracorporeal membrane oxygenation, or death. We assessed hospital length of stay for admitted infants and 3‐day revisits,[23, 24] and revisits resulting in hospitalization for infants discharged from the ED at the index visit. Patients billed for observation care were classified as being hospitalized.[25, 26]
Data Analysis
Accuracy of the 4 case‐identification algorithms (compared with the reference standard) was calculated using sensitivity, specificity, negative predictive value (NPV), and positive predictive value (PPV), along with 95% confidence interval (CI). Prior to analysis, a 5‐fold weighting factor was applied to the no fever/no serious infection group to account for the differential sampling used for this group (5% vs 25% for the other 3 ICD‐9 diagnosis code groups). This weighting was done to approximate the true prevalence of each ICD‐9 code group within the larger population, so that an accurate rate of false negatives (infants with fever who had neither a diagnosis of fever nor serious infection) could be calculated.
We described continuous variables using median and interquartile range or range values and categorical variables using frequencies with 95% CIs. We compared categorical variables using a 2 test. We determined statistical significance as a 2‐tailed P value <0.05. Statistical analyses were performed using SAS version 9.3 (SAS Institute, Cary, NC).
RESULTS
Study Patients
During the 1‐year study period, 23,753 ED encounters for infants <90 days of age were identified in the PHIS database at the 8 participating sites. Of these infant encounters, 2166 (9.2%) were excluded (1658 infants who had a complex chronic condition and 508 transferred into the ED), leaving 21,587 infants available for selection. After applying our sampling strategy, we identified 1797 encounters for medical record review. Seven encounters from 3 hospitals with missing medical records were excluded, resulting in a final cohort of 1790 encounters (Figure 1). Among included infants, 552 (30.8%) were 28 days, 743 (41.5%) were 29 to 56 days, and 495 (27.8%) were 57 to 89 days of age; 737 (41.2%) infants were hospitalized. Patients differed in age, race, payer, and season across ICD‐9 diagnosis code groups (see Supporting Information, Table 1, in the online version of this article).
| ICD‐9 Diagnosis Code Algorithm | Overall | |||
|---|---|---|---|---|
| Sensitivity, % (95% CI) | Specificity, % (95% CI) | Negative Predictive Value, % (95% CI) | Positive Predictive Value, % (95% CI) | |
| ||||
| Discharge diagnosis of fever | 53.2 (50.056.4) | 98.2 (97.898.6) | 90.8 (90.091.6) | 86.1 (83.388.9) |
| Hospitalized | 47.3 (43.151.5) | 97.7 (96.998.5) | 80.6 (78.682.6) | 90.2 (86.893.6) |
| Discharged from ED | 61.4 (56.666.2) | 98.4 (98.098.8) | 95.4 (94.796.1) | 82.1 (77.786.5) |
| Discharge or admission diagnosis of Fever | 71.1 (68.274.0) | 97.7 (97.398.1) | 94.1 (93.494.8) | 86.9 (84.589.3) |
| Hospitalized | 72.5 (68.876.2) | 97.1 (96.298.0) | 88.8 (87.190.5) | 91.7 (89.194.3) |
| Discharged from ED | 69.2 (64.773.7) | 98.0 (97.598.5) | 96.3 (95.796.9) | 80.8 (76.685.0) |
| Discharge diagnosis of fever or serious infection | 63.7 (60.666.8) | 96.5 (96.097.0) | 92.6 (91.893.4) | 79.6 (76.782.5) |
| Hospitalized | 63.9 (59.967.9) | 92.5 (91.094.0) | 85.1 (83.287.0) | 79.1 (75.382.9) |
| Discharged from ED | 63.4 (58.768.1) | 98.1 (97.698.6) | 95.6 (94.996.3) | 80.2 (75.884.6) |
| Discharge or admission diagnosis of fever or serious infection | 76.6 (73.979.3) | 96.2 (95.696.8) | 95.1 (94.595.7) | 81.0 (78.483.6) |
| Hospitalized | 80.8 (77.584.1) | 92.1 (90.693.6) | 91.5 (89.993.1) | 82.1 (78.985.3) |
| Discharged from ED | 71.0 (66.575.5) | 97.7 (97.298.2) | 96.5 (95.997.1) | 79.4 (75.283.6) |
Among the 1790 patient encounters reviewed, a total of 766 infants (42.8%) met the reference standard definition for fever in the cohort. An additional 47 infants had abnormal temperature reported (documentation of tactile fever, history of fever without a specific temperature described, or hypothermia) but were classified as having no fever by the reference standard.
ICD‐9 Code Case‐Identification Algorithm Performance
Compared with the reference standard, the 4 case‐identification algorithms demonstrated specificity of 96.2% to 98.2% but lower sensitivity overall (Figure 2). Discharge diagnosis of fever alone demonstrated the lowest sensitivity. The algorithm of discharge or admission diagnosis of fever resulted in increased sensitivity and the highest PPV of all 4 algorithms (86.9%, 95% CI: 84.5‐89.3). Addition of serious infection codes to this algorithm resulted in a marginal increase in sensitivity and a similar decrease in PPV (Table 1). When limited to hospitalized infants, specificity was highest for the case‐identification algorithm of discharge diagnosis of fever and similarly high for discharge or admission diagnosis of fever; sensitivity was highest for the algorithm of discharge or admission diagnosis of fever or diagnosis of serious infection. For infants discharged from the ED, algorithm specificity was 97.7% to 98.4%, with lower sensitivity for all 4 algorithms (Table 1). Inclusion of the 47 infants with abnormal temperature as fever did not materially change algorithm performance (data not shown).
Across all 3 age groups (28 days, 2956 days, and 5789 days), the 4 case‐identification algorithms demonstrated specificity >96%, whereas algorithm sensitivity was highest in the 29‐ to 56‐days‐old age group and lowest among infants 57 to 89 days old across all 4 algorithms (Figure 2). Similar to the overall cohort, an algorithm of discharge or admission diagnosis of fever demonstrated specificity of nearly 98% in all age groups; addition of serious infection codes to this algorithm increased sensitivity, highest in the 29‐ to 56‐days‐old age group (Figure 2; see also Supporting Information, Table 2, in the online version of this article).
| ICD‐9 Diagnosis Code Algorithm | Sensitivity, Median % (Range) | Specificity, Median % (Range) | Negative Predictive Value, Median % (Range) | Positive Predictive Value, Median % (Range) |
|---|---|---|---|---|
| ||||
| Discharge diagnosis of fever | 56.2 (34.681.0) | 98.3 (96.499.1) | 92.1 (83.297.4) | 87.7 (74.093.2) |
| Discharge or Admission diagnosis of Fever | 76.7 (51.385.0) | 97.8 (96.298.7) | 95.6 (86.997.4) | 87.4 (80.092.9) |
| Discharge diagnosis of fever or serious infection | 68.3 (44.287.3) | 96.5 (95.498.0) | 93.6 (85.298.2) | 78.3 (74.289.0) |
| Discharge or admission diagnosis of fever or serious infection | 83.1 (58.390.7) | 95.8 (95.498.0) | 96.5 (88.598.2) | 79.1 (77.490.4) |
Across the 8 study sites, median specificity was 95.8% to 98.3% for the 4 algorithms, with little interhospital variability; however, algorithm sensitivity varied widely by site. Median PPV was highest for discharge diagnosis of fever alone at 87.7% but ranged from 74.0% to 93.2% across sites. Median PPV for an algorithm of discharge or admission diagnosis of fever was similar (87.4%) but with less variation by site (range 80.0%92.9%) (Table 2).
Outcomes by ICD‐9 Diagnosis Code Group and Case‐Identification Algorithm
When compared with discharge diagnosis of fever, adding admission diagnosis of fever captured a higher proportion of hospitalized infants with SBIs (UTI/pyelonephritis, bacteremia/sepsis, or bacterial meningitis). However, median hospital length of stay, severe outcomes, and 3‐day revisits and revisits with hospitalization did not materially differ when including infants with admission diagnosis of fever in addition to discharge diagnosis of fever. Addition of infants with a diagnosis code for serious infection substantially increased the number of infants with SBIs and severe outcomes but did not capture additional 3‐day revisits (Table 3). There were no additional cases of SBI in the no fever/no serious illness diagnosis code group.
| ICD‐9 Diagnosis Code Algorithm | Outcome | 3‐Day Revisit, % (95% CI) | 3‐Day Revisit With Hospitalization, % (95% CI) | |||
|---|---|---|---|---|---|---|
| Hospitalized, % (95% CI) | UTI/Pyelonephritis, Bacteremia/Sepsis, or Bacterial Meningitis, % (95% CI) | Severe Outcome, % (95% CI)* | Length of Stay in Days, Median (IQR) | |||
| ||||||
| Discharge diagnosis of fever | 44.3 (40.348.4) | 3.3 (1.84.7) | 1.4 (0.42.3) | 3 (23) | 11.7 (8.215.2) | 5.9 (3.38.4) |
| Discharge or admission diagnosis of fever | 52.4 (48.955.9) | 6.1 (4.47.8) | 1.9 (1.02.9) | 3 (23) | 10.9 (7.714.1) | 5.4 (3.17.8) |
| Discharge diagnosis of fever or serious infection | 54.0 (50.457.5) | 15.3 (12.717.8) | 3.8 (2.55.2) | 3 (24) | 11.0 (7.714.2) | 5.5 (3.17.9) |
| Discharge or admission diagnosis of fever or serious infection | 56.5 (53.259.7) | 12.9 (10.715.1) | 3.6 (2.44.8) | 3 (24) | 10.3 (7.313.3) | 5.2 (3.07.4) |
Among infants who met the reference standard for fever but did not have a discharge or admission diagnosis of fever (false negatives), 11.8% had a diagnosis of SBI. Overall, 43.2% of febrile infants (and 84.4% of hospitalized infants) with SBI did not have an ICD‐9 discharge or admission diagnosis of fever. Addition of ICD‐9 diagnosis codes of serious infection to the algorithm of discharge or admission diagnosis of fever captured all additional SBIs, and no false negativeinfants missed with this algorithm had an SBI.
DISCUSSION
We described the performance of 4 ICD‐9 diagnosis code case‐identification algorithms for the identification of febrile young infants <90 days of age at US children's hospitals. Although the specificity was high across algorithms and institutions, the sensitivity was relatively low, particularly for discharge diagnosis of fever, and varied by institution. Given the high specificity, ICD‐9 diagnosis code case‐identification algorithms for fever reliably identify febrile infants using administrative data with low rates of inclusion of infants without fever. However, underidentification of patients, particularly those more prone to SBIs and severe outcomes depending on the algorithm utilized, can impact interpretation of comparative effectiveness studies or the quality of care delivered by an institution.
ICD‐9 discharge diagnosis codes are frequently used to identify pediatric patients across a variety of administrative databases, diseases, and symptoms.[19, 27, 28, 29, 30, 31] Although discharge diagnosis of fever is highly specific, sensitivity is substantially lower than other case‐identification algorithms we studied, particularly for hospitalized infants. This may be due to a fever code sometimes being omitted in favor of a more specific diagnosis (eg, bacteremia) prior to hospital discharge. Therefore, case identification relying only on ICD‐9 discharge diagnosis codes for fever may under‐report clinically important SBI or severe outcomes as demonstrated in our study. This is in contrast to ICD‐9 diagnosis code identification strategies for childhood UTI and pneumonia, which largely have higher sensitivity but lower specificity than fever codes.[13, 14]
Admission diagnosis of fever is important for febrile infants as they may not have an explicit diagnosis at the time of disposition from the ED. Addition of admission diagnosis of fever to an algorithm relying on discharge diagnosis code alone increased sensitivity without a demonstrable reduction in specificity and PPV, likely due to capture of infants with a fever diagnosis at presentation before a specific infection was identified. Although using an algorithm of discharge or admission diagnosis of fever captured a higher percentage of hospitalized febrile infants with SBIs, sensitivity was only 71% overall with this algorithm, and 43% of febrile infants with SBI would still have been missed. Importantly, though, addition of various ICD‐9 codes for serious infection to this algorithm resulted in capture of all febrile infants with SBI and should be used as a sensitivity analysis.
The test characteristics of diagnosis codes were highest in the 29‐ to 56‐days‐old age group. Given the differing low‐risk criteria[6, 7, 8] and lack of best practice guidelines[16] in this age group, the use of administrative data may allow for the comparison of testing and treatment strategies across a large cohort of febrile infants aged 29 to 56 days. However, individual hospital coding practices may affect algorithm performance, in particular sensitivity, which varied substantially by hospital. This variation in algorithm sensitivity may impact comparisons of outcomes across institutions. Therefore, when conducting studies of febrile infants using administrative data, sensitivity analyses or use of chart review should be considered to augment the use of ICD‐9 code‐based identification strategies, particularly for comparative benchmarking and outcomes studies. These additional analyses are particularly important for studies of febrile infants >56 days of age, in whom the sensitivity of diagnosis codes is particularly low. We speculate that the lower sensitivity in older febrile infants may relate to a lack of consensus on the clinical significance of fever in this age group and the varying management strategies employed.[10]
Strengths of this study include the assessment of ICD‐9 code algorithms across multiple institutions for identification of fever in young infants, and the patterns of our findings remained robust when comparing median performance characteristics of the algorithms across hospitals to our overall findings. We were also able to accurately estimate PPV and NPV using a case‐identification strategy weighted to the actual population sizes. Although sensitivity and specificity are the primary measures of test performance, predictive values are highly informative for investigators using administrative data. Additionally, our findings may inform public health efforts including disease surveillance, assessment of seasonal variation, and identification and monitoring of healthcare‐associated infections among febrile infants.
Our study has limitations. We did not review all identified records, which raises the possibility that our evaluated cohort may not be representative of the entire febrile infant population. We attempted to mitigate this possibility by using a random sampling strategy for our population selection that was weighted to the actual population sizes. Second, we identified serious infections using ICD‐9 diagnosis codes determined by group consensus, which may not capture all serious infection codes that identify febrile infants whose fever code was omitted. Third, 47 infants had abnormal temperature that did not meet our reference standard criteria for fever and were included in the no fever group. Although there may be disagreement regarding what constitutes a fever, we used a widely accepted reference standard to define fever.[16] Further, inclusion of these 47 infants as fever did not materially change algorithm performance. Last, our study was conducted at 8 large tertiary‐care children's hospitals, and our results may not be generalizable to other children's hospitals and community‐based hospitals.
CONCLUSIONS
Studies of febrile young infants that rely on ICD‐9 discharge diagnosis code of fever for case ascertainment have high specificity but low sensitivity for the identification of febrile infants, particularly among hospitalized patients. A case‐identification strategy that includes discharge or admission diagnosis of fever demonstrated higher sensitivity, and should be considered for studies of febrile infants using administrative data. However, additional strategies such as incorporation of ICD‐9 codes for serious infection should be used when comparing outcomes across institutions.
Acknowledgements
The Febrile Young Infant Research Collaborative includes the following additional collaborators who are acknowledged for their work on this study: Erica DiLeo, MA, Department of Medical Education and Research, Danbury Hospital, Danbury, Connecticut; Janet Flores, BS, Division of Emergency Medicine, Ann and Robert H. Lurie Children's Hospital of Chicago, Chicago, Illinois.
Disclosures: This project funded in part by The Gerber Foundation Novice Researcher Award, (Ref No. 1827‐3835). Dr. Fran Balamuth received career development support from the National Institutes of Health (NHLBI K12‐HL109009). Funders were not involved in the design or conduct of the study; collection, management, analysis, or interpretation of the data; or preparation, review, or approval of the manuscript. The authors have no conflicts of interest relevant to this article to disclose.
- . The prevalence of serious bacterial infections by age in febrile infants during the first 3 months of life. Pediatr Ann. 1993;22:462–466.
- , , . Performance of low‐risk criteria in the evaluation of young infants with fever: review of the literature. Pediatrics. 2010;125:228–233.
- , , , , , . A week‐by‐week analysis of the low‐risk criteria for serious bacterial infection in febrile neonates. Arch Dis Child. 2009;94:287–292.
- , , , et al. Is 15 days an appropriate cut‐off age for considering serious bacterial infection in the management of febrile infants? Pediatr Infect Dis J. 2012;31:455–458.
- , , . Failure of infant observation scales in detecting serious illness in febrile, 4‐ to 8‐week‐old infants. Pediatrics. 1990;85:1040–1043.
- , , . Outpatient management without antibiotics of fever in selected infants. N Engl J Med. 1993;329:1437–1441.
- , , . Identifying febrile infants at risk for a serious bacterial infection. J Pediatr. 1993;123:489–490.
- , , , et al. Febrile infants at low risk for serious bacterial infection—an appraisal of the Rochester criteria and implications for management. Febrile Infant Collaborative Study Group. Pediatrics. 1994;94:390–396.
- , , , et al. Management of febrile neonates in US pediatric emergency departments. Pediatrics. 2014;133:187–195.
- , , , et al. Variation in care of the febrile young infant <90 days in US pediatric emergency departments. Pediatrics. 2014;134:667–677.
- , , , et al. Association of clinical practice guidelines with emergency department management of febrile infants ≤56 days of age. J Hosp Med. 2015;10:358–365.
- , , , et al. Diagnosis and management of febrile infants (0‐3 months). Evid Rep Technol Assess (Full Rep). 2012;(205):1–297.
- , , , et al. Accuracy of administrative billing codes to detect urinary tract infection hospitalizations. Pediatrics. 2011;128:323–330.
- , , , et al. Identifying pediatric community‐acquired pneumonia hospitalizations: accuracy of administrative billing codes. JAMA Pediatr. 2013;167:851–858.
- , , , , , . Development and use of reporting guidelines for assessing the quality of validation studies of health administrative data. J Clin Epidemiol. 2011;64:821–829.
- American College of Emergency Physicians Clinical Policies Committee; American College of Emergency Physicians Clinical Policies Subcommittee on Pediatric Fever. Clinical policy for children younger than three years presenting to the emergency department with fever. Ann Emerg Med. 2003;42:530–545.
- , , , , , . Variation in occult injury screening for children with suspected abuse in selected US children's hospitals. Pediatrics. 2012;130:853–860.
- . Achieving data quality. How data from a pediatric health information system earns the trust of its users. J AHIMA. 2004;75:22–26.
- , , , . Corticosteroids and mortality in children with bacterial meningitis. JAMA. 2008;299:2048–2055.
- , , , , , . Research electronic data capture (REDCap)—a metadata‐driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42:377–381.
- , , , , , . Deaths attributed to pediatric complex chronic conditions: national trends and implications for supportive care services. Pediatrics. 2001;107:E99.
- US Census Bureau. Geographic terms and concepts—census divisions and census regions. Available at: https://www.census.gov/geo/reference/gtc/gtc_census_divreg.html. Accessed October 20, 2014.
- , , , . Initial emergency department diagnosis and return visits: risk versus perception. Ann Emerg Med. 1998;32:569–573.
- , , , , . A national depiction of children with return visits to the emergency department within 72 hours, 2001–2007. Pediatr Emerg Care. 2012;28:606–610.
- , , , et al. Pediatric observation status: are we overlooking a growing population in children's hospitals? J Hosp Med. 2012;7:530–536.
- , , , et al. Differences in designations of observation care in US freestanding children's hospitals: are they virtual or real? J Hosp Med. 2012;7:287–293.
- , , , , . Trends in the management of viral meningitis at United States children's hospitals. Pediatrics. 2013;131:670–676.
- , , , et al. Impact of increasing ondansetron use on clinical outcomes in children with gastroenteritis. JAMA Pediatr. 2014;168:321–329.
- , , , , . Race, otitis media, and antibiotic selection. Pediatrics. 2014;134:1059–1066.
- , , , et al. Establishing benchmarks for the hospitalized care of children with asthma, bronchiolitis, and pneumonia. Pediatrics. 2014;134:555–562.
- , , , , . Diagnostic testing and treatment of pediatric headache in the emergency department. J Pediatr. 2013;163:1634–1637.
Fever is one of the most common reasons for emergency department (ED) evaluation of infants under 90 days of age.[1] Up to 10% to 20% of febrile young infants will have a serious bacterial infection (SBI),[2, 3, 4] but infants with SBI are difficult to distinguish from those without SBI based upon symptoms and physical examination findings alone.[5] Previously developed clinical prediction algorithms can help to identify febrile infants at low risk for SBI, but differ in age range as well as recommendations for testing and empiric treatment.[6, 7, 8] Consequently, there is widespread variation in management of febrile young infants at US children's hospitals,[9, 10, 11] and defining optimal management strategies remains an important issue in pediatric healthcare.[12] Administrative datasets are convenient and inexpensive, and can be used to evaluate practice variation, trends, and outcomes of a large, diverse group of patients within and across institutions.[9, 10] Accurately identifying febrile infants evaluated for suspected SBI in administrative databases would facilitate comparative effectiveness research, quality improvement initiatives, and institutional benchmarking.
Prior studies have validated the accuracy of administrative billing codes for identification of other common childhood illnesses, including urinary tract infection (UTI)[13] and pneumonia.[14] The accuracy of International Classification of Diseases, Ninth Revision (ICD‐9) diagnosis codes in identifying febrile young infants evaluated for SBI is not known. Reliance on administrative ICD‐9 diagnosis codes for patient identification can lead to misclassification of patients due to variable database quality, the validity of the diagnosis codes being utilized, and hospital coding practices.[15] Additionally, fever is a symptom and not a specific diagnosis. If a particular bacterial or viral diagnosis is established (eg, enterovirus meningitis), a discharge diagnosis of fever may not be attributed to the patient encounter. Thus, evaluating the performance characteristics and capture of clinical outcomes of different combinations of ICD‐9 diagnosis codes for identifying febrile infants is necessary for both the conduct and interpretation of studies that utilize administrative databases. The primary objective of this investigation was to identify the most accurate ICD‐9 coding strategies for the identification of febrile infants aged <90 days using administrative data. We also sought to evaluate capture of clinically important outcomes across identification strategies.
METHODS
Study Design and Setting
For this multicenter retrospective study, we used the Pediatric Health Information System (PHIS) database to identify infants <90 days of age[16] who presented between July 1, 2012 and June 30, 2013 to 1 of 8 EDs. We assessed performance characteristics of ICD‐9 diagnosis code case‐identification algorithms by comparing ICD‐9 code combinations to a fever reference standard determined by medical record review. The institutional review board at each participating site approved the study protocol.
Data Source
Data were obtained from 2 sources: the PHIS database and medical record review. We used the PHIS database to identify eligible patients by ICD‐9 diagnosis codes; patient encounters were randomly selected using a random number generator. The PHIS database contains demographic, diagnosis, and billing data from 44 hospitals affiliated with the Children's Hospital Association (Overland Park, Kansas) and represents 85% of freestanding children's hospitals in the United States.[17] Data are deidentified; encrypted unique patient identifiers permit tracking of patients across visits within a site.[18] The Children's Hospital Association and participating hospitals jointly assure the quality and integrity of the data.[19]
For each patient encounter identified in the PHIS database, detailed medical record review was performed by trained investigators at each of the 8 study sites (see Supporting Information, Appendix, in the online version of this article). A standardized data collection instrument was pilot tested by all investigators prior to use. Data were collected and managed using the Research Electronic Data Capture (REDCap) tool hosted at Boston Children's Hospital.[20]
Exclusions
Using PHIS data, prior to medical record review we excluded infants with a complex chronic condition as defined previously[21] and those transferred from another institution, as these infants may warrant a nonstandard evaluation and/or may have incomplete data.
ICD‐9 Diagnosis Code Groups
In the PHIS database, all patients discharged from the hospital (including hospitalized patients as well as patients discharged from the ED) receive 1 or more ICD‐9 discharge diagnosis codes. These diagnosis codes are ascribed after discharge from the hospital, or for ED patients, after ED discharge. Additionally, patients may receive an admission diagnosis, which reflects the diagnosis ascribed at the time of ED discharge or transfer to the inpatient unit.
We reviewed medical records of infants selected from the following ICD‐9 diagnosis code groups (Figure 1): (1) discharge diagnosis code of fever (780.6 [fever and other physiologic disturbances of temperature regulation], 778.4 [other disturbances of temperature regulation of newborn], 780.60 [fever, unspecified], or 780.61 [fever presenting with conditions classified elsewhere])[9, 10] regardless of the presence of admission diagnosis of fever or diagnosis of serious infection, (2) admission diagnosis code of fever without associated discharge diagnosis code of fever,[10] (3) discharge diagnosis code of serious infection determined a priori (see Supporting Information, Appendix, in the online version of this article) without discharge or admission diagnosis code of fever, and (4) infants without any diagnosis code of fever or serious infection.
Medical records reviewed in each of the 4 ICD‐9 diagnosis code groups were randomly selected from the overall set of ED encounters in the population of infants <90 days of age evaluated during the study period. Twenty‐five percent population sampling was used for 3 of the ICD‐9 diagnosis code groups, whereas 5% sampling was used for the no fever/no serious infection code group. The number of medical records reviewed in each ICD‐9 diagnosis code group was proportional to the distribution of ICD‐9 codes across the entire population of infants <90 days of age. These records were distributed equally across sites (228 records per site), except for 1 site that does not assign admission diagnoses (201 records).
Investigators were blinded to ICD‐9 diagnosis code groups during medical record review. Infants with multiple visits during the study period were eligible to be included more than once if the visits occurred more than 3 days apart. For infants with more than 1 ED visit on a particular calendar day, investigators were instructed to review the initial visit.
For each encounter, we also abstracted demographic characteristics (gender, race/ethnicity), insurance status, hospital region (using US Census categories[22]), and season from the PHIS database.
Reference Standard
The presence of fever was determined by medical record review. We defined fever as any documented temperature 100.4F (38.0C) at home or in the ED.[16]
ICD‐9 Code Case‐Identification Algorithms
Using the aforementioned ICD‐9 diagnosis code groups individually and in combination, the following 4 case‐identification algorithms, determined from prior study or group consensus, were compared to the reference standard: (1) ICD‐9 discharge diagnosis code of fever,[9] (2) ICD‐9 admission or discharge diagnosis code of fever,[10, 11] (3) ICD‐9 discharge diagnosis code of fever or serious infection, and (4) ICD‐9 discharge or admission diagnosis code of fever or serious infection. Algorithms were compared overall, separately for discharged and hospitalized infants, and across 3 distinct age groups (28 days, 2956 days, and 5789 days).
Patient‐Level Outcomes
To compare differences in outcomes by case‐identification algorithm, from the PHIS database we abstracted hospitalization rates, rates of UTI/pyelonephritis,[13] bacteremia/sepsis, and bacterial meningitis.[19] Severe outcomes were defined as intensive care unit admission, mechanical ventilation, central line placement, receipt of extracorporeal membrane oxygenation, or death. We assessed hospital length of stay for admitted infants and 3‐day revisits,[23, 24] and revisits resulting in hospitalization for infants discharged from the ED at the index visit. Patients billed for observation care were classified as being hospitalized.[25, 26]
Data Analysis
Accuracy of the 4 case‐identification algorithms (compared with the reference standard) was calculated using sensitivity, specificity, negative predictive value (NPV), and positive predictive value (PPV), along with 95% confidence interval (CI). Prior to analysis, a 5‐fold weighting factor was applied to the no fever/no serious infection group to account for the differential sampling used for this group (5% vs 25% for the other 3 ICD‐9 diagnosis code groups). This weighting was done to approximate the true prevalence of each ICD‐9 code group within the larger population, so that an accurate rate of false negatives (infants with fever who had neither a diagnosis of fever nor serious infection) could be calculated.
We described continuous variables using median and interquartile range or range values and categorical variables using frequencies with 95% CIs. We compared categorical variables using a 2 test. We determined statistical significance as a 2‐tailed P value <0.05. Statistical analyses were performed using SAS version 9.3 (SAS Institute, Cary, NC).
RESULTS
Study Patients
During the 1‐year study period, 23,753 ED encounters for infants <90 days of age were identified in the PHIS database at the 8 participating sites. Of these infant encounters, 2166 (9.2%) were excluded (1658 infants who had a complex chronic condition and 508 transferred into the ED), leaving 21,587 infants available for selection. After applying our sampling strategy, we identified 1797 encounters for medical record review. Seven encounters from 3 hospitals with missing medical records were excluded, resulting in a final cohort of 1790 encounters (Figure 1). Among included infants, 552 (30.8%) were 28 days, 743 (41.5%) were 29 to 56 days, and 495 (27.8%) were 57 to 89 days of age; 737 (41.2%) infants were hospitalized. Patients differed in age, race, payer, and season across ICD‐9 diagnosis code groups (see Supporting Information, Table 1, in the online version of this article).
| ICD‐9 Diagnosis Code Algorithm | Overall | |||
|---|---|---|---|---|
| Sensitivity, % (95% CI) | Specificity, % (95% CI) | Negative Predictive Value, % (95% CI) | Positive Predictive Value, % (95% CI) | |
| ||||
| Discharge diagnosis of fever | 53.2 (50.056.4) | 98.2 (97.898.6) | 90.8 (90.091.6) | 86.1 (83.388.9) |
| Hospitalized | 47.3 (43.151.5) | 97.7 (96.998.5) | 80.6 (78.682.6) | 90.2 (86.893.6) |
| Discharged from ED | 61.4 (56.666.2) | 98.4 (98.098.8) | 95.4 (94.796.1) | 82.1 (77.786.5) |
| Discharge or admission diagnosis of Fever | 71.1 (68.274.0) | 97.7 (97.398.1) | 94.1 (93.494.8) | 86.9 (84.589.3) |
| Hospitalized | 72.5 (68.876.2) | 97.1 (96.298.0) | 88.8 (87.190.5) | 91.7 (89.194.3) |
| Discharged from ED | 69.2 (64.773.7) | 98.0 (97.598.5) | 96.3 (95.796.9) | 80.8 (76.685.0) |
| Discharge diagnosis of fever or serious infection | 63.7 (60.666.8) | 96.5 (96.097.0) | 92.6 (91.893.4) | 79.6 (76.782.5) |
| Hospitalized | 63.9 (59.967.9) | 92.5 (91.094.0) | 85.1 (83.287.0) | 79.1 (75.382.9) |
| Discharged from ED | 63.4 (58.768.1) | 98.1 (97.698.6) | 95.6 (94.996.3) | 80.2 (75.884.6) |
| Discharge or admission diagnosis of fever or serious infection | 76.6 (73.979.3) | 96.2 (95.696.8) | 95.1 (94.595.7) | 81.0 (78.483.6) |
| Hospitalized | 80.8 (77.584.1) | 92.1 (90.693.6) | 91.5 (89.993.1) | 82.1 (78.985.3) |
| Discharged from ED | 71.0 (66.575.5) | 97.7 (97.298.2) | 96.5 (95.997.1) | 79.4 (75.283.6) |
Among the 1790 patient encounters reviewed, a total of 766 infants (42.8%) met the reference standard definition for fever in the cohort. An additional 47 infants had abnormal temperature reported (documentation of tactile fever, history of fever without a specific temperature described, or hypothermia) but were classified as having no fever by the reference standard.
ICD‐9 Code Case‐Identification Algorithm Performance
Compared with the reference standard, the 4 case‐identification algorithms demonstrated specificity of 96.2% to 98.2% but lower sensitivity overall (Figure 2). Discharge diagnosis of fever alone demonstrated the lowest sensitivity. The algorithm of discharge or admission diagnosis of fever resulted in increased sensitivity and the highest PPV of all 4 algorithms (86.9%, 95% CI: 84.5‐89.3). Addition of serious infection codes to this algorithm resulted in a marginal increase in sensitivity and a similar decrease in PPV (Table 1). When limited to hospitalized infants, specificity was highest for the case‐identification algorithm of discharge diagnosis of fever and similarly high for discharge or admission diagnosis of fever; sensitivity was highest for the algorithm of discharge or admission diagnosis of fever or diagnosis of serious infection. For infants discharged from the ED, algorithm specificity was 97.7% to 98.4%, with lower sensitivity for all 4 algorithms (Table 1). Inclusion of the 47 infants with abnormal temperature as fever did not materially change algorithm performance (data not shown).
Across all 3 age groups (28 days, 2956 days, and 5789 days), the 4 case‐identification algorithms demonstrated specificity >96%, whereas algorithm sensitivity was highest in the 29‐ to 56‐days‐old age group and lowest among infants 57 to 89 days old across all 4 algorithms (Figure 2). Similar to the overall cohort, an algorithm of discharge or admission diagnosis of fever demonstrated specificity of nearly 98% in all age groups; addition of serious infection codes to this algorithm increased sensitivity, highest in the 29‐ to 56‐days‐old age group (Figure 2; see also Supporting Information, Table 2, in the online version of this article).
| ICD‐9 Diagnosis Code Algorithm | Sensitivity, Median % (Range) | Specificity, Median % (Range) | Negative Predictive Value, Median % (Range) | Positive Predictive Value, Median % (Range) |
|---|---|---|---|---|
| ||||
| Discharge diagnosis of fever | 56.2 (34.681.0) | 98.3 (96.499.1) | 92.1 (83.297.4) | 87.7 (74.093.2) |
| Discharge or Admission diagnosis of Fever | 76.7 (51.385.0) | 97.8 (96.298.7) | 95.6 (86.997.4) | 87.4 (80.092.9) |
| Discharge diagnosis of fever or serious infection | 68.3 (44.287.3) | 96.5 (95.498.0) | 93.6 (85.298.2) | 78.3 (74.289.0) |
| Discharge or admission diagnosis of fever or serious infection | 83.1 (58.390.7) | 95.8 (95.498.0) | 96.5 (88.598.2) | 79.1 (77.490.4) |
Across the 8 study sites, median specificity was 95.8% to 98.3% for the 4 algorithms, with little interhospital variability; however, algorithm sensitivity varied widely by site. Median PPV was highest for discharge diagnosis of fever alone at 87.7% but ranged from 74.0% to 93.2% across sites. Median PPV for an algorithm of discharge or admission diagnosis of fever was similar (87.4%) but with less variation by site (range 80.0%92.9%) (Table 2).
Outcomes by ICD‐9 Diagnosis Code Group and Case‐Identification Algorithm
When compared with discharge diagnosis of fever, adding admission diagnosis of fever captured a higher proportion of hospitalized infants with SBIs (UTI/pyelonephritis, bacteremia/sepsis, or bacterial meningitis). However, median hospital length of stay, severe outcomes, and 3‐day revisits and revisits with hospitalization did not materially differ when including infants with admission diagnosis of fever in addition to discharge diagnosis of fever. Addition of infants with a diagnosis code for serious infection substantially increased the number of infants with SBIs and severe outcomes but did not capture additional 3‐day revisits (Table 3). There were no additional cases of SBI in the no fever/no serious illness diagnosis code group.
| ICD‐9 Diagnosis Code Algorithm | Outcome | 3‐Day Revisit, % (95% CI) | 3‐Day Revisit With Hospitalization, % (95% CI) | |||
|---|---|---|---|---|---|---|
| Hospitalized, % (95% CI) | UTI/Pyelonephritis, Bacteremia/Sepsis, or Bacterial Meningitis, % (95% CI) | Severe Outcome, % (95% CI)* | Length of Stay in Days, Median (IQR) | |||
| ||||||
| Discharge diagnosis of fever | 44.3 (40.348.4) | 3.3 (1.84.7) | 1.4 (0.42.3) | 3 (23) | 11.7 (8.215.2) | 5.9 (3.38.4) |
| Discharge or admission diagnosis of fever | 52.4 (48.955.9) | 6.1 (4.47.8) | 1.9 (1.02.9) | 3 (23) | 10.9 (7.714.1) | 5.4 (3.17.8) |
| Discharge diagnosis of fever or serious infection | 54.0 (50.457.5) | 15.3 (12.717.8) | 3.8 (2.55.2) | 3 (24) | 11.0 (7.714.2) | 5.5 (3.17.9) |
| Discharge or admission diagnosis of fever or serious infection | 56.5 (53.259.7) | 12.9 (10.715.1) | 3.6 (2.44.8) | 3 (24) | 10.3 (7.313.3) | 5.2 (3.07.4) |
Among infants who met the reference standard for fever but did not have a discharge or admission diagnosis of fever (false negatives), 11.8% had a diagnosis of SBI. Overall, 43.2% of febrile infants (and 84.4% of hospitalized infants) with SBI did not have an ICD‐9 discharge or admission diagnosis of fever. Addition of ICD‐9 diagnosis codes of serious infection to the algorithm of discharge or admission diagnosis of fever captured all additional SBIs, and no false negativeinfants missed with this algorithm had an SBI.
DISCUSSION
We described the performance of 4 ICD‐9 diagnosis code case‐identification algorithms for the identification of febrile young infants <90 days of age at US children's hospitals. Although the specificity was high across algorithms and institutions, the sensitivity was relatively low, particularly for discharge diagnosis of fever, and varied by institution. Given the high specificity, ICD‐9 diagnosis code case‐identification algorithms for fever reliably identify febrile infants using administrative data with low rates of inclusion of infants without fever. However, underidentification of patients, particularly those more prone to SBIs and severe outcomes depending on the algorithm utilized, can impact interpretation of comparative effectiveness studies or the quality of care delivered by an institution.
ICD‐9 discharge diagnosis codes are frequently used to identify pediatric patients across a variety of administrative databases, diseases, and symptoms.[19, 27, 28, 29, 30, 31] Although discharge diagnosis of fever is highly specific, sensitivity is substantially lower than other case‐identification algorithms we studied, particularly for hospitalized infants. This may be due to a fever code sometimes being omitted in favor of a more specific diagnosis (eg, bacteremia) prior to hospital discharge. Therefore, case identification relying only on ICD‐9 discharge diagnosis codes for fever may under‐report clinically important SBI or severe outcomes as demonstrated in our study. This is in contrast to ICD‐9 diagnosis code identification strategies for childhood UTI and pneumonia, which largely have higher sensitivity but lower specificity than fever codes.[13, 14]
Admission diagnosis of fever is important for febrile infants as they may not have an explicit diagnosis at the time of disposition from the ED. Addition of admission diagnosis of fever to an algorithm relying on discharge diagnosis code alone increased sensitivity without a demonstrable reduction in specificity and PPV, likely due to capture of infants with a fever diagnosis at presentation before a specific infection was identified. Although using an algorithm of discharge or admission diagnosis of fever captured a higher percentage of hospitalized febrile infants with SBIs, sensitivity was only 71% overall with this algorithm, and 43% of febrile infants with SBI would still have been missed. Importantly, though, addition of various ICD‐9 codes for serious infection to this algorithm resulted in capture of all febrile infants with SBI and should be used as a sensitivity analysis.
The test characteristics of diagnosis codes were highest in the 29‐ to 56‐days‐old age group. Given the differing low‐risk criteria[6, 7, 8] and lack of best practice guidelines[16] in this age group, the use of administrative data may allow for the comparison of testing and treatment strategies across a large cohort of febrile infants aged 29 to 56 days. However, individual hospital coding practices may affect algorithm performance, in particular sensitivity, which varied substantially by hospital. This variation in algorithm sensitivity may impact comparisons of outcomes across institutions. Therefore, when conducting studies of febrile infants using administrative data, sensitivity analyses or use of chart review should be considered to augment the use of ICD‐9 code‐based identification strategies, particularly for comparative benchmarking and outcomes studies. These additional analyses are particularly important for studies of febrile infants >56 days of age, in whom the sensitivity of diagnosis codes is particularly low. We speculate that the lower sensitivity in older febrile infants may relate to a lack of consensus on the clinical significance of fever in this age group and the varying management strategies employed.[10]
Strengths of this study include the assessment of ICD‐9 code algorithms across multiple institutions for identification of fever in young infants, and the patterns of our findings remained robust when comparing median performance characteristics of the algorithms across hospitals to our overall findings. We were also able to accurately estimate PPV and NPV using a case‐identification strategy weighted to the actual population sizes. Although sensitivity and specificity are the primary measures of test performance, predictive values are highly informative for investigators using administrative data. Additionally, our findings may inform public health efforts including disease surveillance, assessment of seasonal variation, and identification and monitoring of healthcare‐associated infections among febrile infants.
Our study has limitations. We did not review all identified records, which raises the possibility that our evaluated cohort may not be representative of the entire febrile infant population. We attempted to mitigate this possibility by using a random sampling strategy for our population selection that was weighted to the actual population sizes. Second, we identified serious infections using ICD‐9 diagnosis codes determined by group consensus, which may not capture all serious infection codes that identify febrile infants whose fever code was omitted. Third, 47 infants had abnormal temperature that did not meet our reference standard criteria for fever and were included in the no fever group. Although there may be disagreement regarding what constitutes a fever, we used a widely accepted reference standard to define fever.[16] Further, inclusion of these 47 infants as fever did not materially change algorithm performance. Last, our study was conducted at 8 large tertiary‐care children's hospitals, and our results may not be generalizable to other children's hospitals and community‐based hospitals.
CONCLUSIONS
Studies of febrile young infants that rely on ICD‐9 discharge diagnosis code of fever for case ascertainment have high specificity but low sensitivity for the identification of febrile infants, particularly among hospitalized patients. A case‐identification strategy that includes discharge or admission diagnosis of fever demonstrated higher sensitivity, and should be considered for studies of febrile infants using administrative data. However, additional strategies such as incorporation of ICD‐9 codes for serious infection should be used when comparing outcomes across institutions.
Acknowledgements
The Febrile Young Infant Research Collaborative includes the following additional collaborators who are acknowledged for their work on this study: Erica DiLeo, MA, Department of Medical Education and Research, Danbury Hospital, Danbury, Connecticut; Janet Flores, BS, Division of Emergency Medicine, Ann and Robert H. Lurie Children's Hospital of Chicago, Chicago, Illinois.
Disclosures: This project funded in part by The Gerber Foundation Novice Researcher Award, (Ref No. 1827‐3835). Dr. Fran Balamuth received career development support from the National Institutes of Health (NHLBI K12‐HL109009). Funders were not involved in the design or conduct of the study; collection, management, analysis, or interpretation of the data; or preparation, review, or approval of the manuscript. The authors have no conflicts of interest relevant to this article to disclose.
Fever is one of the most common reasons for emergency department (ED) evaluation of infants under 90 days of age.[1] Up to 10% to 20% of febrile young infants will have a serious bacterial infection (SBI),[2, 3, 4] but infants with SBI are difficult to distinguish from those without SBI based upon symptoms and physical examination findings alone.[5] Previously developed clinical prediction algorithms can help to identify febrile infants at low risk for SBI, but differ in age range as well as recommendations for testing and empiric treatment.[6, 7, 8] Consequently, there is widespread variation in management of febrile young infants at US children's hospitals,[9, 10, 11] and defining optimal management strategies remains an important issue in pediatric healthcare.[12] Administrative datasets are convenient and inexpensive, and can be used to evaluate practice variation, trends, and outcomes of a large, diverse group of patients within and across institutions.[9, 10] Accurately identifying febrile infants evaluated for suspected SBI in administrative databases would facilitate comparative effectiveness research, quality improvement initiatives, and institutional benchmarking.
Prior studies have validated the accuracy of administrative billing codes for identification of other common childhood illnesses, including urinary tract infection (UTI)[13] and pneumonia.[14] The accuracy of International Classification of Diseases, Ninth Revision (ICD‐9) diagnosis codes in identifying febrile young infants evaluated for SBI is not known. Reliance on administrative ICD‐9 diagnosis codes for patient identification can lead to misclassification of patients due to variable database quality, the validity of the diagnosis codes being utilized, and hospital coding practices.[15] Additionally, fever is a symptom and not a specific diagnosis. If a particular bacterial or viral diagnosis is established (eg, enterovirus meningitis), a discharge diagnosis of fever may not be attributed to the patient encounter. Thus, evaluating the performance characteristics and capture of clinical outcomes of different combinations of ICD‐9 diagnosis codes for identifying febrile infants is necessary for both the conduct and interpretation of studies that utilize administrative databases. The primary objective of this investigation was to identify the most accurate ICD‐9 coding strategies for the identification of febrile infants aged <90 days using administrative data. We also sought to evaluate capture of clinically important outcomes across identification strategies.
METHODS
Study Design and Setting
For this multicenter retrospective study, we used the Pediatric Health Information System (PHIS) database to identify infants <90 days of age[16] who presented between July 1, 2012 and June 30, 2013 to 1 of 8 EDs. We assessed performance characteristics of ICD‐9 diagnosis code case‐identification algorithms by comparing ICD‐9 code combinations to a fever reference standard determined by medical record review. The institutional review board at each participating site approved the study protocol.
Data Source
Data were obtained from 2 sources: the PHIS database and medical record review. We used the PHIS database to identify eligible patients by ICD‐9 diagnosis codes; patient encounters were randomly selected using a random number generator. The PHIS database contains demographic, diagnosis, and billing data from 44 hospitals affiliated with the Children's Hospital Association (Overland Park, Kansas) and represents 85% of freestanding children's hospitals in the United States.[17] Data are deidentified; encrypted unique patient identifiers permit tracking of patients across visits within a site.[18] The Children's Hospital Association and participating hospitals jointly assure the quality and integrity of the data.[19]
For each patient encounter identified in the PHIS database, detailed medical record review was performed by trained investigators at each of the 8 study sites (see Supporting Information, Appendix, in the online version of this article). A standardized data collection instrument was pilot tested by all investigators prior to use. Data were collected and managed using the Research Electronic Data Capture (REDCap) tool hosted at Boston Children's Hospital.[20]
Exclusions
Using PHIS data, prior to medical record review we excluded infants with a complex chronic condition as defined previously[21] and those transferred from another institution, as these infants may warrant a nonstandard evaluation and/or may have incomplete data.
ICD‐9 Diagnosis Code Groups
In the PHIS database, all patients discharged from the hospital (including hospitalized patients as well as patients discharged from the ED) receive 1 or more ICD‐9 discharge diagnosis codes. These diagnosis codes are ascribed after discharge from the hospital, or for ED patients, after ED discharge. Additionally, patients may receive an admission diagnosis, which reflects the diagnosis ascribed at the time of ED discharge or transfer to the inpatient unit.
We reviewed medical records of infants selected from the following ICD‐9 diagnosis code groups (Figure 1): (1) discharge diagnosis code of fever (780.6 [fever and other physiologic disturbances of temperature regulation], 778.4 [other disturbances of temperature regulation of newborn], 780.60 [fever, unspecified], or 780.61 [fever presenting with conditions classified elsewhere])[9, 10] regardless of the presence of admission diagnosis of fever or diagnosis of serious infection, (2) admission diagnosis code of fever without associated discharge diagnosis code of fever,[10] (3) discharge diagnosis code of serious infection determined a priori (see Supporting Information, Appendix, in the online version of this article) without discharge or admission diagnosis code of fever, and (4) infants without any diagnosis code of fever or serious infection.
Medical records reviewed in each of the 4 ICD‐9 diagnosis code groups were randomly selected from the overall set of ED encounters in the population of infants <90 days of age evaluated during the study period. Twenty‐five percent population sampling was used for 3 of the ICD‐9 diagnosis code groups, whereas 5% sampling was used for the no fever/no serious infection code group. The number of medical records reviewed in each ICD‐9 diagnosis code group was proportional to the distribution of ICD‐9 codes across the entire population of infants <90 days of age. These records were distributed equally across sites (228 records per site), except for 1 site that does not assign admission diagnoses (201 records).
Investigators were blinded to ICD‐9 diagnosis code groups during medical record review. Infants with multiple visits during the study period were eligible to be included more than once if the visits occurred more than 3 days apart. For infants with more than 1 ED visit on a particular calendar day, investigators were instructed to review the initial visit.
For each encounter, we also abstracted demographic characteristics (gender, race/ethnicity), insurance status, hospital region (using US Census categories[22]), and season from the PHIS database.
Reference Standard
The presence of fever was determined by medical record review. We defined fever as any documented temperature 100.4F (38.0C) at home or in the ED.[16]
ICD‐9 Code Case‐Identification Algorithms
Using the aforementioned ICD‐9 diagnosis code groups individually and in combination, the following 4 case‐identification algorithms, determined from prior study or group consensus, were compared to the reference standard: (1) ICD‐9 discharge diagnosis code of fever,[9] (2) ICD‐9 admission or discharge diagnosis code of fever,[10, 11] (3) ICD‐9 discharge diagnosis code of fever or serious infection, and (4) ICD‐9 discharge or admission diagnosis code of fever or serious infection. Algorithms were compared overall, separately for discharged and hospitalized infants, and across 3 distinct age groups (28 days, 2956 days, and 5789 days).
Patient‐Level Outcomes
To compare differences in outcomes by case‐identification algorithm, from the PHIS database we abstracted hospitalization rates, rates of UTI/pyelonephritis,[13] bacteremia/sepsis, and bacterial meningitis.[19] Severe outcomes were defined as intensive care unit admission, mechanical ventilation, central line placement, receipt of extracorporeal membrane oxygenation, or death. We assessed hospital length of stay for admitted infants and 3‐day revisits,[23, 24] and revisits resulting in hospitalization for infants discharged from the ED at the index visit. Patients billed for observation care were classified as being hospitalized.[25, 26]
Data Analysis
Accuracy of the 4 case‐identification algorithms (compared with the reference standard) was calculated using sensitivity, specificity, negative predictive value (NPV), and positive predictive value (PPV), along with 95% confidence interval (CI). Prior to analysis, a 5‐fold weighting factor was applied to the no fever/no serious infection group to account for the differential sampling used for this group (5% vs 25% for the other 3 ICD‐9 diagnosis code groups). This weighting was done to approximate the true prevalence of each ICD‐9 code group within the larger population, so that an accurate rate of false negatives (infants with fever who had neither a diagnosis of fever nor serious infection) could be calculated.
We described continuous variables using median and interquartile range or range values and categorical variables using frequencies with 95% CIs. We compared categorical variables using a 2 test. We determined statistical significance as a 2‐tailed P value <0.05. Statistical analyses were performed using SAS version 9.3 (SAS Institute, Cary, NC).
RESULTS
Study Patients
During the 1‐year study period, 23,753 ED encounters for infants <90 days of age were identified in the PHIS database at the 8 participating sites. Of these infant encounters, 2166 (9.2%) were excluded (1658 infants who had a complex chronic condition and 508 transferred into the ED), leaving 21,587 infants available for selection. After applying our sampling strategy, we identified 1797 encounters for medical record review. Seven encounters from 3 hospitals with missing medical records were excluded, resulting in a final cohort of 1790 encounters (Figure 1). Among included infants, 552 (30.8%) were 28 days, 743 (41.5%) were 29 to 56 days, and 495 (27.8%) were 57 to 89 days of age; 737 (41.2%) infants were hospitalized. Patients differed in age, race, payer, and season across ICD‐9 diagnosis code groups (see Supporting Information, Table 1, in the online version of this article).
| ICD‐9 Diagnosis Code Algorithm | Overall | |||
|---|---|---|---|---|
| Sensitivity, % (95% CI) | Specificity, % (95% CI) | Negative Predictive Value, % (95% CI) | Positive Predictive Value, % (95% CI) | |
| ||||
| Discharge diagnosis of fever | 53.2 (50.056.4) | 98.2 (97.898.6) | 90.8 (90.091.6) | 86.1 (83.388.9) |
| Hospitalized | 47.3 (43.151.5) | 97.7 (96.998.5) | 80.6 (78.682.6) | 90.2 (86.893.6) |
| Discharged from ED | 61.4 (56.666.2) | 98.4 (98.098.8) | 95.4 (94.796.1) | 82.1 (77.786.5) |
| Discharge or admission diagnosis of Fever | 71.1 (68.274.0) | 97.7 (97.398.1) | 94.1 (93.494.8) | 86.9 (84.589.3) |
| Hospitalized | 72.5 (68.876.2) | 97.1 (96.298.0) | 88.8 (87.190.5) | 91.7 (89.194.3) |
| Discharged from ED | 69.2 (64.773.7) | 98.0 (97.598.5) | 96.3 (95.796.9) | 80.8 (76.685.0) |
| Discharge diagnosis of fever or serious infection | 63.7 (60.666.8) | 96.5 (96.097.0) | 92.6 (91.893.4) | 79.6 (76.782.5) |
| Hospitalized | 63.9 (59.967.9) | 92.5 (91.094.0) | 85.1 (83.287.0) | 79.1 (75.382.9) |
| Discharged from ED | 63.4 (58.768.1) | 98.1 (97.698.6) | 95.6 (94.996.3) | 80.2 (75.884.6) |
| Discharge or admission diagnosis of fever or serious infection | 76.6 (73.979.3) | 96.2 (95.696.8) | 95.1 (94.595.7) | 81.0 (78.483.6) |
| Hospitalized | 80.8 (77.584.1) | 92.1 (90.693.6) | 91.5 (89.993.1) | 82.1 (78.985.3) |
| Discharged from ED | 71.0 (66.575.5) | 97.7 (97.298.2) | 96.5 (95.997.1) | 79.4 (75.283.6) |
Among the 1790 patient encounters reviewed, a total of 766 infants (42.8%) met the reference standard definition for fever in the cohort. An additional 47 infants had abnormal temperature reported (documentation of tactile fever, history of fever without a specific temperature described, or hypothermia) but were classified as having no fever by the reference standard.
ICD‐9 Code Case‐Identification Algorithm Performance
Compared with the reference standard, the 4 case‐identification algorithms demonstrated specificity of 96.2% to 98.2% but lower sensitivity overall (Figure 2). Discharge diagnosis of fever alone demonstrated the lowest sensitivity. The algorithm of discharge or admission diagnosis of fever resulted in increased sensitivity and the highest PPV of all 4 algorithms (86.9%, 95% CI: 84.5‐89.3). Addition of serious infection codes to this algorithm resulted in a marginal increase in sensitivity and a similar decrease in PPV (Table 1). When limited to hospitalized infants, specificity was highest for the case‐identification algorithm of discharge diagnosis of fever and similarly high for discharge or admission diagnosis of fever; sensitivity was highest for the algorithm of discharge or admission diagnosis of fever or diagnosis of serious infection. For infants discharged from the ED, algorithm specificity was 97.7% to 98.4%, with lower sensitivity for all 4 algorithms (Table 1). Inclusion of the 47 infants with abnormal temperature as fever did not materially change algorithm performance (data not shown).
Across all 3 age groups (28 days, 2956 days, and 5789 days), the 4 case‐identification algorithms demonstrated specificity >96%, whereas algorithm sensitivity was highest in the 29‐ to 56‐days‐old age group and lowest among infants 57 to 89 days old across all 4 algorithms (Figure 2). Similar to the overall cohort, an algorithm of discharge or admission diagnosis of fever demonstrated specificity of nearly 98% in all age groups; addition of serious infection codes to this algorithm increased sensitivity, highest in the 29‐ to 56‐days‐old age group (Figure 2; see also Supporting Information, Table 2, in the online version of this article).
| ICD‐9 Diagnosis Code Algorithm | Sensitivity, Median % (Range) | Specificity, Median % (Range) | Negative Predictive Value, Median % (Range) | Positive Predictive Value, Median % (Range) |
|---|---|---|---|---|
| ||||
| Discharge diagnosis of fever | 56.2 (34.681.0) | 98.3 (96.499.1) | 92.1 (83.297.4) | 87.7 (74.093.2) |
| Discharge or Admission diagnosis of Fever | 76.7 (51.385.0) | 97.8 (96.298.7) | 95.6 (86.997.4) | 87.4 (80.092.9) |
| Discharge diagnosis of fever or serious infection | 68.3 (44.287.3) | 96.5 (95.498.0) | 93.6 (85.298.2) | 78.3 (74.289.0) |
| Discharge or admission diagnosis of fever or serious infection | 83.1 (58.390.7) | 95.8 (95.498.0) | 96.5 (88.598.2) | 79.1 (77.490.4) |
Across the 8 study sites, median specificity was 95.8% to 98.3% for the 4 algorithms, with little interhospital variability; however, algorithm sensitivity varied widely by site. Median PPV was highest for discharge diagnosis of fever alone at 87.7% but ranged from 74.0% to 93.2% across sites. Median PPV for an algorithm of discharge or admission diagnosis of fever was similar (87.4%) but with less variation by site (range 80.0%92.9%) (Table 2).
Outcomes by ICD‐9 Diagnosis Code Group and Case‐Identification Algorithm
When compared with discharge diagnosis of fever, adding admission diagnosis of fever captured a higher proportion of hospitalized infants with SBIs (UTI/pyelonephritis, bacteremia/sepsis, or bacterial meningitis). However, median hospital length of stay, severe outcomes, and 3‐day revisits and revisits with hospitalization did not materially differ when including infants with admission diagnosis of fever in addition to discharge diagnosis of fever. Addition of infants with a diagnosis code for serious infection substantially increased the number of infants with SBIs and severe outcomes but did not capture additional 3‐day revisits (Table 3). There were no additional cases of SBI in the no fever/no serious illness diagnosis code group.
| ICD‐9 Diagnosis Code Algorithm | Outcome | 3‐Day Revisit, % (95% CI) | 3‐Day Revisit With Hospitalization, % (95% CI) | |||
|---|---|---|---|---|---|---|
| Hospitalized, % (95% CI) | UTI/Pyelonephritis, Bacteremia/Sepsis, or Bacterial Meningitis, % (95% CI) | Severe Outcome, % (95% CI)* | Length of Stay in Days, Median (IQR) | |||
| ||||||
| Discharge diagnosis of fever | 44.3 (40.348.4) | 3.3 (1.84.7) | 1.4 (0.42.3) | 3 (23) | 11.7 (8.215.2) | 5.9 (3.38.4) |
| Discharge or admission diagnosis of fever | 52.4 (48.955.9) | 6.1 (4.47.8) | 1.9 (1.02.9) | 3 (23) | 10.9 (7.714.1) | 5.4 (3.17.8) |
| Discharge diagnosis of fever or serious infection | 54.0 (50.457.5) | 15.3 (12.717.8) | 3.8 (2.55.2) | 3 (24) | 11.0 (7.714.2) | 5.5 (3.17.9) |
| Discharge or admission diagnosis of fever or serious infection | 56.5 (53.259.7) | 12.9 (10.715.1) | 3.6 (2.44.8) | 3 (24) | 10.3 (7.313.3) | 5.2 (3.07.4) |
Among infants who met the reference standard for fever but did not have a discharge or admission diagnosis of fever (false negatives), 11.8% had a diagnosis of SBI. Overall, 43.2% of febrile infants (and 84.4% of hospitalized infants) with SBI did not have an ICD‐9 discharge or admission diagnosis of fever. Addition of ICD‐9 diagnosis codes of serious infection to the algorithm of discharge or admission diagnosis of fever captured all additional SBIs, and no false negativeinfants missed with this algorithm had an SBI.
DISCUSSION
We described the performance of 4 ICD‐9 diagnosis code case‐identification algorithms for the identification of febrile young infants <90 days of age at US children's hospitals. Although the specificity was high across algorithms and institutions, the sensitivity was relatively low, particularly for discharge diagnosis of fever, and varied by institution. Given the high specificity, ICD‐9 diagnosis code case‐identification algorithms for fever reliably identify febrile infants using administrative data with low rates of inclusion of infants without fever. However, underidentification of patients, particularly those more prone to SBIs and severe outcomes depending on the algorithm utilized, can impact interpretation of comparative effectiveness studies or the quality of care delivered by an institution.
ICD‐9 discharge diagnosis codes are frequently used to identify pediatric patients across a variety of administrative databases, diseases, and symptoms.[19, 27, 28, 29, 30, 31] Although discharge diagnosis of fever is highly specific, sensitivity is substantially lower than other case‐identification algorithms we studied, particularly for hospitalized infants. This may be due to a fever code sometimes being omitted in favor of a more specific diagnosis (eg, bacteremia) prior to hospital discharge. Therefore, case identification relying only on ICD‐9 discharge diagnosis codes for fever may under‐report clinically important SBI or severe outcomes as demonstrated in our study. This is in contrast to ICD‐9 diagnosis code identification strategies for childhood UTI and pneumonia, which largely have higher sensitivity but lower specificity than fever codes.[13, 14]
Admission diagnosis of fever is important for febrile infants as they may not have an explicit diagnosis at the time of disposition from the ED. Addition of admission diagnosis of fever to an algorithm relying on discharge diagnosis code alone increased sensitivity without a demonstrable reduction in specificity and PPV, likely due to capture of infants with a fever diagnosis at presentation before a specific infection was identified. Although using an algorithm of discharge or admission diagnosis of fever captured a higher percentage of hospitalized febrile infants with SBIs, sensitivity was only 71% overall with this algorithm, and 43% of febrile infants with SBI would still have been missed. Importantly, though, addition of various ICD‐9 codes for serious infection to this algorithm resulted in capture of all febrile infants with SBI and should be used as a sensitivity analysis.
The test characteristics of diagnosis codes were highest in the 29‐ to 56‐days‐old age group. Given the differing low‐risk criteria[6, 7, 8] and lack of best practice guidelines[16] in this age group, the use of administrative data may allow for the comparison of testing and treatment strategies across a large cohort of febrile infants aged 29 to 56 days. However, individual hospital coding practices may affect algorithm performance, in particular sensitivity, which varied substantially by hospital. This variation in algorithm sensitivity may impact comparisons of outcomes across institutions. Therefore, when conducting studies of febrile infants using administrative data, sensitivity analyses or use of chart review should be considered to augment the use of ICD‐9 code‐based identification strategies, particularly for comparative benchmarking and outcomes studies. These additional analyses are particularly important for studies of febrile infants >56 days of age, in whom the sensitivity of diagnosis codes is particularly low. We speculate that the lower sensitivity in older febrile infants may relate to a lack of consensus on the clinical significance of fever in this age group and the varying management strategies employed.[10]
Strengths of this study include the assessment of ICD‐9 code algorithms across multiple institutions for identification of fever in young infants, and the patterns of our findings remained robust when comparing median performance characteristics of the algorithms across hospitals to our overall findings. We were also able to accurately estimate PPV and NPV using a case‐identification strategy weighted to the actual population sizes. Although sensitivity and specificity are the primary measures of test performance, predictive values are highly informative for investigators using administrative data. Additionally, our findings may inform public health efforts including disease surveillance, assessment of seasonal variation, and identification and monitoring of healthcare‐associated infections among febrile infants.
Our study has limitations. We did not review all identified records, which raises the possibility that our evaluated cohort may not be representative of the entire febrile infant population. We attempted to mitigate this possibility by using a random sampling strategy for our population selection that was weighted to the actual population sizes. Second, we identified serious infections using ICD‐9 diagnosis codes determined by group consensus, which may not capture all serious infection codes that identify febrile infants whose fever code was omitted. Third, 47 infants had abnormal temperature that did not meet our reference standard criteria for fever and were included in the no fever group. Although there may be disagreement regarding what constitutes a fever, we used a widely accepted reference standard to define fever.[16] Further, inclusion of these 47 infants as fever did not materially change algorithm performance. Last, our study was conducted at 8 large tertiary‐care children's hospitals, and our results may not be generalizable to other children's hospitals and community‐based hospitals.
CONCLUSIONS
Studies of febrile young infants that rely on ICD‐9 discharge diagnosis code of fever for case ascertainment have high specificity but low sensitivity for the identification of febrile infants, particularly among hospitalized patients. A case‐identification strategy that includes discharge or admission diagnosis of fever demonstrated higher sensitivity, and should be considered for studies of febrile infants using administrative data. However, additional strategies such as incorporation of ICD‐9 codes for serious infection should be used when comparing outcomes across institutions.
Acknowledgements
The Febrile Young Infant Research Collaborative includes the following additional collaborators who are acknowledged for their work on this study: Erica DiLeo, MA, Department of Medical Education and Research, Danbury Hospital, Danbury, Connecticut; Janet Flores, BS, Division of Emergency Medicine, Ann and Robert H. Lurie Children's Hospital of Chicago, Chicago, Illinois.
Disclosures: This project funded in part by The Gerber Foundation Novice Researcher Award, (Ref No. 1827‐3835). Dr. Fran Balamuth received career development support from the National Institutes of Health (NHLBI K12‐HL109009). Funders were not involved in the design or conduct of the study; collection, management, analysis, or interpretation of the data; or preparation, review, or approval of the manuscript. The authors have no conflicts of interest relevant to this article to disclose.
- . The prevalence of serious bacterial infections by age in febrile infants during the first 3 months of life. Pediatr Ann. 1993;22:462–466.
- , , . Performance of low‐risk criteria in the evaluation of young infants with fever: review of the literature. Pediatrics. 2010;125:228–233.
- , , , , , . A week‐by‐week analysis of the low‐risk criteria for serious bacterial infection in febrile neonates. Arch Dis Child. 2009;94:287–292.
- , , , et al. Is 15 days an appropriate cut‐off age for considering serious bacterial infection in the management of febrile infants? Pediatr Infect Dis J. 2012;31:455–458.
- , , . Failure of infant observation scales in detecting serious illness in febrile, 4‐ to 8‐week‐old infants. Pediatrics. 1990;85:1040–1043.
- , , . Outpatient management without antibiotics of fever in selected infants. N Engl J Med. 1993;329:1437–1441.
- , , . Identifying febrile infants at risk for a serious bacterial infection. J Pediatr. 1993;123:489–490.
- , , , et al. Febrile infants at low risk for serious bacterial infection—an appraisal of the Rochester criteria and implications for management. Febrile Infant Collaborative Study Group. Pediatrics. 1994;94:390–396.
- , , , et al. Management of febrile neonates in US pediatric emergency departments. Pediatrics. 2014;133:187–195.
- , , , et al. Variation in care of the febrile young infant <90 days in US pediatric emergency departments. Pediatrics. 2014;134:667–677.
- , , , et al. Association of clinical practice guidelines with emergency department management of febrile infants ≤56 days of age. J Hosp Med. 2015;10:358–365.
- , , , et al. Diagnosis and management of febrile infants (0‐3 months). Evid Rep Technol Assess (Full Rep). 2012;(205):1–297.
- , , , et al. Accuracy of administrative billing codes to detect urinary tract infection hospitalizations. Pediatrics. 2011;128:323–330.
- , , , et al. Identifying pediatric community‐acquired pneumonia hospitalizations: accuracy of administrative billing codes. JAMA Pediatr. 2013;167:851–858.
- , , , , , . Development and use of reporting guidelines for assessing the quality of validation studies of health administrative data. J Clin Epidemiol. 2011;64:821–829.
- American College of Emergency Physicians Clinical Policies Committee; American College of Emergency Physicians Clinical Policies Subcommittee on Pediatric Fever. Clinical policy for children younger than three years presenting to the emergency department with fever. Ann Emerg Med. 2003;42:530–545.
- , , , , , . Variation in occult injury screening for children with suspected abuse in selected US children's hospitals. Pediatrics. 2012;130:853–860.
- . Achieving data quality. How data from a pediatric health information system earns the trust of its users. J AHIMA. 2004;75:22–26.
- , , , . Corticosteroids and mortality in children with bacterial meningitis. JAMA. 2008;299:2048–2055.
- , , , , , . Research electronic data capture (REDCap)—a metadata‐driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42:377–381.
- , , , , , . Deaths attributed to pediatric complex chronic conditions: national trends and implications for supportive care services. Pediatrics. 2001;107:E99.
- US Census Bureau. Geographic terms and concepts—census divisions and census regions. Available at: https://www.census.gov/geo/reference/gtc/gtc_census_divreg.html. Accessed October 20, 2014.
- , , , . Initial emergency department diagnosis and return visits: risk versus perception. Ann Emerg Med. 1998;32:569–573.
- , , , , . A national depiction of children with return visits to the emergency department within 72 hours, 2001–2007. Pediatr Emerg Care. 2012;28:606–610.
- , , , et al. Pediatric observation status: are we overlooking a growing population in children's hospitals? J Hosp Med. 2012;7:530–536.
- , , , et al. Differences in designations of observation care in US freestanding children's hospitals: are they virtual or real? J Hosp Med. 2012;7:287–293.
- , , , , . Trends in the management of viral meningitis at United States children's hospitals. Pediatrics. 2013;131:670–676.
- , , , et al. Impact of increasing ondansetron use on clinical outcomes in children with gastroenteritis. JAMA Pediatr. 2014;168:321–329.
- , , , , . Race, otitis media, and antibiotic selection. Pediatrics. 2014;134:1059–1066.
- , , , et al. Establishing benchmarks for the hospitalized care of children with asthma, bronchiolitis, and pneumonia. Pediatrics. 2014;134:555–562.
- , , , , . Diagnostic testing and treatment of pediatric headache in the emergency department. J Pediatr. 2013;163:1634–1637.
- . The prevalence of serious bacterial infections by age in febrile infants during the first 3 months of life. Pediatr Ann. 1993;22:462–466.
- , , . Performance of low‐risk criteria in the evaluation of young infants with fever: review of the literature. Pediatrics. 2010;125:228–233.
- , , , , , . A week‐by‐week analysis of the low‐risk criteria for serious bacterial infection in febrile neonates. Arch Dis Child. 2009;94:287–292.
- , , , et al. Is 15 days an appropriate cut‐off age for considering serious bacterial infection in the management of febrile infants? Pediatr Infect Dis J. 2012;31:455–458.
- , , . Failure of infant observation scales in detecting serious illness in febrile, 4‐ to 8‐week‐old infants. Pediatrics. 1990;85:1040–1043.
- , , . Outpatient management without antibiotics of fever in selected infants. N Engl J Med. 1993;329:1437–1441.
- , , . Identifying febrile infants at risk for a serious bacterial infection. J Pediatr. 1993;123:489–490.
- , , , et al. Febrile infants at low risk for serious bacterial infection—an appraisal of the Rochester criteria and implications for management. Febrile Infant Collaborative Study Group. Pediatrics. 1994;94:390–396.
- , , , et al. Management of febrile neonates in US pediatric emergency departments. Pediatrics. 2014;133:187–195.
- , , , et al. Variation in care of the febrile young infant <90 days in US pediatric emergency departments. Pediatrics. 2014;134:667–677.
- , , , et al. Association of clinical practice guidelines with emergency department management of febrile infants ≤56 days of age. J Hosp Med. 2015;10:358–365.
- , , , et al. Diagnosis and management of febrile infants (0‐3 months). Evid Rep Technol Assess (Full Rep). 2012;(205):1–297.
- , , , et al. Accuracy of administrative billing codes to detect urinary tract infection hospitalizations. Pediatrics. 2011;128:323–330.
- , , , et al. Identifying pediatric community‐acquired pneumonia hospitalizations: accuracy of administrative billing codes. JAMA Pediatr. 2013;167:851–858.
- , , , , , . Development and use of reporting guidelines for assessing the quality of validation studies of health administrative data. J Clin Epidemiol. 2011;64:821–829.
- American College of Emergency Physicians Clinical Policies Committee; American College of Emergency Physicians Clinical Policies Subcommittee on Pediatric Fever. Clinical policy for children younger than three years presenting to the emergency department with fever. Ann Emerg Med. 2003;42:530–545.
- , , , , , . Variation in occult injury screening for children with suspected abuse in selected US children's hospitals. Pediatrics. 2012;130:853–860.
- . Achieving data quality. How data from a pediatric health information system earns the trust of its users. J AHIMA. 2004;75:22–26.
- , , , . Corticosteroids and mortality in children with bacterial meningitis. JAMA. 2008;299:2048–2055.
- , , , , , . Research electronic data capture (REDCap)—a metadata‐driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42:377–381.
- , , , , , . Deaths attributed to pediatric complex chronic conditions: national trends and implications for supportive care services. Pediatrics. 2001;107:E99.
- US Census Bureau. Geographic terms and concepts—census divisions and census regions. Available at: https://www.census.gov/geo/reference/gtc/gtc_census_divreg.html. Accessed October 20, 2014.
- , , , . Initial emergency department diagnosis and return visits: risk versus perception. Ann Emerg Med. 1998;32:569–573.
- , , , , . A national depiction of children with return visits to the emergency department within 72 hours, 2001–2007. Pediatr Emerg Care. 2012;28:606–610.
- , , , et al. Pediatric observation status: are we overlooking a growing population in children's hospitals? J Hosp Med. 2012;7:530–536.
- , , , et al. Differences in designations of observation care in US freestanding children's hospitals: are they virtual or real? J Hosp Med. 2012;7:287–293.
- , , , , . Trends in the management of viral meningitis at United States children's hospitals. Pediatrics. 2013;131:670–676.
- , , , et al. Impact of increasing ondansetron use on clinical outcomes in children with gastroenteritis. JAMA Pediatr. 2014;168:321–329.
- , , , , . Race, otitis media, and antibiotic selection. Pediatrics. 2014;134:1059–1066.
- , , , et al. Establishing benchmarks for the hospitalized care of children with asthma, bronchiolitis, and pneumonia. Pediatrics. 2014;134:555–562.
- , , , , . Diagnostic testing and treatment of pediatric headache in the emergency department. J Pediatr. 2013;163:1634–1637.
© 2015 Society of Hospital Medicine
