Lise E. Nigrovic, MD, MPH

Study Design and Setting

For this multicenter retrospective study, we used the Pediatric Health Information System (PHIS) database to identify infants <90 days of age[16] who presented between July 1, 2012 and June 30, 2013 to 1 of 8 EDs. We assessed performance characteristics of ICD‐9 diagnosis code case‐identification algorithms by comparing ICD‐9 code combinations to a fever reference standard determined by medical record review. The institutional review board at each participating site approved the study protocol.

Data Source

Data were obtained from 2 sources: the PHIS database and medical record review. We used the PHIS database to identify eligible patients by ICD‐9 diagnosis codes; patient encounters were randomly selected using a random number generator. The PHIS database contains demographic, diagnosis, and billing data from 44 hospitals affiliated with the Children's Hospital Association (Overland Park, Kansas) and represents 85% of freestanding children's hospitals in the United States.[17] Data are deidentified; encrypted unique patient identifiers permit tracking of patients across visits within a site.[18] The Children's Hospital Association and participating hospitals jointly assure the quality and integrity of the data.[19]

For each patient encounter identified in the PHIS database, detailed medical record review was performed by trained investigators at each of the 8 study sites (see Supporting Information, Appendix, in the online version of this article). A standardized data collection instrument was pilot tested by all investigators prior to use. Data were collected and managed using the Research Electronic Data Capture (REDCap) tool hosted at Boston Children's Hospital.[20]

Exclusions

Using PHIS data, prior to medical record review we excluded infants with a complex chronic condition as defined previously[21] and those transferred from another institution, as these infants may warrant a nonstandard evaluation and/or may have incomplete data.

ICD‐9 Diagnosis Code Groups

In the PHIS database, all patients discharged from the hospital (including hospitalized patients as well as patients discharged from the ED) receive 1 or more ICD‐9 discharge diagnosis codes. These diagnosis codes are ascribed after discharge from the hospital, or for ED patients, after ED discharge. Additionally, patients may receive an admission diagnosis, which reflects the diagnosis ascribed at the time of ED discharge or transfer to the inpatient unit.

We reviewed medical records of infants selected from the following ICD‐9 diagnosis code groups (Figure 1): (1) discharge diagnosis code of fever (780.6 [fever and other physiologic disturbances of temperature regulation], 778.4 [other disturbances of temperature regulation of newborn], 780.60 [fever, unspecified], or 780.61 [fever presenting with conditions classified elsewhere])[9, 10] regardless of the presence of admission diagnosis of fever or diagnosis of serious infection, (2) admission diagnosis code of fever without associated discharge diagnosis code of fever,[10] (3) discharge diagnosis code of serious infection determined a priori (see Supporting Information, Appendix, in the online version of this article) without discharge or admission diagnosis code of fever, and (4) infants without any diagnosis code of fever or serious infection.

Figure 1

Medical records reviewed in each of the 4 ICD‐9 diagnosis code groups were randomly selected from the overall set of ED encounters in the population of infants <90 days of age evaluated during the study period. Twenty‐five percent population sampling was used for 3 of the ICD‐9 diagnosis code groups, whereas 5% sampling was used for the no fever/no serious infection code group. The number of medical records reviewed in each ICD‐9 diagnosis code group was proportional to the distribution of ICD‐9 codes across the entire population of infants <90 days of age. These records were distributed equally across sites (228 records per site), except for 1 site that does not assign admission diagnoses (201 records).

Investigators were blinded to ICD‐9 diagnosis code groups during medical record review. Infants with multiple visits during the study period were eligible to be included more than once if the visits occurred more than 3 days apart. For infants with more than 1 ED visit on a particular calendar day, investigators were instructed to review the initial visit.

For each encounter, we also abstracted demographic characteristics (gender, race/ethnicity), insurance status, hospital region (using US Census categories[22]), and season from the PHIS database.

Reference Standard

The presence of fever was determined by medical record review. We defined fever as any documented temperature 100.4F (38.0C) at home or in the ED.[16]

ICD‐9 Code Case‐Identification Algorithms

Using the aforementioned ICD‐9 diagnosis code groups individually and in combination, the following 4 case‐identification algorithms, determined from prior study or group consensus, were compared to the reference standard: (1) ICD‐9 discharge diagnosis code of fever,[9] (2) ICD‐9 admission or discharge diagnosis code of fever,[10, 11] (3) ICD‐9 discharge diagnosis code of fever or serious infection, and (4) ICD‐9 discharge or admission diagnosis code of fever or serious infection. Algorithms were compared overall, separately for discharged and hospitalized infants, and across 3 distinct age groups (28 days, 2956 days, and 5789 days).

Patient‐Level Outcomes

To compare differences in outcomes by case‐identification algorithm, from the PHIS database we abstracted hospitalization rates, rates of UTI/pyelonephritis,[13] bacteremia/sepsis, and bacterial meningitis.[19] Severe outcomes were defined as intensive care unit admission, mechanical ventilation, central line placement, receipt of extracorporeal membrane oxygenation, or death. We assessed hospital length of stay for admitted infants and 3‐day revisits,[23, 24] and revisits resulting in hospitalization for infants discharged from the ED at the index visit. Patients billed for observation care were classified as being hospitalized.[25, 26]

Data Analysis

Accuracy of the 4 case‐identification algorithms (compared with the reference standard) was calculated using sensitivity, specificity, negative predictive value (NPV), and positive predictive value (PPV), along with 95% confidence interval (CI). Prior to analysis, a 5‐fold weighting factor was applied to the no fever/no serious infection group to account for the differential sampling used for this group (5% vs 25% for the other 3 ICD‐9 diagnosis code groups). This weighting was done to approximate the true prevalence of each ICD‐9 code group within the larger population, so that an accurate rate of false negatives (infants with fever who had neither a diagnosis of fever nor serious infection) could be calculated.

We described continuous variables using median and interquartile range or range values and categorical variables using frequencies with 95% CIs. We compared categorical variables using a ² test. We determined statistical significance as a 2‐tailed P value <0.05. Statistical analyses were performed using SAS version 9.3 (SAS Institute, Cary, NC).

Study Patients

During the 1‐year study period, 23,753 ED encounters for infants <90 days of age were identified in the PHIS database at the 8 participating sites. Of these infant encounters, 2166 (9.2%) were excluded (1658 infants who had a complex chronic condition and 508 transferred into the ED), leaving 21,587 infants available for selection. After applying our sampling strategy, we identified 1797 encounters for medical record review. Seven encounters from 3 hospitals with missing medical records were excluded, resulting in a final cohort of 1790 encounters (Figure 1). Among included infants, 552 (30.8%) were 28 days, 743 (41.5%) were 29 to 56 days, and 495 (27.8%) were 57 to 89 days of age; 737 (41.2%) infants were hospitalized. Patients differed in age, race, payer, and season across ICD‐9 diagnosis code groups (see Supporting Information, Table 1, in the online version of this article).

Among the 1790 patient encounters reviewed, a total of 766 infants (42.8%) met the reference standard definition for fever in the cohort. An additional 47 infants had abnormal temperature reported (documentation of tactile fever, history of fever without a specific temperature described, or hypothermia) but were classified as having no fever by the reference standard.

ICD‐9 Code Case‐Identification Algorithm Performance

Compared with the reference standard, the 4 case‐identification algorithms demonstrated specificity of 96.2% to 98.2% but lower sensitivity overall (Figure 2). Discharge diagnosis of fever alone demonstrated the lowest sensitivity. The algorithm of discharge or admission diagnosis of fever resulted in increased sensitivity and the highest PPV of all 4 algorithms (86.9%, 95% CI: 84.5‐89.3). Addition of serious infection codes to this algorithm resulted in a marginal increase in sensitivity and a similar decrease in PPV (Table 1). When limited to hospitalized infants, specificity was highest for the case‐identification algorithm of discharge diagnosis of fever and similarly high for discharge or admission diagnosis of fever; sensitivity was highest for the algorithm of discharge or admission diagnosis of fever or diagnosis of serious infection. For infants discharged from the ED, algorithm specificity was 97.7% to 98.4%, with lower sensitivity for all 4 algorithms (Table 1). Inclusion of the 47 infants with abnormal temperature as fever did not materially change algorithm performance (data not shown).

Figure 2

Across all 3 age groups (28 days, 2956 days, and 5789 days), the 4 case‐identification algorithms demonstrated specificity >96%, whereas algorithm sensitivity was highest in the 29‐ to 56‐days‐old age group and lowest among infants 57 to 89 days old across all 4 algorithms (Figure 2). Similar to the overall cohort, an algorithm of discharge or admission diagnosis of fever demonstrated specificity of nearly 98% in all age groups; addition of serious infection codes to this algorithm increased sensitivity, highest in the 29‐ to 56‐days‐old age group (Figure 2; see also Supporting Information, Table 2, in the online version of this article).

Across the 8 study sites, median specificity was 95.8% to 98.3% for the 4 algorithms, with little interhospital variability; however, algorithm sensitivity varied widely by site. Median PPV was highest for discharge diagnosis of fever alone at 87.7% but ranged from 74.0% to 93.2% across sites. Median PPV for an algorithm of discharge or admission diagnosis of fever was similar (87.4%) but with less variation by site (range 80.0%92.9%) (Table 2).

Outcomes by ICD‐9 Diagnosis Code Group and Case‐Identification Algorithm

When compared with discharge diagnosis of fever, adding admission diagnosis of fever captured a higher proportion of hospitalized infants with SBIs (UTI/pyelonephritis, bacteremia/sepsis, or bacterial meningitis). However, median hospital length of stay, severe outcomes, and 3‐day revisits and revisits with hospitalization did not materially differ when including infants with admission diagnosis of fever in addition to discharge diagnosis of fever. Addition of infants with a diagnosis code for serious infection substantially increased the number of infants with SBIs and severe outcomes but did not capture additional 3‐day revisits (Table 3). There were no additional cases of SBI in the no fever/no serious illness diagnosis code group.

Among infants who met the reference standard for fever but did not have a discharge or admission diagnosis of fever (false negatives), 11.8% had a diagnosis of SBI. Overall, 43.2% of febrile infants (and 84.4% of hospitalized infants) with SBI did not have an ICD‐9 discharge or admission diagnosis of fever. Addition of ICD‐9 diagnosis codes of serious infection to the algorithm of discharge or admission diagnosis of fever captured all additional SBIs, and no false negativeinfants missed with this algorithm had an SBI.

Study Design and Setting

For this multicenter retrospective study, we used the Pediatric Health Information System (PHIS) database to identify infants <90 days of age[16] who presented between July 1, 2012 and June 30, 2013 to 1 of 8 EDs. We assessed performance characteristics of ICD‐9 diagnosis code case‐identification algorithms by comparing ICD‐9 code combinations to a fever reference standard determined by medical record review. The institutional review board at each participating site approved the study protocol.

Data Source

Data were obtained from 2 sources: the PHIS database and medical record review. We used the PHIS database to identify eligible patients by ICD‐9 diagnosis codes; patient encounters were randomly selected using a random number generator. The PHIS database contains demographic, diagnosis, and billing data from 44 hospitals affiliated with the Children's Hospital Association (Overland Park, Kansas) and represents 85% of freestanding children's hospitals in the United States.[17] Data are deidentified; encrypted unique patient identifiers permit tracking of patients across visits within a site.[18] The Children's Hospital Association and participating hospitals jointly assure the quality and integrity of the data.[19]

For each patient encounter identified in the PHIS database, detailed medical record review was performed by trained investigators at each of the 8 study sites (see Supporting Information, Appendix, in the online version of this article). A standardized data collection instrument was pilot tested by all investigators prior to use. Data were collected and managed using the Research Electronic Data Capture (REDCap) tool hosted at Boston Children's Hospital.[20]

Exclusions

Using PHIS data, prior to medical record review we excluded infants with a complex chronic condition as defined previously[21] and those transferred from another institution, as these infants may warrant a nonstandard evaluation and/or may have incomplete data.

ICD‐9 Diagnosis Code Groups

In the PHIS database, all patients discharged from the hospital (including hospitalized patients as well as patients discharged from the ED) receive 1 or more ICD‐9 discharge diagnosis codes. These diagnosis codes are ascribed after discharge from the hospital, or for ED patients, after ED discharge. Additionally, patients may receive an admission diagnosis, which reflects the diagnosis ascribed at the time of ED discharge or transfer to the inpatient unit.

We reviewed medical records of infants selected from the following ICD‐9 diagnosis code groups (Figure 1): (1) discharge diagnosis code of fever (780.6 [fever and other physiologic disturbances of temperature regulation], 778.4 [other disturbances of temperature regulation of newborn], 780.60 [fever, unspecified], or 780.61 [fever presenting with conditions classified elsewhere])[9, 10] regardless of the presence of admission diagnosis of fever or diagnosis of serious infection, (2) admission diagnosis code of fever without associated discharge diagnosis code of fever,[10] (3) discharge diagnosis code of serious infection determined a priori (see Supporting Information, Appendix, in the online version of this article) without discharge or admission diagnosis code of fever, and (4) infants without any diagnosis code of fever or serious infection.

Figure 1

Medical records reviewed in each of the 4 ICD‐9 diagnosis code groups were randomly selected from the overall set of ED encounters in the population of infants <90 days of age evaluated during the study period. Twenty‐five percent population sampling was used for 3 of the ICD‐9 diagnosis code groups, whereas 5% sampling was used for the no fever/no serious infection code group. The number of medical records reviewed in each ICD‐9 diagnosis code group was proportional to the distribution of ICD‐9 codes across the entire population of infants <90 days of age. These records were distributed equally across sites (228 records per site), except for 1 site that does not assign admission diagnoses (201 records).

Investigators were blinded to ICD‐9 diagnosis code groups during medical record review. Infants with multiple visits during the study period were eligible to be included more than once if the visits occurred more than 3 days apart. For infants with more than 1 ED visit on a particular calendar day, investigators were instructed to review the initial visit.

For each encounter, we also abstracted demographic characteristics (gender, race/ethnicity), insurance status, hospital region (using US Census categories[22]), and season from the PHIS database.

Reference Standard

The presence of fever was determined by medical record review. We defined fever as any documented temperature 100.4F (38.0C) at home or in the ED.[16]

ICD‐9 Code Case‐Identification Algorithms

Using the aforementioned ICD‐9 diagnosis code groups individually and in combination, the following 4 case‐identification algorithms, determined from prior study or group consensus, were compared to the reference standard: (1) ICD‐9 discharge diagnosis code of fever,[9] (2) ICD‐9 admission or discharge diagnosis code of fever,[10, 11] (3) ICD‐9 discharge diagnosis code of fever or serious infection, and (4) ICD‐9 discharge or admission diagnosis code of fever or serious infection. Algorithms were compared overall, separately for discharged and hospitalized infants, and across 3 distinct age groups (28 days, 2956 days, and 5789 days).

Patient‐Level Outcomes

To compare differences in outcomes by case‐identification algorithm, from the PHIS database we abstracted hospitalization rates, rates of UTI/pyelonephritis,[13] bacteremia/sepsis, and bacterial meningitis.[19] Severe outcomes were defined as intensive care unit admission, mechanical ventilation, central line placement, receipt of extracorporeal membrane oxygenation, or death. We assessed hospital length of stay for admitted infants and 3‐day revisits,[23, 24] and revisits resulting in hospitalization for infants discharged from the ED at the index visit. Patients billed for observation care were classified as being hospitalized.[25, 26]

Data Analysis

Accuracy of the 4 case‐identification algorithms (compared with the reference standard) was calculated using sensitivity, specificity, negative predictive value (NPV), and positive predictive value (PPV), along with 95% confidence interval (CI). Prior to analysis, a 5‐fold weighting factor was applied to the no fever/no serious infection group to account for the differential sampling used for this group (5% vs 25% for the other 3 ICD‐9 diagnosis code groups). This weighting was done to approximate the true prevalence of each ICD‐9 code group within the larger population, so that an accurate rate of false negatives (infants with fever who had neither a diagnosis of fever nor serious infection) could be calculated.

We described continuous variables using median and interquartile range or range values and categorical variables using frequencies with 95% CIs. We compared categorical variables using a ² test. We determined statistical significance as a 2‐tailed P value <0.05. Statistical analyses were performed using SAS version 9.3 (SAS Institute, Cary, NC).

Study Patients

During the 1‐year study period, 23,753 ED encounters for infants <90 days of age were identified in the PHIS database at the 8 participating sites. Of these infant encounters, 2166 (9.2%) were excluded (1658 infants who had a complex chronic condition and 508 transferred into the ED), leaving 21,587 infants available for selection. After applying our sampling strategy, we identified 1797 encounters for medical record review. Seven encounters from 3 hospitals with missing medical records were excluded, resulting in a final cohort of 1790 encounters (Figure 1). Among included infants, 552 (30.8%) were 28 days, 743 (41.5%) were 29 to 56 days, and 495 (27.8%) were 57 to 89 days of age; 737 (41.2%) infants were hospitalized. Patients differed in age, race, payer, and season across ICD‐9 diagnosis code groups (see Supporting Information, Table 1, in the online version of this article).

Among the 1790 patient encounters reviewed, a total of 766 infants (42.8%) met the reference standard definition for fever in the cohort. An additional 47 infants had abnormal temperature reported (documentation of tactile fever, history of fever without a specific temperature described, or hypothermia) but were classified as having no fever by the reference standard.

ICD‐9 Code Case‐Identification Algorithm Performance

Compared with the reference standard, the 4 case‐identification algorithms demonstrated specificity of 96.2% to 98.2% but lower sensitivity overall (Figure 2). Discharge diagnosis of fever alone demonstrated the lowest sensitivity. The algorithm of discharge or admission diagnosis of fever resulted in increased sensitivity and the highest PPV of all 4 algorithms (86.9%, 95% CI: 84.5‐89.3). Addition of serious infection codes to this algorithm resulted in a marginal increase in sensitivity and a similar decrease in PPV (Table 1). When limited to hospitalized infants, specificity was highest for the case‐identification algorithm of discharge diagnosis of fever and similarly high for discharge or admission diagnosis of fever; sensitivity was highest for the algorithm of discharge or admission diagnosis of fever or diagnosis of serious infection. For infants discharged from the ED, algorithm specificity was 97.7% to 98.4%, with lower sensitivity for all 4 algorithms (Table 1). Inclusion of the 47 infants with abnormal temperature as fever did not materially change algorithm performance (data not shown).

Figure 2

Across all 3 age groups (28 days, 2956 days, and 5789 days), the 4 case‐identification algorithms demonstrated specificity >96%, whereas algorithm sensitivity was highest in the 29‐ to 56‐days‐old age group and lowest among infants 57 to 89 days old across all 4 algorithms (Figure 2). Similar to the overall cohort, an algorithm of discharge or admission diagnosis of fever demonstrated specificity of nearly 98% in all age groups; addition of serious infection codes to this algorithm increased sensitivity, highest in the 29‐ to 56‐days‐old age group (Figure 2; see also Supporting Information, Table 2, in the online version of this article).

Across the 8 study sites, median specificity was 95.8% to 98.3% for the 4 algorithms, with little interhospital variability; however, algorithm sensitivity varied widely by site. Median PPV was highest for discharge diagnosis of fever alone at 87.7% but ranged from 74.0% to 93.2% across sites. Median PPV for an algorithm of discharge or admission diagnosis of fever was similar (87.4%) but with less variation by site (range 80.0%92.9%) (Table 2).

Outcomes by ICD‐9 Diagnosis Code Group and Case‐Identification Algorithm

When compared with discharge diagnosis of fever, adding admission diagnosis of fever captured a higher proportion of hospitalized infants with SBIs (UTI/pyelonephritis, bacteremia/sepsis, or bacterial meningitis). However, median hospital length of stay, severe outcomes, and 3‐day revisits and revisits with hospitalization did not materially differ when including infants with admission diagnosis of fever in addition to discharge diagnosis of fever. Addition of infants with a diagnosis code for serious infection substantially increased the number of infants with SBIs and severe outcomes but did not capture additional 3‐day revisits (Table 3). There were no additional cases of SBI in the no fever/no serious illness diagnosis code group.

Among infants who met the reference standard for fever but did not have a discharge or admission diagnosis of fever (false negatives), 11.8% had a diagnosis of SBI. Overall, 43.2% of febrile infants (and 84.4% of hospitalized infants) with SBI did not have an ICD‐9 discharge or admission diagnosis of fever. Addition of ICD‐9 diagnosis codes of serious infection to the algorithm of discharge or admission diagnosis of fever captured all additional SBIs, and no false negativeinfants missed with this algorithm had an SBI.

Study Design and Setting

For this multicenter retrospective study, we used the Pediatric Health Information System (PHIS) database to identify infants <90 days of age[16] who presented between July 1, 2012 and June 30, 2013 to 1 of 8 EDs. We assessed performance characteristics of ICD‐9 diagnosis code case‐identification algorithms by comparing ICD‐9 code combinations to a fever reference standard determined by medical record review. The institutional review board at each participating site approved the study protocol.

Data Source

Data were obtained from 2 sources: the PHIS database and medical record review. We used the PHIS database to identify eligible patients by ICD‐9 diagnosis codes; patient encounters were randomly selected using a random number generator. The PHIS database contains demographic, diagnosis, and billing data from 44 hospitals affiliated with the Children's Hospital Association (Overland Park, Kansas) and represents 85% of freestanding children's hospitals in the United States.[17] Data are deidentified; encrypted unique patient identifiers permit tracking of patients across visits within a site.[18] The Children's Hospital Association and participating hospitals jointly assure the quality and integrity of the data.[19]

For each patient encounter identified in the PHIS database, detailed medical record review was performed by trained investigators at each of the 8 study sites (see Supporting Information, Appendix, in the online version of this article). A standardized data collection instrument was pilot tested by all investigators prior to use. Data were collected and managed using the Research Electronic Data Capture (REDCap) tool hosted at Boston Children's Hospital.[20]

Exclusions

Using PHIS data, prior to medical record review we excluded infants with a complex chronic condition as defined previously[21] and those transferred from another institution, as these infants may warrant a nonstandard evaluation and/or may have incomplete data.

ICD‐9 Diagnosis Code Groups

In the PHIS database, all patients discharged from the hospital (including hospitalized patients as well as patients discharged from the ED) receive 1 or more ICD‐9 discharge diagnosis codes. These diagnosis codes are ascribed after discharge from the hospital, or for ED patients, after ED discharge. Additionally, patients may receive an admission diagnosis, which reflects the diagnosis ascribed at the time of ED discharge or transfer to the inpatient unit.

We reviewed medical records of infants selected from the following ICD‐9 diagnosis code groups (Figure 1): (1) discharge diagnosis code of fever (780.6 [fever and other physiologic disturbances of temperature regulation], 778.4 [other disturbances of temperature regulation of newborn], 780.60 [fever, unspecified], or 780.61 [fever presenting with conditions classified elsewhere])[9, 10] regardless of the presence of admission diagnosis of fever or diagnosis of serious infection, (2) admission diagnosis code of fever without associated discharge diagnosis code of fever,[10] (3) discharge diagnosis code of serious infection determined a priori (see Supporting Information, Appendix, in the online version of this article) without discharge or admission diagnosis code of fever, and (4) infants without any diagnosis code of fever or serious infection.

Figure 1

Medical records reviewed in each of the 4 ICD‐9 diagnosis code groups were randomly selected from the overall set of ED encounters in the population of infants <90 days of age evaluated during the study period. Twenty‐five percent population sampling was used for 3 of the ICD‐9 diagnosis code groups, whereas 5% sampling was used for the no fever/no serious infection code group. The number of medical records reviewed in each ICD‐9 diagnosis code group was proportional to the distribution of ICD‐9 codes across the entire population of infants <90 days of age. These records were distributed equally across sites (228 records per site), except for 1 site that does not assign admission diagnoses (201 records).

Investigators were blinded to ICD‐9 diagnosis code groups during medical record review. Infants with multiple visits during the study period were eligible to be included more than once if the visits occurred more than 3 days apart. For infants with more than 1 ED visit on a particular calendar day, investigators were instructed to review the initial visit.

For each encounter, we also abstracted demographic characteristics (gender, race/ethnicity), insurance status, hospital region (using US Census categories[22]), and season from the PHIS database.

Reference Standard

The presence of fever was determined by medical record review. We defined fever as any documented temperature 100.4F (38.0C) at home or in the ED.[16]

ICD‐9 Code Case‐Identification Algorithms

Using the aforementioned ICD‐9 diagnosis code groups individually and in combination, the following 4 case‐identification algorithms, determined from prior study or group consensus, were compared to the reference standard: (1) ICD‐9 discharge diagnosis code of fever,[9] (2) ICD‐9 admission or discharge diagnosis code of fever,[10, 11] (3) ICD‐9 discharge diagnosis code of fever or serious infection, and (4) ICD‐9 discharge or admission diagnosis code of fever or serious infection. Algorithms were compared overall, separately for discharged and hospitalized infants, and across 3 distinct age groups (28 days, 2956 days, and 5789 days).

Patient‐Level Outcomes

To compare differences in outcomes by case‐identification algorithm, from the PHIS database we abstracted hospitalization rates, rates of UTI/pyelonephritis,[13] bacteremia/sepsis, and bacterial meningitis.[19] Severe outcomes were defined as intensive care unit admission, mechanical ventilation, central line placement, receipt of extracorporeal membrane oxygenation, or death. We assessed hospital length of stay for admitted infants and 3‐day revisits,[23, 24] and revisits resulting in hospitalization for infants discharged from the ED at the index visit. Patients billed for observation care were classified as being hospitalized.[25, 26]

Data Analysis

Accuracy of the 4 case‐identification algorithms (compared with the reference standard) was calculated using sensitivity, specificity, negative predictive value (NPV), and positive predictive value (PPV), along with 95% confidence interval (CI). Prior to analysis, a 5‐fold weighting factor was applied to the no fever/no serious infection group to account for the differential sampling used for this group (5% vs 25% for the other 3 ICD‐9 diagnosis code groups). This weighting was done to approximate the true prevalence of each ICD‐9 code group within the larger population, so that an accurate rate of false negatives (infants with fever who had neither a diagnosis of fever nor serious infection) could be calculated.

We described continuous variables using median and interquartile range or range values and categorical variables using frequencies with 95% CIs. We compared categorical variables using a ² test. We determined statistical significance as a 2‐tailed P value <0.05. Statistical analyses were performed using SAS version 9.3 (SAS Institute, Cary, NC).

Study Patients

During the 1‐year study period, 23,753 ED encounters for infants <90 days of age were identified in the PHIS database at the 8 participating sites. Of these infant encounters, 2166 (9.2%) were excluded (1658 infants who had a complex chronic condition and 508 transferred into the ED), leaving 21,587 infants available for selection. After applying our sampling strategy, we identified 1797 encounters for medical record review. Seven encounters from 3 hospitals with missing medical records were excluded, resulting in a final cohort of 1790 encounters (Figure 1). Among included infants, 552 (30.8%) were 28 days, 743 (41.5%) were 29 to 56 days, and 495 (27.8%) were 57 to 89 days of age; 737 (41.2%) infants were hospitalized. Patients differed in age, race, payer, and season across ICD‐9 diagnosis code groups (see Supporting Information, Table 1, in the online version of this article).

Among the 1790 patient encounters reviewed, a total of 766 infants (42.8%) met the reference standard definition for fever in the cohort. An additional 47 infants had abnormal temperature reported (documentation of tactile fever, history of fever without a specific temperature described, or hypothermia) but were classified as having no fever by the reference standard.

ICD‐9 Code Case‐Identification Algorithm Performance

Compared with the reference standard, the 4 case‐identification algorithms demonstrated specificity of 96.2% to 98.2% but lower sensitivity overall (Figure 2). Discharge diagnosis of fever alone demonstrated the lowest sensitivity. The algorithm of discharge or admission diagnosis of fever resulted in increased sensitivity and the highest PPV of all 4 algorithms (86.9%, 95% CI: 84.5‐89.3). Addition of serious infection codes to this algorithm resulted in a marginal increase in sensitivity and a similar decrease in PPV (Table 1). When limited to hospitalized infants, specificity was highest for the case‐identification algorithm of discharge diagnosis of fever and similarly high for discharge or admission diagnosis of fever; sensitivity was highest for the algorithm of discharge or admission diagnosis of fever or diagnosis of serious infection. For infants discharged from the ED, algorithm specificity was 97.7% to 98.4%, with lower sensitivity for all 4 algorithms (Table 1). Inclusion of the 47 infants with abnormal temperature as fever did not materially change algorithm performance (data not shown).

Figure 2

Across all 3 age groups (28 days, 2956 days, and 5789 days), the 4 case‐identification algorithms demonstrated specificity >96%, whereas algorithm sensitivity was highest in the 29‐ to 56‐days‐old age group and lowest among infants 57 to 89 days old across all 4 algorithms (Figure 2). Similar to the overall cohort, an algorithm of discharge or admission diagnosis of fever demonstrated specificity of nearly 98% in all age groups; addition of serious infection codes to this algorithm increased sensitivity, highest in the 29‐ to 56‐days‐old age group (Figure 2; see also Supporting Information, Table 2, in the online version of this article).

Across the 8 study sites, median specificity was 95.8% to 98.3% for the 4 algorithms, with little interhospital variability; however, algorithm sensitivity varied widely by site. Median PPV was highest for discharge diagnosis of fever alone at 87.7% but ranged from 74.0% to 93.2% across sites. Median PPV for an algorithm of discharge or admission diagnosis of fever was similar (87.4%) but with less variation by site (range 80.0%92.9%) (Table 2).

Outcomes by ICD‐9 Diagnosis Code Group and Case‐Identification Algorithm

When compared with discharge diagnosis of fever, adding admission diagnosis of fever captured a higher proportion of hospitalized infants with SBIs (UTI/pyelonephritis, bacteremia/sepsis, or bacterial meningitis). However, median hospital length of stay, severe outcomes, and 3‐day revisits and revisits with hospitalization did not materially differ when including infants with admission diagnosis of fever in addition to discharge diagnosis of fever. Addition of infants with a diagnosis code for serious infection substantially increased the number of infants with SBIs and severe outcomes but did not capture additional 3‐day revisits (Table 3). There were no additional cases of SBI in the no fever/no serious illness diagnosis code group.

Among infants who met the reference standard for fever but did not have a discharge or admission diagnosis of fever (false negatives), 11.8% had a diagnosis of SBI. Overall, 43.2% of febrile infants (and 84.4% of hospitalized infants) with SBI did not have an ICD‐9 discharge or admission diagnosis of fever. Addition of ICD‐9 diagnosis codes of serious infection to the algorithm of discharge or admission diagnosis of fever captured all additional SBIs, and no false negativeinfants missed with this algorithm had an SBI.

Study Design

We used the Pediatric Health Information System (PHIS) to identify febrile infants 56 days of age who presented to the ED between January 1, 2013 and December 31, 2013. We also surveyed ED providers at participating PHIS hospitals. Informed consent was obtained from survey respondents. The institutional review board at Boston Children's Hospital approved the study protocol.

Clinical Practice Guideline Survey

We sent an electronic survey to medical directors or division directors at 37 pediatric EDs to determine whether their ED utilized a CPG for the management of the febrile young infant in 2013. If no response was received after the second attempt, we queried ED fellowship directors or other ED attending physicians at nonresponding hospitals. Survey items included the presence of a febrile young infant CPG, and if present, the year of implementation, ages targeted, and CPG content. As applicable, respondents were asked to share their CPG and/or provide the specific CPG recommendations.

We collected and managed survey data using the Research Electronic Data Capture (REDCap) electronic data capture tools hosted at Boston Children's Hospital. REDCap is a secure, Web‐based application designed to support data capture for research studies.[13]

Data Source

The PHIS database contains administrative data from 44 US children's hospitals. These hospitals, affiliated with the Children's Hospital Association, represent 85% of freestanding US children's hospitals.[14] Encrypted patient identifiers permit tracking of patients across encounters.[15] Data quality and integrity are assured jointly by the Children's Hospital Association and participating hospitals.[16] For this study, 7 hospitals were excluded due to incomplete ED data or known data‐quality issues.[17]

Patients

We identified study infants using the following International Classification of Diseases, 9th Revision (ICD‐9) admission or discharge diagnosis codes for fever as defined previously[8, 9]: 780.6, 778.4, 780.60, or 780.61. We excluded infants with a complex chronic condition[18] and those transferred from another institution, as these infants may warrant a nonstandard evaluation and/or may have incomplete data. For infants with >1 ED visit for fever during the study period, repeat visits within 3 days of an index visit were considered a revisit for the same episode of illness; visits >3 days following an index visit were considered as a new index visit.

Study Definitions

From the PHIS database, we abstracted demographic characteristics (gender, race/ethnicity), insurance status, and region where the hospital was located (using US Census categories[19]). Billing codes were used to assess whether urine, blood, and CSF testing (as defined previously[9]) were performed during the ED evaluation. To account for ED visits that spanned the midnight hour, for hospitalized patients we considered any testing or treatment occurring on the initial or second hospital day to be performed in the ED; billing code data in PHIS are based upon calendar day and do not distinguish testing performed in the ED versus inpatient setting.[8, 9] Patients billed for observation care were classified as being hospitalized.[20, 21]

We identified the presence of an SBI using ICD‐9 diagnosis codes for the following infections as described previously[9]: urinary tract infection or pyelonephritis,[22] bacteremia or sepsis, bacterial meningitis,[16] pneumonia,[23] or bacterial enteritis. To assess return visits for SBI that required inpatient management, we defined an ED revisit for an SBI as a return visit within 3 days of ED discharge[24, 25] that resulted in hospitalization with an associated ICD‐9 discharge diagnosis code for an SBI.

Hospitals charges in PHIS database were adjusted for hospital location by using the Centers for Medicare and Medicaid Services price/wage index. Costs were estimated by applying hospital‐level cost‐to‐charge ratios to charge data.[26]

Measured Exposures

The primary exposure was the presence of an ED‐based CPG for management of the febrile young infant aged 28 days and 29 to 56 days; 56 days was used as the upper age limit as all of the CPGs included infants up to this age or beyond. Six institutions utilized CPGs with different thresholds to define the age categories (eg, dichotomized at 27 or 30 days); these CPGs were classified into the aforementioned age groups to permit comparisons across standardized age groups. We classified institutions based on the presence of a CPG. To assess differences in the application of low‐risk criteria, the CPGs were further classified a priori based upon specific recommendations around laboratory testing and hospitalization, as well as ceftriaxone use for infants aged 29 to 56 days discharged from the ED. CPGs were categorized based upon whether testing, hospitalization, and ceftriaxone use were: (1) recommended for all patients, (2) recommended only if patients were classified as high risk (absence of low‐risk criteria), (3) recommended against, or (4) recommended to consider at clinician discretion.

Outcome Measures

Measured outcomes were performance of urine, blood, CSF testing, and hospitalization rate, as well as rate of ceftriaxone use for discharged infants aged 29 to 56 days, 3‐day revisits for SBI, and costs per visit, which included hospitalization costs for admitted patients.

Data Analysis

We described continuous variables using median and interquartile range or range values and categorical variables using frequencies. We compared medians using Wilcoxon rank sum and categorical variables using a [2] test. We compared rates of testing, hospitalization, ceftriaxone use, and 3‐day revisits for SBI based on the presence of a CPG, and when present, the specific CPG recommendations. Costs per visit were compared between institutions with and without CPGs and assessed separately for admitted and discharged patients. To adjust for potential confounders and clustering of patients within hospitals, we used generalized estimating equations with logistic regression to generate adjusted odd ratios (aORs) and 95% confidence intervals (CIs). Models were adjusted for geographic region, payer, race, and gender. Statistical analyses were performed by using SAS version 9.3 (SAS Institute, Cary, NC). We determined statistical significance as a 2‐tailed P value <0.05.

Febrile infants with bronchiolitis or a history of prematurity may be managed differently from full‐term febrile young infants without bronchiolitis.[6, 27] Therefore, we performed a subgroup analysis after exclusion of infants with an ICD‐9 discharge diagnosis code for bronchiolitis (466.11 and 466.19)[28] or prematurity (765).

Because our study included ED encounters in 2013, we repeated our analyses after exclusion of hospitals with CPGs implemented during the 2013 calendar year.

CPG by Institution

Thirty‐three (89.2%) of the 37 EDs surveyed completed the questionnaire. Overall, 21 (63.6%) of the 33 EDs had a CPG; 15 (45.5%) had a CPG for all infants 56 days of age, 5 (15.2%) had a CPG for infants 28 days only, and 1 (3.0%) had a CPG for infants 29 to 56 days but not 28 days of age (Figure 1). Seventeen EDs had an established CPG prior to 2013, and 4 hospitals implemented a CPG during the 2013 calendar year, 2 with CPGs for neonates 28 days and 2 with CPGs for both 28 days and 29 to 56 days of age. Hospitals with CPGs were more likely to be located in the Northeast and West regions of the United States and provide care to a higher proportion of non‐Hispanic white patients, as well as those with commercial insurance (Table 1).

Figure 1

All 20 CPGs for the febrile young infant 28 days of age recommended urine, blood, CSF testing, and hospitalization for all infants (Figure 1). Of the 16 hospitals with CPGs for febrile infants aged 29 to 56 days, all recommended urine and blood testing for all patients, except for 1 CPG, which recommended consideration of blood testing but not to obtain routinely. Hospitals varied in recommendations for CSF testing among infants aged 29 to 56 days: 8 (50%) recommended CSF testing in all patients and 8 (50%) recommended CSF testing only if the patient was high risk per defined criteria (based on history, physical examination, urine, and blood testing). In all 16 CPGs, hospitalization was recommended only for high‐risk infants. For low‐risk infants aged 2956 days being discharged from the ED, 3 hospitals recommended ceftriaxone for all, 9 recommended consideration of ceftriaxone, and 4 recommended against antibiotics (Figure 1).

Study Patients

During the study period, there were 10,415 infants 56 days old with a diagnosis of fever at the 33 participating hospitals. After exclusion of 635 (6.1%) infants with a complex chronic condition and 445 (4.3%) transferred from another institution (including 42 with a complex chronic condition), 9377 infants remained in our study cohort. Approximately one‐third of the cohort was 28 days of age and two‐thirds aged 29 to 56 days. The overall SBI rate was 8.5% but varied by age (11.9% in infants 28 days and 6.9% in infants 29 to 56 days of age) (Table 1).

CPGs and Use of Diagnostic Testing, Hospitalization Rates, Ceftriaxone Use, and Revisits for SBI

For infants 28 days of age, the presence of a CPG was not associated with urine, blood, CSF testing, or hospitalization after multivariable adjustment (Table 2). Among infants aged 29 to 56 days, urine testing did not differ based on the presence of a CPG, whereas blood testing was performed less often at the 1 hospital whose CPG recommended to consider, but not routinely obtain, testing (aOR: 0.4, 95% CI: 0.3‐0.7, P=0.001). Compared to hospitals without a CPG, CSF testing was performed less often at hospitals with CPG recommendations to only obtain CSF if high risk (aOR: 0.5, 95% CI: 0.3‐0.8, P=0.002). However, the odds of hospitalization did not differ at institutions with and without a febrile infant CPG (aOR: 0.7, 95% CI: 0.5‐1.1, P=0.10). For infants aged 29 to 56 days discharged from the ED, ceftriaxone was administered more often at hospitals with CPGs that recommended ceftriaxone for all discharged patients (aOR: 4.6, 95% CI: 2.39.3, P<0.001) and less often at hospitals whose CPGs recommended against antibiotics (aOR: 0.3, 95% CI: 0.1‐0.9, P=0.03) (Table 3). Our findings were similar in the subgroup of infants without bronchiolitis or prematurity (see Supporting Tables 1 and 2 in the online version of this article). After exclusion of hospitals with a CPG implemented during the 2013 calendar year (4 hospitals excluded in the 28 days age group and 2 hospitals excluded in the 29 to 56 days age group), infants aged 29 to 56 days cared for at a hospital with a CPG experienced a lower odds of hospitalization (aOR: 0.7, 95% CI: 0.4‐0.98, P=0.04). Otherwise, our findings in both age groups did not materially differ from the main analyses.

Three‐day revisits for SBI were similarly low at hospitals with and without CPGs among infants 28 days (1.5% vs 0.8%, P=0.44) and 29 to 56 days of age (1.4% vs 1.1%, P=0.44) and did not differ after exclusion of hospitals with a CPG implemented in 2013.

CPGs and Costs

Among infants 28 days of age, costs per visit did not differ for admitted and discharged patients based on CPG presence. The presence of an ED febrile infant CPG was associated with higher costs for both admitted and discharged infants 29 to 56 days of age (Table 4). The cost analysis did not significantly differ after exclusion of hospitals with CPGs implemented in 2013.

Study Design

We used the Pediatric Health Information System (PHIS) to identify febrile infants 56 days of age who presented to the ED between January 1, 2013 and December 31, 2013. We also surveyed ED providers at participating PHIS hospitals. Informed consent was obtained from survey respondents. The institutional review board at Boston Children's Hospital approved the study protocol.

Clinical Practice Guideline Survey

We sent an electronic survey to medical directors or division directors at 37 pediatric EDs to determine whether their ED utilized a CPG for the management of the febrile young infant in 2013. If no response was received after the second attempt, we queried ED fellowship directors or other ED attending physicians at nonresponding hospitals. Survey items included the presence of a febrile young infant CPG, and if present, the year of implementation, ages targeted, and CPG content. As applicable, respondents were asked to share their CPG and/or provide the specific CPG recommendations.

We collected and managed survey data using the Research Electronic Data Capture (REDCap) electronic data capture tools hosted at Boston Children's Hospital. REDCap is a secure, Web‐based application designed to support data capture for research studies.[13]

Data Source

The PHIS database contains administrative data from 44 US children's hospitals. These hospitals, affiliated with the Children's Hospital Association, represent 85% of freestanding US children's hospitals.[14] Encrypted patient identifiers permit tracking of patients across encounters.[15] Data quality and integrity are assured jointly by the Children's Hospital Association and participating hospitals.[16] For this study, 7 hospitals were excluded due to incomplete ED data or known data‐quality issues.[17]

Patients

We identified study infants using the following International Classification of Diseases, 9th Revision (ICD‐9) admission or discharge diagnosis codes for fever as defined previously[8, 9]: 780.6, 778.4, 780.60, or 780.61. We excluded infants with a complex chronic condition[18] and those transferred from another institution, as these infants may warrant a nonstandard evaluation and/or may have incomplete data. For infants with >1 ED visit for fever during the study period, repeat visits within 3 days of an index visit were considered a revisit for the same episode of illness; visits >3 days following an index visit were considered as a new index visit.

Study Definitions

From the PHIS database, we abstracted demographic characteristics (gender, race/ethnicity), insurance status, and region where the hospital was located (using US Census categories[19]). Billing codes were used to assess whether urine, blood, and CSF testing (as defined previously[9]) were performed during the ED evaluation. To account for ED visits that spanned the midnight hour, for hospitalized patients we considered any testing or treatment occurring on the initial or second hospital day to be performed in the ED; billing code data in PHIS are based upon calendar day and do not distinguish testing performed in the ED versus inpatient setting.[8, 9] Patients billed for observation care were classified as being hospitalized.[20, 21]

We identified the presence of an SBI using ICD‐9 diagnosis codes for the following infections as described previously[9]: urinary tract infection or pyelonephritis,[22] bacteremia or sepsis, bacterial meningitis,[16] pneumonia,[23] or bacterial enteritis. To assess return visits for SBI that required inpatient management, we defined an ED revisit for an SBI as a return visit within 3 days of ED discharge[24, 25] that resulted in hospitalization with an associated ICD‐9 discharge diagnosis code for an SBI.

Hospitals charges in PHIS database were adjusted for hospital location by using the Centers for Medicare and Medicaid Services price/wage index. Costs were estimated by applying hospital‐level cost‐to‐charge ratios to charge data.[26]

Measured Exposures

The primary exposure was the presence of an ED‐based CPG for management of the febrile young infant aged 28 days and 29 to 56 days; 56 days was used as the upper age limit as all of the CPGs included infants up to this age or beyond. Six institutions utilized CPGs with different thresholds to define the age categories (eg, dichotomized at 27 or 30 days); these CPGs were classified into the aforementioned age groups to permit comparisons across standardized age groups. We classified institutions based on the presence of a CPG. To assess differences in the application of low‐risk criteria, the CPGs were further classified a priori based upon specific recommendations around laboratory testing and hospitalization, as well as ceftriaxone use for infants aged 29 to 56 days discharged from the ED. CPGs were categorized based upon whether testing, hospitalization, and ceftriaxone use were: (1) recommended for all patients, (2) recommended only if patients were classified as high risk (absence of low‐risk criteria), (3) recommended against, or (4) recommended to consider at clinician discretion.

Outcome Measures

Measured outcomes were performance of urine, blood, CSF testing, and hospitalization rate, as well as rate of ceftriaxone use for discharged infants aged 29 to 56 days, 3‐day revisits for SBI, and costs per visit, which included hospitalization costs for admitted patients.

Data Analysis

We described continuous variables using median and interquartile range or range values and categorical variables using frequencies. We compared medians using Wilcoxon rank sum and categorical variables using a [2] test. We compared rates of testing, hospitalization, ceftriaxone use, and 3‐day revisits for SBI based on the presence of a CPG, and when present, the specific CPG recommendations. Costs per visit were compared between institutions with and without CPGs and assessed separately for admitted and discharged patients. To adjust for potential confounders and clustering of patients within hospitals, we used generalized estimating equations with logistic regression to generate adjusted odd ratios (aORs) and 95% confidence intervals (CIs). Models were adjusted for geographic region, payer, race, and gender. Statistical analyses were performed by using SAS version 9.3 (SAS Institute, Cary, NC). We determined statistical significance as a 2‐tailed P value <0.05.

Febrile infants with bronchiolitis or a history of prematurity may be managed differently from full‐term febrile young infants without bronchiolitis.[6, 27] Therefore, we performed a subgroup analysis after exclusion of infants with an ICD‐9 discharge diagnosis code for bronchiolitis (466.11 and 466.19)[28] or prematurity (765).

Because our study included ED encounters in 2013, we repeated our analyses after exclusion of hospitals with CPGs implemented during the 2013 calendar year.

CPG by Institution

Thirty‐three (89.2%) of the 37 EDs surveyed completed the questionnaire. Overall, 21 (63.6%) of the 33 EDs had a CPG; 15 (45.5%) had a CPG for all infants 56 days of age, 5 (15.2%) had a CPG for infants 28 days only, and 1 (3.0%) had a CPG for infants 29 to 56 days but not 28 days of age (Figure 1). Seventeen EDs had an established CPG prior to 2013, and 4 hospitals implemented a CPG during the 2013 calendar year, 2 with CPGs for neonates 28 days and 2 with CPGs for both 28 days and 29 to 56 days of age. Hospitals with CPGs were more likely to be located in the Northeast and West regions of the United States and provide care to a higher proportion of non‐Hispanic white patients, as well as those with commercial insurance (Table 1).

Figure 1

All 20 CPGs for the febrile young infant 28 days of age recommended urine, blood, CSF testing, and hospitalization for all infants (Figure 1). Of the 16 hospitals with CPGs for febrile infants aged 29 to 56 days, all recommended urine and blood testing for all patients, except for 1 CPG, which recommended consideration of blood testing but not to obtain routinely. Hospitals varied in recommendations for CSF testing among infants aged 29 to 56 days: 8 (50%) recommended CSF testing in all patients and 8 (50%) recommended CSF testing only if the patient was high risk per defined criteria (based on history, physical examination, urine, and blood testing). In all 16 CPGs, hospitalization was recommended only for high‐risk infants. For low‐risk infants aged 2956 days being discharged from the ED, 3 hospitals recommended ceftriaxone for all, 9 recommended consideration of ceftriaxone, and 4 recommended against antibiotics (Figure 1).

Study Patients

During the study period, there were 10,415 infants 56 days old with a diagnosis of fever at the 33 participating hospitals. After exclusion of 635 (6.1%) infants with a complex chronic condition and 445 (4.3%) transferred from another institution (including 42 with a complex chronic condition), 9377 infants remained in our study cohort. Approximately one‐third of the cohort was 28 days of age and two‐thirds aged 29 to 56 days. The overall SBI rate was 8.5% but varied by age (11.9% in infants 28 days and 6.9% in infants 29 to 56 days of age) (Table 1).

CPGs and Use of Diagnostic Testing, Hospitalization Rates, Ceftriaxone Use, and Revisits for SBI

For infants 28 days of age, the presence of a CPG was not associated with urine, blood, CSF testing, or hospitalization after multivariable adjustment (Table 2). Among infants aged 29 to 56 days, urine testing did not differ based on the presence of a CPG, whereas blood testing was performed less often at the 1 hospital whose CPG recommended to consider, but not routinely obtain, testing (aOR: 0.4, 95% CI: 0.3‐0.7, P=0.001). Compared to hospitals without a CPG, CSF testing was performed less often at hospitals with CPG recommendations to only obtain CSF if high risk (aOR: 0.5, 95% CI: 0.3‐0.8, P=0.002). However, the odds of hospitalization did not differ at institutions with and without a febrile infant CPG (aOR: 0.7, 95% CI: 0.5‐1.1, P=0.10). For infants aged 29 to 56 days discharged from the ED, ceftriaxone was administered more often at hospitals with CPGs that recommended ceftriaxone for all discharged patients (aOR: 4.6, 95% CI: 2.39.3, P<0.001) and less often at hospitals whose CPGs recommended against antibiotics (aOR: 0.3, 95% CI: 0.1‐0.9, P=0.03) (Table 3). Our findings were similar in the subgroup of infants without bronchiolitis or prematurity (see Supporting Tables 1 and 2 in the online version of this article). After exclusion of hospitals with a CPG implemented during the 2013 calendar year (4 hospitals excluded in the 28 days age group and 2 hospitals excluded in the 29 to 56 days age group), infants aged 29 to 56 days cared for at a hospital with a CPG experienced a lower odds of hospitalization (aOR: 0.7, 95% CI: 0.4‐0.98, P=0.04). Otherwise, our findings in both age groups did not materially differ from the main analyses.

Three‐day revisits for SBI were similarly low at hospitals with and without CPGs among infants 28 days (1.5% vs 0.8%, P=0.44) and 29 to 56 days of age (1.4% vs 1.1%, P=0.44) and did not differ after exclusion of hospitals with a CPG implemented in 2013.

CPGs and Costs

Among infants 28 days of age, costs per visit did not differ for admitted and discharged patients based on CPG presence. The presence of an ED febrile infant CPG was associated with higher costs for both admitted and discharged infants 29 to 56 days of age (Table 4). The cost analysis did not significantly differ after exclusion of hospitals with CPGs implemented in 2013.

Study Design

We used the Pediatric Health Information System (PHIS) to identify febrile infants 56 days of age who presented to the ED between January 1, 2013 and December 31, 2013. We also surveyed ED providers at participating PHIS hospitals. Informed consent was obtained from survey respondents. The institutional review board at Boston Children's Hospital approved the study protocol.

Clinical Practice Guideline Survey

We sent an electronic survey to medical directors or division directors at 37 pediatric EDs to determine whether their ED utilized a CPG for the management of the febrile young infant in 2013. If no response was received after the second attempt, we queried ED fellowship directors or other ED attending physicians at nonresponding hospitals. Survey items included the presence of a febrile young infant CPG, and if present, the year of implementation, ages targeted, and CPG content. As applicable, respondents were asked to share their CPG and/or provide the specific CPG recommendations.

We collected and managed survey data using the Research Electronic Data Capture (REDCap) electronic data capture tools hosted at Boston Children's Hospital. REDCap is a secure, Web‐based application designed to support data capture for research studies.[13]

Data Source

The PHIS database contains administrative data from 44 US children's hospitals. These hospitals, affiliated with the Children's Hospital Association, represent 85% of freestanding US children's hospitals.[14] Encrypted patient identifiers permit tracking of patients across encounters.[15] Data quality and integrity are assured jointly by the Children's Hospital Association and participating hospitals.[16] For this study, 7 hospitals were excluded due to incomplete ED data or known data‐quality issues.[17]

Patients

We identified study infants using the following International Classification of Diseases, 9th Revision (ICD‐9) admission or discharge diagnosis codes for fever as defined previously[8, 9]: 780.6, 778.4, 780.60, or 780.61. We excluded infants with a complex chronic condition[18] and those transferred from another institution, as these infants may warrant a nonstandard evaluation and/or may have incomplete data. For infants with >1 ED visit for fever during the study period, repeat visits within 3 days of an index visit were considered a revisit for the same episode of illness; visits >3 days following an index visit were considered as a new index visit.

Study Definitions

From the PHIS database, we abstracted demographic characteristics (gender, race/ethnicity), insurance status, and region where the hospital was located (using US Census categories[19]). Billing codes were used to assess whether urine, blood, and CSF testing (as defined previously[9]) were performed during the ED evaluation. To account for ED visits that spanned the midnight hour, for hospitalized patients we considered any testing or treatment occurring on the initial or second hospital day to be performed in the ED; billing code data in PHIS are based upon calendar day and do not distinguish testing performed in the ED versus inpatient setting.[8, 9] Patients billed for observation care were classified as being hospitalized.[20, 21]

We identified the presence of an SBI using ICD‐9 diagnosis codes for the following infections as described previously[9]: urinary tract infection or pyelonephritis,[22] bacteremia or sepsis, bacterial meningitis,[16] pneumonia,[23] or bacterial enteritis. To assess return visits for SBI that required inpatient management, we defined an ED revisit for an SBI as a return visit within 3 days of ED discharge[24, 25] that resulted in hospitalization with an associated ICD‐9 discharge diagnosis code for an SBI.

Hospitals charges in PHIS database were adjusted for hospital location by using the Centers for Medicare and Medicaid Services price/wage index. Costs were estimated by applying hospital‐level cost‐to‐charge ratios to charge data.[26]

Measured Exposures

The primary exposure was the presence of an ED‐based CPG for management of the febrile young infant aged 28 days and 29 to 56 days; 56 days was used as the upper age limit as all of the CPGs included infants up to this age or beyond. Six institutions utilized CPGs with different thresholds to define the age categories (eg, dichotomized at 27 or 30 days); these CPGs were classified into the aforementioned age groups to permit comparisons across standardized age groups. We classified institutions based on the presence of a CPG. To assess differences in the application of low‐risk criteria, the CPGs were further classified a priori based upon specific recommendations around laboratory testing and hospitalization, as well as ceftriaxone use for infants aged 29 to 56 days discharged from the ED. CPGs were categorized based upon whether testing, hospitalization, and ceftriaxone use were: (1) recommended for all patients, (2) recommended only if patients were classified as high risk (absence of low‐risk criteria), (3) recommended against, or (4) recommended to consider at clinician discretion.

Outcome Measures

Measured outcomes were performance of urine, blood, CSF testing, and hospitalization rate, as well as rate of ceftriaxone use for discharged infants aged 29 to 56 days, 3‐day revisits for SBI, and costs per visit, which included hospitalization costs for admitted patients.

Data Analysis

We described continuous variables using median and interquartile range or range values and categorical variables using frequencies. We compared medians using Wilcoxon rank sum and categorical variables using a [2] test. We compared rates of testing, hospitalization, ceftriaxone use, and 3‐day revisits for SBI based on the presence of a CPG, and when present, the specific CPG recommendations. Costs per visit were compared between institutions with and without CPGs and assessed separately for admitted and discharged patients. To adjust for potential confounders and clustering of patients within hospitals, we used generalized estimating equations with logistic regression to generate adjusted odd ratios (aORs) and 95% confidence intervals (CIs). Models were adjusted for geographic region, payer, race, and gender. Statistical analyses were performed by using SAS version 9.3 (SAS Institute, Cary, NC). We determined statistical significance as a 2‐tailed P value <0.05.

Febrile infants with bronchiolitis or a history of prematurity may be managed differently from full‐term febrile young infants without bronchiolitis.[6, 27] Therefore, we performed a subgroup analysis after exclusion of infants with an ICD‐9 discharge diagnosis code for bronchiolitis (466.11 and 466.19)[28] or prematurity (765).

Because our study included ED encounters in 2013, we repeated our analyses after exclusion of hospitals with CPGs implemented during the 2013 calendar year.

CPG by Institution

Thirty‐three (89.2%) of the 37 EDs surveyed completed the questionnaire. Overall, 21 (63.6%) of the 33 EDs had a CPG; 15 (45.5%) had a CPG for all infants 56 days of age, 5 (15.2%) had a CPG for infants 28 days only, and 1 (3.0%) had a CPG for infants 29 to 56 days but not 28 days of age (Figure 1). Seventeen EDs had an established CPG prior to 2013, and 4 hospitals implemented a CPG during the 2013 calendar year, 2 with CPGs for neonates 28 days and 2 with CPGs for both 28 days and 29 to 56 days of age. Hospitals with CPGs were more likely to be located in the Northeast and West regions of the United States and provide care to a higher proportion of non‐Hispanic white patients, as well as those with commercial insurance (Table 1).

Figure 1

All 20 CPGs for the febrile young infant 28 days of age recommended urine, blood, CSF testing, and hospitalization for all infants (Figure 1). Of the 16 hospitals with CPGs for febrile infants aged 29 to 56 days, all recommended urine and blood testing for all patients, except for 1 CPG, which recommended consideration of blood testing but not to obtain routinely. Hospitals varied in recommendations for CSF testing among infants aged 29 to 56 days: 8 (50%) recommended CSF testing in all patients and 8 (50%) recommended CSF testing only if the patient was high risk per defined criteria (based on history, physical examination, urine, and blood testing). In all 16 CPGs, hospitalization was recommended only for high‐risk infants. For low‐risk infants aged 2956 days being discharged from the ED, 3 hospitals recommended ceftriaxone for all, 9 recommended consideration of ceftriaxone, and 4 recommended against antibiotics (Figure 1).

Study Patients

During the study period, there were 10,415 infants 56 days old with a diagnosis of fever at the 33 participating hospitals. After exclusion of 635 (6.1%) infants with a complex chronic condition and 445 (4.3%) transferred from another institution (including 42 with a complex chronic condition), 9377 infants remained in our study cohort. Approximately one‐third of the cohort was 28 days of age and two‐thirds aged 29 to 56 days. The overall SBI rate was 8.5% but varied by age (11.9% in infants 28 days and 6.9% in infants 29 to 56 days of age) (Table 1).

CPGs and Use of Diagnostic Testing, Hospitalization Rates, Ceftriaxone Use, and Revisits for SBI

For infants 28 days of age, the presence of a CPG was not associated with urine, blood, CSF testing, or hospitalization after multivariable adjustment (Table 2). Among infants aged 29 to 56 days, urine testing did not differ based on the presence of a CPG, whereas blood testing was performed less often at the 1 hospital whose CPG recommended to consider, but not routinely obtain, testing (aOR: 0.4, 95% CI: 0.3‐0.7, P=0.001). Compared to hospitals without a CPG, CSF testing was performed less often at hospitals with CPG recommendations to only obtain CSF if high risk (aOR: 0.5, 95% CI: 0.3‐0.8, P=0.002). However, the odds of hospitalization did not differ at institutions with and without a febrile infant CPG (aOR: 0.7, 95% CI: 0.5‐1.1, P=0.10). For infants aged 29 to 56 days discharged from the ED, ceftriaxone was administered more often at hospitals with CPGs that recommended ceftriaxone for all discharged patients (aOR: 4.6, 95% CI: 2.39.3, P<0.001) and less often at hospitals whose CPGs recommended against antibiotics (aOR: 0.3, 95% CI: 0.1‐0.9, P=0.03) (Table 3). Our findings were similar in the subgroup of infants without bronchiolitis or prematurity (see Supporting Tables 1 and 2 in the online version of this article). After exclusion of hospitals with a CPG implemented during the 2013 calendar year (4 hospitals excluded in the 28 days age group and 2 hospitals excluded in the 29 to 56 days age group), infants aged 29 to 56 days cared for at a hospital with a CPG experienced a lower odds of hospitalization (aOR: 0.7, 95% CI: 0.4‐0.98, P=0.04). Otherwise, our findings in both age groups did not materially differ from the main analyses.

Three‐day revisits for SBI were similarly low at hospitals with and without CPGs among infants 28 days (1.5% vs 0.8%, P=0.44) and 29 to 56 days of age (1.4% vs 1.1%, P=0.44) and did not differ after exclusion of hospitals with a CPG implemented in 2013.

CPGs and Costs

Among infants 28 days of age, costs per visit did not differ for admitted and discharged patients based on CPG presence. The presence of an ED febrile infant CPG was associated with higher costs for both admitted and discharged infants 29 to 56 days of age (Table 4). The cost analysis did not significantly differ after exclusion of hospitals with CPGs implemented in 2013.

Study Design and Population

We identified children, <19 years of age, with meningitis evaluated at a single tertiary care pediatric center between July 2006 and June 2010. We defined meningitis as a CSF white blood cell (WBC) count 10 cells/mm³ corrected for the presence of CSF red blood cells (RBCs) (1 WBC for every 500 RBCs).6 We excluded children with any of the following: critical illness (defined as hypotension or respiratory failure), purpura, recent neurosurgery, ventricular shunt, immunosuppression, focal bacterial infection requiring parenteral antibiotics, positive CSF Gram stain, or known Lyme disease. The Institutional Review Board approved this study with waiver of informed consent.

Data Collection and Case Definitions

We abstracted historical and physical examination findings, as well as laboratory and microbiologic results, from the medical record. We defined bacterial meningitis as the isolation of pathogenic bacteria from the CSF or blood cultures. Children who had received antibiotics within 72 hours of diagnostic lumbar puncture, with negative cultures, were considered to have pretreated culture‐negative meningitis. Otherwise, children with negative bacterial cultures were classified as having aseptic meningitis.

EVPCR Testing

During the study pre‐period (July 1, 2006 through June 23, 2008), EVPCR tests were flown once daily to a commercial laboratory (ARUP Laboratories, Salt Lake City, UT) where they were run in batches. During the post‐period (June 24, 2008 through June 30, 2010), the study institution replaced the send‐out test with an in‐hospital EVPCR test (Gene Xpert EV Technology; Cepheid, Sunnyvale, CA)7 that allows multiple specimens to be run simultaneously, multiple times daily (between 7:00 _AM and 10:00 _PM), with results available in as little as 2.5 hours. We defined turnaround time for the test from specimen obtainment to test result.

Outcome Measures

Our 2 primary outcomes were length of stay and duration of parenteral antibiotics. Length of stay was measured as time from emergency department arrival to discharge (emergency department or inpatient discharge). We defined the duration of parenteral antibiotics as time from the first to the last dose of parenteral antibiotics administered, plus the standard antibiotic dosing interval for that antibiotic. For children with Lyme meningitis, the duration of parenteral antibiotic coverage was defined a priori as 48 hours, the standard time to reliably exclude bacterial growth from culture.8

Statistical Methods

Primary outcomes were compared using univariate analyses in 6 patient groups: 1) all patients, and those with 2) a positive EVPCR test, 3) a negative EVPCR test, and a positive test who were 4) 90 days old, 5) >90 days old, and 6) presented during peak enteroviral season (June through October). We utilized MannWhitney tests for continuous variables and ² tests for proportions. We compared the median turnaround time for EVPCR results and the percentage of tests returning prior to discharge between the pre‐ and post‐periods. We performed interrupted time series spline analyses to assess for trends in our primary outcomes, independent of the change in EVPCR testing protocol. All analyses were conducted using the Statistical Package for the Social Sciences (IBM SPSS Inc, Chicago, IL).9

Study Design and Population

We identified children, <19 years of age, with meningitis evaluated at a single tertiary care pediatric center between July 2006 and June 2010. We defined meningitis as a CSF white blood cell (WBC) count 10 cells/mm³ corrected for the presence of CSF red blood cells (RBCs) (1 WBC for every 500 RBCs).6 We excluded children with any of the following: critical illness (defined as hypotension or respiratory failure), purpura, recent neurosurgery, ventricular shunt, immunosuppression, focal bacterial infection requiring parenteral antibiotics, positive CSF Gram stain, or known Lyme disease. The Institutional Review Board approved this study with waiver of informed consent.

Data Collection and Case Definitions

We abstracted historical and physical examination findings, as well as laboratory and microbiologic results, from the medical record. We defined bacterial meningitis as the isolation of pathogenic bacteria from the CSF or blood cultures. Children who had received antibiotics within 72 hours of diagnostic lumbar puncture, with negative cultures, were considered to have pretreated culture‐negative meningitis. Otherwise, children with negative bacterial cultures were classified as having aseptic meningitis.

EVPCR Testing

During the study pre‐period (July 1, 2006 through June 23, 2008), EVPCR tests were flown once daily to a commercial laboratory (ARUP Laboratories, Salt Lake City, UT) where they were run in batches. During the post‐period (June 24, 2008 through June 30, 2010), the study institution replaced the send‐out test with an in‐hospital EVPCR test (Gene Xpert EV Technology; Cepheid, Sunnyvale, CA)7 that allows multiple specimens to be run simultaneously, multiple times daily (between 7:00 _AM and 10:00 _PM), with results available in as little as 2.5 hours. We defined turnaround time for the test from specimen obtainment to test result.

Outcome Measures

Our 2 primary outcomes were length of stay and duration of parenteral antibiotics. Length of stay was measured as time from emergency department arrival to discharge (emergency department or inpatient discharge). We defined the duration of parenteral antibiotics as time from the first to the last dose of parenteral antibiotics administered, plus the standard antibiotic dosing interval for that antibiotic. For children with Lyme meningitis, the duration of parenteral antibiotic coverage was defined a priori as 48 hours, the standard time to reliably exclude bacterial growth from culture.8

Statistical Methods

Primary outcomes were compared using univariate analyses in 6 patient groups: 1) all patients, and those with 2) a positive EVPCR test, 3) a negative EVPCR test, and a positive test who were 4) 90 days old, 5) >90 days old, and 6) presented during peak enteroviral season (June through October). We utilized MannWhitney tests for continuous variables and ² tests for proportions. We compared the median turnaround time for EVPCR results and the percentage of tests returning prior to discharge between the pre‐ and post‐periods. We performed interrupted time series spline analyses to assess for trends in our primary outcomes, independent of the change in EVPCR testing protocol. All analyses were conducted using the Statistical Package for the Social Sciences (IBM SPSS Inc, Chicago, IL).9

Study Design and Population

We identified children, <19 years of age, with meningitis evaluated at a single tertiary care pediatric center between July 2006 and June 2010. We defined meningitis as a CSF white blood cell (WBC) count 10 cells/mm³ corrected for the presence of CSF red blood cells (RBCs) (1 WBC for every 500 RBCs).6 We excluded children with any of the following: critical illness (defined as hypotension or respiratory failure), purpura, recent neurosurgery, ventricular shunt, immunosuppression, focal bacterial infection requiring parenteral antibiotics, positive CSF Gram stain, or known Lyme disease. The Institutional Review Board approved this study with waiver of informed consent.

Data Collection and Case Definitions

We abstracted historical and physical examination findings, as well as laboratory and microbiologic results, from the medical record. We defined bacterial meningitis as the isolation of pathogenic bacteria from the CSF or blood cultures. Children who had received antibiotics within 72 hours of diagnostic lumbar puncture, with negative cultures, were considered to have pretreated culture‐negative meningitis. Otherwise, children with negative bacterial cultures were classified as having aseptic meningitis.

EVPCR Testing

During the study pre‐period (July 1, 2006 through June 23, 2008), EVPCR tests were flown once daily to a commercial laboratory (ARUP Laboratories, Salt Lake City, UT) where they were run in batches. During the post‐period (June 24, 2008 through June 30, 2010), the study institution replaced the send‐out test with an in‐hospital EVPCR test (Gene Xpert EV Technology; Cepheid, Sunnyvale, CA)7 that allows multiple specimens to be run simultaneously, multiple times daily (between 7:00 _AM and 10:00 _PM), with results available in as little as 2.5 hours. We defined turnaround time for the test from specimen obtainment to test result.

Outcome Measures

Our 2 primary outcomes were length of stay and duration of parenteral antibiotics. Length of stay was measured as time from emergency department arrival to discharge (emergency department or inpatient discharge). We defined the duration of parenteral antibiotics as time from the first to the last dose of parenteral antibiotics administered, plus the standard antibiotic dosing interval for that antibiotic. For children with Lyme meningitis, the duration of parenteral antibiotic coverage was defined a priori as 48 hours, the standard time to reliably exclude bacterial growth from culture.8

Statistical Methods

Primary outcomes were compared using univariate analyses in 6 patient groups: 1) all patients, and those with 2) a positive EVPCR test, 3) a negative EVPCR test, and a positive test who were 4) 90 days old, 5) >90 days old, and 6) presented during peak enteroviral season (June through October). We utilized MannWhitney tests for continuous variables and ² tests for proportions. We compared the median turnaround time for EVPCR results and the percentage of tests returning prior to discharge between the pre‐ and post‐periods. We performed interrupted time series spline analyses to assess for trends in our primary outcomes, independent of the change in EVPCR testing protocol. All analyses were conducted using the Statistical Package for the Social Sciences (IBM SPSS Inc, Chicago, IL).9