Samir S. Shah, MD, MSCE

Short‐Term Focused Engagement Via Focus Groups With Parents/Caregivers

The first aim of the H2O study used 15 focus groups and semistructured interviews with parents/caregivers of recently discharged patients to identify barriers to and metrics of successful transitions of care from the hospital to home. The focus group question guide was developed by the research team and adapted as the focus groups progressed to incorporate new issues raised by participants. Analysis of focus group data revealed opportunities to improve the transitional home visit and identified outcomes important to families, including the need for emotional reassurance in the immediate period after discharge and the impact on family finances.

Short‐Term Focused Engagement Via Phone Calls With Parents/Caregivers

To continuously improve study processes and the transitional home visit during the second aim of H2O, we relied on short‐term focused engagement from 2 stakeholder groups, families and field nurses. We completed 107 phone calls with families who received a transitional home visit during the visit optimization period. These calls, completed 3 to 7 days after the visit, assessed parental perceptions of the effect of recent visit modifications through a standardized survey documented in an electronic database. These data were utilized in plan‐do‐study‐act cycles,[5] every 1 to 2 weeks, to determine if additional modifications to the visits were necessary. A cycle ended when the calls no longer provided new information. The questions asked on the calls also changed over time as different interventions were tested.

As an example, in aim 1, families highlighted the lack of clarity of discharge instructions, particularly regarding when and why to return for medical care. Thus, we developed condition‐specific red flag reminder cards to be shared at transitional home visits to help families remember and recognize concerning signs and symptoms and understand when additional evaluation may be warranted (Figure 2). Families in postvisit calls endorsed the concept of red flags, but sometimes preferred electronic rather than paper versions of the red flag cards to facilitate sharing with family members. Thus, we tested and refined texting the red flag information to families. Subsequent calls strongly supported this practice, so we will continue to use it during the third aim, the randomized trial of the transitional home visit.

Figure 2

The remaining calls (N=72) were completed 14 days after the visit to mirror the time frame for follow‐up calls in the planned randomized trial. These calls allowed us to test measurement of family‐identified outcomes and determine their usability in the trial. We used family feedback to shorten the survey and reorder questions. We also used feedback from these calls to develop an optimal call‐back strategy to maximize family contacts.

Short‐Term Focused Engagement Via Discussions With Nurses

We also incorporated feedback from HHC nurses on 60 visits to ensure that the visit modifications were feasible to implement. HHC nurse feedback, which aligned with aim 1 data from families, highlighted the potential benefits of standardizing the transitional home visit to be more condition specific. The nurses also provided ongoing ad hoc feedback on other changes to the transitional home visit, which indicated both when tests were successful and when they were challenging to implement. The study team wanted to ensure that the nurses performing the visits were involved in the modification process.

Short‐Term Focused Engagement Via Focus Groups With Parents/Caregivers

The first aim of the H2O study used 15 focus groups and semistructured interviews with parents/caregivers of recently discharged patients to identify barriers to and metrics of successful transitions of care from the hospital to home. The focus group question guide was developed by the research team and adapted as the focus groups progressed to incorporate new issues raised by participants. Analysis of focus group data revealed opportunities to improve the transitional home visit and identified outcomes important to families, including the need for emotional reassurance in the immediate period after discharge and the impact on family finances.

Short‐Term Focused Engagement Via Phone Calls With Parents/Caregivers

To continuously improve study processes and the transitional home visit during the second aim of H2O, we relied on short‐term focused engagement from 2 stakeholder groups, families and field nurses. We completed 107 phone calls with families who received a transitional home visit during the visit optimization period. These calls, completed 3 to 7 days after the visit, assessed parental perceptions of the effect of recent visit modifications through a standardized survey documented in an electronic database. These data were utilized in plan‐do‐study‐act cycles,[5] every 1 to 2 weeks, to determine if additional modifications to the visits were necessary. A cycle ended when the calls no longer provided new information. The questions asked on the calls also changed over time as different interventions were tested.

As an example, in aim 1, families highlighted the lack of clarity of discharge instructions, particularly regarding when and why to return for medical care. Thus, we developed condition‐specific red flag reminder cards to be shared at transitional home visits to help families remember and recognize concerning signs and symptoms and understand when additional evaluation may be warranted (Figure 2). Families in postvisit calls endorsed the concept of red flags, but sometimes preferred electronic rather than paper versions of the red flag cards to facilitate sharing with family members. Thus, we tested and refined texting the red flag information to families. Subsequent calls strongly supported this practice, so we will continue to use it during the third aim, the randomized trial of the transitional home visit.

Figure 2

The remaining calls (N=72) were completed 14 days after the visit to mirror the time frame for follow‐up calls in the planned randomized trial. These calls allowed us to test measurement of family‐identified outcomes and determine their usability in the trial. We used family feedback to shorten the survey and reorder questions. We also used feedback from these calls to develop an optimal call‐back strategy to maximize family contacts.

Short‐Term Focused Engagement Via Discussions With Nurses

We also incorporated feedback from HHC nurses on 60 visits to ensure that the visit modifications were feasible to implement. HHC nurse feedback, which aligned with aim 1 data from families, highlighted the potential benefits of standardizing the transitional home visit to be more condition specific. The nurses also provided ongoing ad hoc feedback on other changes to the transitional home visit, which indicated both when tests were successful and when they were challenging to implement. The study team wanted to ensure that the nurses performing the visits were involved in the modification process.

Short‐Term Focused Engagement Via Focus Groups With Parents/Caregivers

The first aim of the H2O study used 15 focus groups and semistructured interviews with parents/caregivers of recently discharged patients to identify barriers to and metrics of successful transitions of care from the hospital to home. The focus group question guide was developed by the research team and adapted as the focus groups progressed to incorporate new issues raised by participants. Analysis of focus group data revealed opportunities to improve the transitional home visit and identified outcomes important to families, including the need for emotional reassurance in the immediate period after discharge and the impact on family finances.

Short‐Term Focused Engagement Via Phone Calls With Parents/Caregivers

To continuously improve study processes and the transitional home visit during the second aim of H2O, we relied on short‐term focused engagement from 2 stakeholder groups, families and field nurses. We completed 107 phone calls with families who received a transitional home visit during the visit optimization period. These calls, completed 3 to 7 days after the visit, assessed parental perceptions of the effect of recent visit modifications through a standardized survey documented in an electronic database. These data were utilized in plan‐do‐study‐act cycles,[5] every 1 to 2 weeks, to determine if additional modifications to the visits were necessary. A cycle ended when the calls no longer provided new information. The questions asked on the calls also changed over time as different interventions were tested.

As an example, in aim 1, families highlighted the lack of clarity of discharge instructions, particularly regarding when and why to return for medical care. Thus, we developed condition‐specific red flag reminder cards to be shared at transitional home visits to help families remember and recognize concerning signs and symptoms and understand when additional evaluation may be warranted (Figure 2). Families in postvisit calls endorsed the concept of red flags, but sometimes preferred electronic rather than paper versions of the red flag cards to facilitate sharing with family members. Thus, we tested and refined texting the red flag information to families. Subsequent calls strongly supported this practice, so we will continue to use it during the third aim, the randomized trial of the transitional home visit.

Figure 2

The remaining calls (N=72) were completed 14 days after the visit to mirror the time frame for follow‐up calls in the planned randomized trial. These calls allowed us to test measurement of family‐identified outcomes and determine their usability in the trial. We used family feedback to shorten the survey and reorder questions. We also used feedback from these calls to develop an optimal call‐back strategy to maximize family contacts.

Short‐Term Focused Engagement Via Discussions With Nurses

We also incorporated feedback from HHC nurses on 60 visits to ensure that the visit modifications were feasible to implement. HHC nurse feedback, which aligned with aim 1 data from families, highlighted the potential benefits of standardizing the transitional home visit to be more condition specific. The nurses also provided ongoing ad hoc feedback on other changes to the transitional home visit, which indicated both when tests were successful and when they were challenging to implement. The study team wanted to ensure that the nurses performing the visits were involved in the modification process.

Study Design and Setting

For this multicenter retrospective study, we used the Pediatric Health Information System (PHIS) database to identify infants <90 days of age[16] who presented between July 1, 2012 and June 30, 2013 to 1 of 8 EDs. We assessed performance characteristics of ICD‐9 diagnosis code case‐identification algorithms by comparing ICD‐9 code combinations to a fever reference standard determined by medical record review. The institutional review board at each participating site approved the study protocol.

Data Source

Data were obtained from 2 sources: the PHIS database and medical record review. We used the PHIS database to identify eligible patients by ICD‐9 diagnosis codes; patient encounters were randomly selected using a random number generator. The PHIS database contains demographic, diagnosis, and billing data from 44 hospitals affiliated with the Children's Hospital Association (Overland Park, Kansas) and represents 85% of freestanding children's hospitals in the United States.[17] Data are deidentified; encrypted unique patient identifiers permit tracking of patients across visits within a site.[18] The Children's Hospital Association and participating hospitals jointly assure the quality and integrity of the data.[19]

For each patient encounter identified in the PHIS database, detailed medical record review was performed by trained investigators at each of the 8 study sites (see Supporting Information, Appendix, in the online version of this article). A standardized data collection instrument was pilot tested by all investigators prior to use. Data were collected and managed using the Research Electronic Data Capture (REDCap) tool hosted at Boston Children's Hospital.[20]

Exclusions

Using PHIS data, prior to medical record review we excluded infants with a complex chronic condition as defined previously[21] and those transferred from another institution, as these infants may warrant a nonstandard evaluation and/or may have incomplete data.

ICD‐9 Diagnosis Code Groups

In the PHIS database, all patients discharged from the hospital (including hospitalized patients as well as patients discharged from the ED) receive 1 or more ICD‐9 discharge diagnosis codes. These diagnosis codes are ascribed after discharge from the hospital, or for ED patients, after ED discharge. Additionally, patients may receive an admission diagnosis, which reflects the diagnosis ascribed at the time of ED discharge or transfer to the inpatient unit.

We reviewed medical records of infants selected from the following ICD‐9 diagnosis code groups (Figure 1): (1) discharge diagnosis code of fever (780.6 [fever and other physiologic disturbances of temperature regulation], 778.4 [other disturbances of temperature regulation of newborn], 780.60 [fever, unspecified], or 780.61 [fever presenting with conditions classified elsewhere])[9, 10] regardless of the presence of admission diagnosis of fever or diagnosis of serious infection, (2) admission diagnosis code of fever without associated discharge diagnosis code of fever,[10] (3) discharge diagnosis code of serious infection determined a priori (see Supporting Information, Appendix, in the online version of this article) without discharge or admission diagnosis code of fever, and (4) infants without any diagnosis code of fever or serious infection.

Figure 1

Medical records reviewed in each of the 4 ICD‐9 diagnosis code groups were randomly selected from the overall set of ED encounters in the population of infants <90 days of age evaluated during the study period. Twenty‐five percent population sampling was used for 3 of the ICD‐9 diagnosis code groups, whereas 5% sampling was used for the no fever/no serious infection code group. The number of medical records reviewed in each ICD‐9 diagnosis code group was proportional to the distribution of ICD‐9 codes across the entire population of infants <90 days of age. These records were distributed equally across sites (228 records per site), except for 1 site that does not assign admission diagnoses (201 records).

Investigators were blinded to ICD‐9 diagnosis code groups during medical record review. Infants with multiple visits during the study period were eligible to be included more than once if the visits occurred more than 3 days apart. For infants with more than 1 ED visit on a particular calendar day, investigators were instructed to review the initial visit.

For each encounter, we also abstracted demographic characteristics (gender, race/ethnicity), insurance status, hospital region (using US Census categories[22]), and season from the PHIS database.

Reference Standard

The presence of fever was determined by medical record review. We defined fever as any documented temperature 100.4F (38.0C) at home or in the ED.[16]

ICD‐9 Code Case‐Identification Algorithms

Using the aforementioned ICD‐9 diagnosis code groups individually and in combination, the following 4 case‐identification algorithms, determined from prior study or group consensus, were compared to the reference standard: (1) ICD‐9 discharge diagnosis code of fever,[9] (2) ICD‐9 admission or discharge diagnosis code of fever,[10, 11] (3) ICD‐9 discharge diagnosis code of fever or serious infection, and (4) ICD‐9 discharge or admission diagnosis code of fever or serious infection. Algorithms were compared overall, separately for discharged and hospitalized infants, and across 3 distinct age groups (28 days, 2956 days, and 5789 days).

Patient‐Level Outcomes

To compare differences in outcomes by case‐identification algorithm, from the PHIS database we abstracted hospitalization rates, rates of UTI/pyelonephritis,[13] bacteremia/sepsis, and bacterial meningitis.[19] Severe outcomes were defined as intensive care unit admission, mechanical ventilation, central line placement, receipt of extracorporeal membrane oxygenation, or death. We assessed hospital length of stay for admitted infants and 3‐day revisits,[23, 24] and revisits resulting in hospitalization for infants discharged from the ED at the index visit. Patients billed for observation care were classified as being hospitalized.[25, 26]

Data Analysis

Accuracy of the 4 case‐identification algorithms (compared with the reference standard) was calculated using sensitivity, specificity, negative predictive value (NPV), and positive predictive value (PPV), along with 95% confidence interval (CI). Prior to analysis, a 5‐fold weighting factor was applied to the no fever/no serious infection group to account for the differential sampling used for this group (5% vs 25% for the other 3 ICD‐9 diagnosis code groups). This weighting was done to approximate the true prevalence of each ICD‐9 code group within the larger population, so that an accurate rate of false negatives (infants with fever who had neither a diagnosis of fever nor serious infection) could be calculated.

We described continuous variables using median and interquartile range or range values and categorical variables using frequencies with 95% CIs. We compared categorical variables using a ² test. We determined statistical significance as a 2‐tailed P value <0.05. Statistical analyses were performed using SAS version 9.3 (SAS Institute, Cary, NC).

Study Patients

During the 1‐year study period, 23,753 ED encounters for infants <90 days of age were identified in the PHIS database at the 8 participating sites. Of these infant encounters, 2166 (9.2%) were excluded (1658 infants who had a complex chronic condition and 508 transferred into the ED), leaving 21,587 infants available for selection. After applying our sampling strategy, we identified 1797 encounters for medical record review. Seven encounters from 3 hospitals with missing medical records were excluded, resulting in a final cohort of 1790 encounters (Figure 1). Among included infants, 552 (30.8%) were 28 days, 743 (41.5%) were 29 to 56 days, and 495 (27.8%) were 57 to 89 days of age; 737 (41.2%) infants were hospitalized. Patients differed in age, race, payer, and season across ICD‐9 diagnosis code groups (see Supporting Information, Table 1, in the online version of this article).

Among the 1790 patient encounters reviewed, a total of 766 infants (42.8%) met the reference standard definition for fever in the cohort. An additional 47 infants had abnormal temperature reported (documentation of tactile fever, history of fever without a specific temperature described, or hypothermia) but were classified as having no fever by the reference standard.

ICD‐9 Code Case‐Identification Algorithm Performance

Compared with the reference standard, the 4 case‐identification algorithms demonstrated specificity of 96.2% to 98.2% but lower sensitivity overall (Figure 2). Discharge diagnosis of fever alone demonstrated the lowest sensitivity. The algorithm of discharge or admission diagnosis of fever resulted in increased sensitivity and the highest PPV of all 4 algorithms (86.9%, 95% CI: 84.5‐89.3). Addition of serious infection codes to this algorithm resulted in a marginal increase in sensitivity and a similar decrease in PPV (Table 1). When limited to hospitalized infants, specificity was highest for the case‐identification algorithm of discharge diagnosis of fever and similarly high for discharge or admission diagnosis of fever; sensitivity was highest for the algorithm of discharge or admission diagnosis of fever or diagnosis of serious infection. For infants discharged from the ED, algorithm specificity was 97.7% to 98.4%, with lower sensitivity for all 4 algorithms (Table 1). Inclusion of the 47 infants with abnormal temperature as fever did not materially change algorithm performance (data not shown).

Figure 2

Across all 3 age groups (28 days, 2956 days, and 5789 days), the 4 case‐identification algorithms demonstrated specificity >96%, whereas algorithm sensitivity was highest in the 29‐ to 56‐days‐old age group and lowest among infants 57 to 89 days old across all 4 algorithms (Figure 2). Similar to the overall cohort, an algorithm of discharge or admission diagnosis of fever demonstrated specificity of nearly 98% in all age groups; addition of serious infection codes to this algorithm increased sensitivity, highest in the 29‐ to 56‐days‐old age group (Figure 2; see also Supporting Information, Table 2, in the online version of this article).

Across the 8 study sites, median specificity was 95.8% to 98.3% for the 4 algorithms, with little interhospital variability; however, algorithm sensitivity varied widely by site. Median PPV was highest for discharge diagnosis of fever alone at 87.7% but ranged from 74.0% to 93.2% across sites. Median PPV for an algorithm of discharge or admission diagnosis of fever was similar (87.4%) but with less variation by site (range 80.0%92.9%) (Table 2).

Outcomes by ICD‐9 Diagnosis Code Group and Case‐Identification Algorithm

When compared with discharge diagnosis of fever, adding admission diagnosis of fever captured a higher proportion of hospitalized infants with SBIs (UTI/pyelonephritis, bacteremia/sepsis, or bacterial meningitis). However, median hospital length of stay, severe outcomes, and 3‐day revisits and revisits with hospitalization did not materially differ when including infants with admission diagnosis of fever in addition to discharge diagnosis of fever. Addition of infants with a diagnosis code for serious infection substantially increased the number of infants with SBIs and severe outcomes but did not capture additional 3‐day revisits (Table 3). There were no additional cases of SBI in the no fever/no serious illness diagnosis code group.

Among infants who met the reference standard for fever but did not have a discharge or admission diagnosis of fever (false negatives), 11.8% had a diagnosis of SBI. Overall, 43.2% of febrile infants (and 84.4% of hospitalized infants) with SBI did not have an ICD‐9 discharge or admission diagnosis of fever. Addition of ICD‐9 diagnosis codes of serious infection to the algorithm of discharge or admission diagnosis of fever captured all additional SBIs, and no false negativeinfants missed with this algorithm had an SBI.

Study Design and Setting

For this multicenter retrospective study, we used the Pediatric Health Information System (PHIS) database to identify infants <90 days of age[16] who presented between July 1, 2012 and June 30, 2013 to 1 of 8 EDs. We assessed performance characteristics of ICD‐9 diagnosis code case‐identification algorithms by comparing ICD‐9 code combinations to a fever reference standard determined by medical record review. The institutional review board at each participating site approved the study protocol.

Data Source

Data were obtained from 2 sources: the PHIS database and medical record review. We used the PHIS database to identify eligible patients by ICD‐9 diagnosis codes; patient encounters were randomly selected using a random number generator. The PHIS database contains demographic, diagnosis, and billing data from 44 hospitals affiliated with the Children's Hospital Association (Overland Park, Kansas) and represents 85% of freestanding children's hospitals in the United States.[17] Data are deidentified; encrypted unique patient identifiers permit tracking of patients across visits within a site.[18] The Children's Hospital Association and participating hospitals jointly assure the quality and integrity of the data.[19]

For each patient encounter identified in the PHIS database, detailed medical record review was performed by trained investigators at each of the 8 study sites (see Supporting Information, Appendix, in the online version of this article). A standardized data collection instrument was pilot tested by all investigators prior to use. Data were collected and managed using the Research Electronic Data Capture (REDCap) tool hosted at Boston Children's Hospital.[20]

Exclusions

Using PHIS data, prior to medical record review we excluded infants with a complex chronic condition as defined previously[21] and those transferred from another institution, as these infants may warrant a nonstandard evaluation and/or may have incomplete data.

ICD‐9 Diagnosis Code Groups

In the PHIS database, all patients discharged from the hospital (including hospitalized patients as well as patients discharged from the ED) receive 1 or more ICD‐9 discharge diagnosis codes. These diagnosis codes are ascribed after discharge from the hospital, or for ED patients, after ED discharge. Additionally, patients may receive an admission diagnosis, which reflects the diagnosis ascribed at the time of ED discharge or transfer to the inpatient unit.

We reviewed medical records of infants selected from the following ICD‐9 diagnosis code groups (Figure 1): (1) discharge diagnosis code of fever (780.6 [fever and other physiologic disturbances of temperature regulation], 778.4 [other disturbances of temperature regulation of newborn], 780.60 [fever, unspecified], or 780.61 [fever presenting with conditions classified elsewhere])[9, 10] regardless of the presence of admission diagnosis of fever or diagnosis of serious infection, (2) admission diagnosis code of fever without associated discharge diagnosis code of fever,[10] (3) discharge diagnosis code of serious infection determined a priori (see Supporting Information, Appendix, in the online version of this article) without discharge or admission diagnosis code of fever, and (4) infants without any diagnosis code of fever or serious infection.

Figure 1

Medical records reviewed in each of the 4 ICD‐9 diagnosis code groups were randomly selected from the overall set of ED encounters in the population of infants <90 days of age evaluated during the study period. Twenty‐five percent population sampling was used for 3 of the ICD‐9 diagnosis code groups, whereas 5% sampling was used for the no fever/no serious infection code group. The number of medical records reviewed in each ICD‐9 diagnosis code group was proportional to the distribution of ICD‐9 codes across the entire population of infants <90 days of age. These records were distributed equally across sites (228 records per site), except for 1 site that does not assign admission diagnoses (201 records).

Investigators were blinded to ICD‐9 diagnosis code groups during medical record review. Infants with multiple visits during the study period were eligible to be included more than once if the visits occurred more than 3 days apart. For infants with more than 1 ED visit on a particular calendar day, investigators were instructed to review the initial visit.

For each encounter, we also abstracted demographic characteristics (gender, race/ethnicity), insurance status, hospital region (using US Census categories[22]), and season from the PHIS database.

Reference Standard

The presence of fever was determined by medical record review. We defined fever as any documented temperature 100.4F (38.0C) at home or in the ED.[16]

ICD‐9 Code Case‐Identification Algorithms

Using the aforementioned ICD‐9 diagnosis code groups individually and in combination, the following 4 case‐identification algorithms, determined from prior study or group consensus, were compared to the reference standard: (1) ICD‐9 discharge diagnosis code of fever,[9] (2) ICD‐9 admission or discharge diagnosis code of fever,[10, 11] (3) ICD‐9 discharge diagnosis code of fever or serious infection, and (4) ICD‐9 discharge or admission diagnosis code of fever or serious infection. Algorithms were compared overall, separately for discharged and hospitalized infants, and across 3 distinct age groups (28 days, 2956 days, and 5789 days).

Patient‐Level Outcomes

To compare differences in outcomes by case‐identification algorithm, from the PHIS database we abstracted hospitalization rates, rates of UTI/pyelonephritis,[13] bacteremia/sepsis, and bacterial meningitis.[19] Severe outcomes were defined as intensive care unit admission, mechanical ventilation, central line placement, receipt of extracorporeal membrane oxygenation, or death. We assessed hospital length of stay for admitted infants and 3‐day revisits,[23, 24] and revisits resulting in hospitalization for infants discharged from the ED at the index visit. Patients billed for observation care were classified as being hospitalized.[25, 26]

Data Analysis

Accuracy of the 4 case‐identification algorithms (compared with the reference standard) was calculated using sensitivity, specificity, negative predictive value (NPV), and positive predictive value (PPV), along with 95% confidence interval (CI). Prior to analysis, a 5‐fold weighting factor was applied to the no fever/no serious infection group to account for the differential sampling used for this group (5% vs 25% for the other 3 ICD‐9 diagnosis code groups). This weighting was done to approximate the true prevalence of each ICD‐9 code group within the larger population, so that an accurate rate of false negatives (infants with fever who had neither a diagnosis of fever nor serious infection) could be calculated.

We described continuous variables using median and interquartile range or range values and categorical variables using frequencies with 95% CIs. We compared categorical variables using a ² test. We determined statistical significance as a 2‐tailed P value <0.05. Statistical analyses were performed using SAS version 9.3 (SAS Institute, Cary, NC).

Study Patients

During the 1‐year study period, 23,753 ED encounters for infants <90 days of age were identified in the PHIS database at the 8 participating sites. Of these infant encounters, 2166 (9.2%) were excluded (1658 infants who had a complex chronic condition and 508 transferred into the ED), leaving 21,587 infants available for selection. After applying our sampling strategy, we identified 1797 encounters for medical record review. Seven encounters from 3 hospitals with missing medical records were excluded, resulting in a final cohort of 1790 encounters (Figure 1). Among included infants, 552 (30.8%) were 28 days, 743 (41.5%) were 29 to 56 days, and 495 (27.8%) were 57 to 89 days of age; 737 (41.2%) infants were hospitalized. Patients differed in age, race, payer, and season across ICD‐9 diagnosis code groups (see Supporting Information, Table 1, in the online version of this article).

Among the 1790 patient encounters reviewed, a total of 766 infants (42.8%) met the reference standard definition for fever in the cohort. An additional 47 infants had abnormal temperature reported (documentation of tactile fever, history of fever without a specific temperature described, or hypothermia) but were classified as having no fever by the reference standard.

ICD‐9 Code Case‐Identification Algorithm Performance

Compared with the reference standard, the 4 case‐identification algorithms demonstrated specificity of 96.2% to 98.2% but lower sensitivity overall (Figure 2). Discharge diagnosis of fever alone demonstrated the lowest sensitivity. The algorithm of discharge or admission diagnosis of fever resulted in increased sensitivity and the highest PPV of all 4 algorithms (86.9%, 95% CI: 84.5‐89.3). Addition of serious infection codes to this algorithm resulted in a marginal increase in sensitivity and a similar decrease in PPV (Table 1). When limited to hospitalized infants, specificity was highest for the case‐identification algorithm of discharge diagnosis of fever and similarly high for discharge or admission diagnosis of fever; sensitivity was highest for the algorithm of discharge or admission diagnosis of fever or diagnosis of serious infection. For infants discharged from the ED, algorithm specificity was 97.7% to 98.4%, with lower sensitivity for all 4 algorithms (Table 1). Inclusion of the 47 infants with abnormal temperature as fever did not materially change algorithm performance (data not shown).

Figure 2

Across all 3 age groups (28 days, 2956 days, and 5789 days), the 4 case‐identification algorithms demonstrated specificity >96%, whereas algorithm sensitivity was highest in the 29‐ to 56‐days‐old age group and lowest among infants 57 to 89 days old across all 4 algorithms (Figure 2). Similar to the overall cohort, an algorithm of discharge or admission diagnosis of fever demonstrated specificity of nearly 98% in all age groups; addition of serious infection codes to this algorithm increased sensitivity, highest in the 29‐ to 56‐days‐old age group (Figure 2; see also Supporting Information, Table 2, in the online version of this article).

Across the 8 study sites, median specificity was 95.8% to 98.3% for the 4 algorithms, with little interhospital variability; however, algorithm sensitivity varied widely by site. Median PPV was highest for discharge diagnosis of fever alone at 87.7% but ranged from 74.0% to 93.2% across sites. Median PPV for an algorithm of discharge or admission diagnosis of fever was similar (87.4%) but with less variation by site (range 80.0%92.9%) (Table 2).

Outcomes by ICD‐9 Diagnosis Code Group and Case‐Identification Algorithm

When compared with discharge diagnosis of fever, adding admission diagnosis of fever captured a higher proportion of hospitalized infants with SBIs (UTI/pyelonephritis, bacteremia/sepsis, or bacterial meningitis). However, median hospital length of stay, severe outcomes, and 3‐day revisits and revisits with hospitalization did not materially differ when including infants with admission diagnosis of fever in addition to discharge diagnosis of fever. Addition of infants with a diagnosis code for serious infection substantially increased the number of infants with SBIs and severe outcomes but did not capture additional 3‐day revisits (Table 3). There were no additional cases of SBI in the no fever/no serious illness diagnosis code group.

Among infants who met the reference standard for fever but did not have a discharge or admission diagnosis of fever (false negatives), 11.8% had a diagnosis of SBI. Overall, 43.2% of febrile infants (and 84.4% of hospitalized infants) with SBI did not have an ICD‐9 discharge or admission diagnosis of fever. Addition of ICD‐9 diagnosis codes of serious infection to the algorithm of discharge or admission diagnosis of fever captured all additional SBIs, and no false negativeinfants missed with this algorithm had an SBI.

Study Design and Setting

For this multicenter retrospective study, we used the Pediatric Health Information System (PHIS) database to identify infants <90 days of age[16] who presented between July 1, 2012 and June 30, 2013 to 1 of 8 EDs. We assessed performance characteristics of ICD‐9 diagnosis code case‐identification algorithms by comparing ICD‐9 code combinations to a fever reference standard determined by medical record review. The institutional review board at each participating site approved the study protocol.

Data Source

Data were obtained from 2 sources: the PHIS database and medical record review. We used the PHIS database to identify eligible patients by ICD‐9 diagnosis codes; patient encounters were randomly selected using a random number generator. The PHIS database contains demographic, diagnosis, and billing data from 44 hospitals affiliated with the Children's Hospital Association (Overland Park, Kansas) and represents 85% of freestanding children's hospitals in the United States.[17] Data are deidentified; encrypted unique patient identifiers permit tracking of patients across visits within a site.[18] The Children's Hospital Association and participating hospitals jointly assure the quality and integrity of the data.[19]

For each patient encounter identified in the PHIS database, detailed medical record review was performed by trained investigators at each of the 8 study sites (see Supporting Information, Appendix, in the online version of this article). A standardized data collection instrument was pilot tested by all investigators prior to use. Data were collected and managed using the Research Electronic Data Capture (REDCap) tool hosted at Boston Children's Hospital.[20]

Exclusions

Using PHIS data, prior to medical record review we excluded infants with a complex chronic condition as defined previously[21] and those transferred from another institution, as these infants may warrant a nonstandard evaluation and/or may have incomplete data.

ICD‐9 Diagnosis Code Groups

In the PHIS database, all patients discharged from the hospital (including hospitalized patients as well as patients discharged from the ED) receive 1 or more ICD‐9 discharge diagnosis codes. These diagnosis codes are ascribed after discharge from the hospital, or for ED patients, after ED discharge. Additionally, patients may receive an admission diagnosis, which reflects the diagnosis ascribed at the time of ED discharge or transfer to the inpatient unit.

We reviewed medical records of infants selected from the following ICD‐9 diagnosis code groups (Figure 1): (1) discharge diagnosis code of fever (780.6 [fever and other physiologic disturbances of temperature regulation], 778.4 [other disturbances of temperature regulation of newborn], 780.60 [fever, unspecified], or 780.61 [fever presenting with conditions classified elsewhere])[9, 10] regardless of the presence of admission diagnosis of fever or diagnosis of serious infection, (2) admission diagnosis code of fever without associated discharge diagnosis code of fever,[10] (3) discharge diagnosis code of serious infection determined a priori (see Supporting Information, Appendix, in the online version of this article) without discharge or admission diagnosis code of fever, and (4) infants without any diagnosis code of fever or serious infection.

Figure 1

Medical records reviewed in each of the 4 ICD‐9 diagnosis code groups were randomly selected from the overall set of ED encounters in the population of infants <90 days of age evaluated during the study period. Twenty‐five percent population sampling was used for 3 of the ICD‐9 diagnosis code groups, whereas 5% sampling was used for the no fever/no serious infection code group. The number of medical records reviewed in each ICD‐9 diagnosis code group was proportional to the distribution of ICD‐9 codes across the entire population of infants <90 days of age. These records were distributed equally across sites (228 records per site), except for 1 site that does not assign admission diagnoses (201 records).

Investigators were blinded to ICD‐9 diagnosis code groups during medical record review. Infants with multiple visits during the study period were eligible to be included more than once if the visits occurred more than 3 days apart. For infants with more than 1 ED visit on a particular calendar day, investigators were instructed to review the initial visit.

For each encounter, we also abstracted demographic characteristics (gender, race/ethnicity), insurance status, hospital region (using US Census categories[22]), and season from the PHIS database.

Reference Standard

The presence of fever was determined by medical record review. We defined fever as any documented temperature 100.4F (38.0C) at home or in the ED.[16]

ICD‐9 Code Case‐Identification Algorithms

Using the aforementioned ICD‐9 diagnosis code groups individually and in combination, the following 4 case‐identification algorithms, determined from prior study or group consensus, were compared to the reference standard: (1) ICD‐9 discharge diagnosis code of fever,[9] (2) ICD‐9 admission or discharge diagnosis code of fever,[10, 11] (3) ICD‐9 discharge diagnosis code of fever or serious infection, and (4) ICD‐9 discharge or admission diagnosis code of fever or serious infection. Algorithms were compared overall, separately for discharged and hospitalized infants, and across 3 distinct age groups (28 days, 2956 days, and 5789 days).

Patient‐Level Outcomes

To compare differences in outcomes by case‐identification algorithm, from the PHIS database we abstracted hospitalization rates, rates of UTI/pyelonephritis,[13] bacteremia/sepsis, and bacterial meningitis.[19] Severe outcomes were defined as intensive care unit admission, mechanical ventilation, central line placement, receipt of extracorporeal membrane oxygenation, or death. We assessed hospital length of stay for admitted infants and 3‐day revisits,[23, 24] and revisits resulting in hospitalization for infants discharged from the ED at the index visit. Patients billed for observation care were classified as being hospitalized.[25, 26]

Data Analysis

Accuracy of the 4 case‐identification algorithms (compared with the reference standard) was calculated using sensitivity, specificity, negative predictive value (NPV), and positive predictive value (PPV), along with 95% confidence interval (CI). Prior to analysis, a 5‐fold weighting factor was applied to the no fever/no serious infection group to account for the differential sampling used for this group (5% vs 25% for the other 3 ICD‐9 diagnosis code groups). This weighting was done to approximate the true prevalence of each ICD‐9 code group within the larger population, so that an accurate rate of false negatives (infants with fever who had neither a diagnosis of fever nor serious infection) could be calculated.

We described continuous variables using median and interquartile range or range values and categorical variables using frequencies with 95% CIs. We compared categorical variables using a ² test. We determined statistical significance as a 2‐tailed P value <0.05. Statistical analyses were performed using SAS version 9.3 (SAS Institute, Cary, NC).

Study Patients

During the 1‐year study period, 23,753 ED encounters for infants <90 days of age were identified in the PHIS database at the 8 participating sites. Of these infant encounters, 2166 (9.2%) were excluded (1658 infants who had a complex chronic condition and 508 transferred into the ED), leaving 21,587 infants available for selection. After applying our sampling strategy, we identified 1797 encounters for medical record review. Seven encounters from 3 hospitals with missing medical records were excluded, resulting in a final cohort of 1790 encounters (Figure 1). Among included infants, 552 (30.8%) were 28 days, 743 (41.5%) were 29 to 56 days, and 495 (27.8%) were 57 to 89 days of age; 737 (41.2%) infants were hospitalized. Patients differed in age, race, payer, and season across ICD‐9 diagnosis code groups (see Supporting Information, Table 1, in the online version of this article).

Among the 1790 patient encounters reviewed, a total of 766 infants (42.8%) met the reference standard definition for fever in the cohort. An additional 47 infants had abnormal temperature reported (documentation of tactile fever, history of fever without a specific temperature described, or hypothermia) but were classified as having no fever by the reference standard.

ICD‐9 Code Case‐Identification Algorithm Performance

Compared with the reference standard, the 4 case‐identification algorithms demonstrated specificity of 96.2% to 98.2% but lower sensitivity overall (Figure 2). Discharge diagnosis of fever alone demonstrated the lowest sensitivity. The algorithm of discharge or admission diagnosis of fever resulted in increased sensitivity and the highest PPV of all 4 algorithms (86.9%, 95% CI: 84.5‐89.3). Addition of serious infection codes to this algorithm resulted in a marginal increase in sensitivity and a similar decrease in PPV (Table 1). When limited to hospitalized infants, specificity was highest for the case‐identification algorithm of discharge diagnosis of fever and similarly high for discharge or admission diagnosis of fever; sensitivity was highest for the algorithm of discharge or admission diagnosis of fever or diagnosis of serious infection. For infants discharged from the ED, algorithm specificity was 97.7% to 98.4%, with lower sensitivity for all 4 algorithms (Table 1). Inclusion of the 47 infants with abnormal temperature as fever did not materially change algorithm performance (data not shown).

Figure 2

Across all 3 age groups (28 days, 2956 days, and 5789 days), the 4 case‐identification algorithms demonstrated specificity >96%, whereas algorithm sensitivity was highest in the 29‐ to 56‐days‐old age group and lowest among infants 57 to 89 days old across all 4 algorithms (Figure 2). Similar to the overall cohort, an algorithm of discharge or admission diagnosis of fever demonstrated specificity of nearly 98% in all age groups; addition of serious infection codes to this algorithm increased sensitivity, highest in the 29‐ to 56‐days‐old age group (Figure 2; see also Supporting Information, Table 2, in the online version of this article).

Across the 8 study sites, median specificity was 95.8% to 98.3% for the 4 algorithms, with little interhospital variability; however, algorithm sensitivity varied widely by site. Median PPV was highest for discharge diagnosis of fever alone at 87.7% but ranged from 74.0% to 93.2% across sites. Median PPV for an algorithm of discharge or admission diagnosis of fever was similar (87.4%) but with less variation by site (range 80.0%92.9%) (Table 2).

Outcomes by ICD‐9 Diagnosis Code Group and Case‐Identification Algorithm

When compared with discharge diagnosis of fever, adding admission diagnosis of fever captured a higher proportion of hospitalized infants with SBIs (UTI/pyelonephritis, bacteremia/sepsis, or bacterial meningitis). However, median hospital length of stay, severe outcomes, and 3‐day revisits and revisits with hospitalization did not materially differ when including infants with admission diagnosis of fever in addition to discharge diagnosis of fever. Addition of infants with a diagnosis code for serious infection substantially increased the number of infants with SBIs and severe outcomes but did not capture additional 3‐day revisits (Table 3). There were no additional cases of SBI in the no fever/no serious illness diagnosis code group.

Among infants who met the reference standard for fever but did not have a discharge or admission diagnosis of fever (false negatives), 11.8% had a diagnosis of SBI. Overall, 43.2% of febrile infants (and 84.4% of hospitalized infants) with SBI did not have an ICD‐9 discharge or admission diagnosis of fever. Addition of ICD‐9 diagnosis codes of serious infection to the algorithm of discharge or admission diagnosis of fever captured all additional SBIs, and no false negativeinfants missed with this algorithm had an SBI.

Hospital Selection and Hospital Characteristics

The study sample was drawn from the 31 hospitals that reported observation‐status patient data to PHIS in 2011. Analyses were conducted in 2013, at which time 2011 was the most recent year of data. We categorized 14 hospitals as having a dedicated OU during 2011 based on information collected in 2013.⁷ To summarize briefly, we interviewed by telephone representatives of hospitals responding to an email query as to the presence of a geographically distinct OU for the care of unscheduled patients from the ED. Three of the 14 representatives reported their hospital had 2 OUs, 1 of which was a separate surgical OU. Ten OUs cared for both ED patients and patients with scheduled procedures; 8 units received patients from non‐ED sources. Hospitalists provided staffing in more than half of the OUs.

We attempted to identify administrative data that would signal care delivered in a dedicated OU using hospital charge codes reported to PHIS, but learned this was not possible due to between‐hospital variation in the specificity of the charge codes. Therefore, we were unable to determine if patient care was delivered in a dedicated OU or another setting, such as a general inpatient unit or the ED. Other hospital characteristics available from the PHIS dataset included the number of inpatient beds, ED visits, inpatient admissions, observation‐status stays, and payer mix. We calculated the percentage of ED visits resulting in admission by dividing the number of ED visits with associated inpatient or observation status by the total number of ED visits and the percentage of admissions under observation status by dividing the number of observation‐status stays by the total number of admissions under observation or inpatient status.

Visit Selection and Patient Characteristics

All observation‐status stays regardless of the point of entry into the hospital were eligible for this study. We excluded stays that were birth‐related, included intensive care, or resulted in transfer or death. Patient demographic characteristics used to describe the cohort included age, gender, race/ethnicity, and primary payer. Stays that began in the ED were identified by an emergency room charge within PHIS. Eligible stays were categorized using All Patient Refined Diagnosis Related Groups (APR‐DRGs) version 24 using the ICD‐9‐CM code‐based proprietary 3M software (3M Health Information Systems, St. Paul, MN). We determined the 15 top‐ranking APR‐DRGs among observation‐status stays in hospitals with a dedicated OU and hospitals without. Procedural stays were identified based on procedural APR‐DRGs (eg, tonsil and adenoid procedures) or the presence of an ICD‐9‐CM procedure code (eg, 331 spinal tap).

Measured Outcomes

Outcomes of observation‐status stays were determined within 4 categories: (1) LOS, (2) standardized costs, (3) conversion to inpatient status, and (4) return visits and readmissions. LOS was calculated in terms of nights spent in hospital for all stays by subtracting the discharge date from the admission date and in terms of hours for stays in the 28 hospitals that report admission and discharge hour to the PHIS database. Discharge timing was examined in 4, 6‐hour blocks starting at midnight. Standardized costs were derived from a charge master index that was created by taking the median costs from all PHIS hospitals for each charged service.[17] Standardized costs represent the estimated cost of providing any particular clinical activity but are not the cost to patients, nor do they represent the actual cost to any given hospital. This approach allows for cost comparisons across hospitals, without biases arising from using charges or from deriving costs using hospitals' ratios of costs to charges.[18] Conversion from observation to inpatient status was calculated by dividing the number of inpatient‐status stays with observation codes by the number of observation‐statusonly stays plus the number of inpatient‐status stays with observation codes. All‐cause 3‐day ED return visits and 30‐day readmissions to the same hospital were assessed using patient‐specific identifiers that allowed for tracking of ED return visits and readmissions following the index observation stay.

Data Analysis

Descriptive statistics were calculated for hospital and patient characteristics using medians and interquartile ranges (IQRs) for continuous factors and frequencies with percentages for categorical factors. Comparisons of these factors between hospitals with dedicated OUs and without were made using [2] and Wilcoxon rank sum tests as appropriate. Multivariable regression was performed using generalized linear mixed models treating hospital as a random effect and used patient age, the case‐mix index based on the APR‐DRG severity of illness, ED visit, and procedures associated with the index observation‐status stay. For continuous outcomes, we performed a log transformation on the outcome, confirmed the normality assumption, and back transformed the results. Sensitivity analyses were conducted to compare LOS, standardized costs, and conversation rates by hospital type for 10 of the 15 top‐ranking APR‐DRGs commonly cared for by pediatric hospitalists and to compare hospitals that reported the presence of an OU that was consistently open (24 hours per day, 7 days per week) and operating during the entire 2011 calendar year, and those without. Based on information gathered from the telephone interviews, hospitals with partially open OUs were similar to hospitals with continuously open OUs, such that they were included in our main analyses. All statistical analyses were performed using SAS version 9.3 (SAS Institute, Cary, NC). P values <0.05 were considered statistically significant.

Hospital Characteristics

Dedicated OUs were present in 14 of the 31 hospitals that reported observation‐status patient data to PHIS (Figure 1). Three of these hospitals had OUs that were open for 5 months or less in 2011; 1 unit opened, 1 unit closed, and 1 hospital operated a seasonal unit. The remaining 17 hospitals reported no OU that admitted unscheduled patients from the ED during 2011. Hospitals with a dedicated OU had more inpatient beds and higher median number of inpatient admissions than those without (Table 1). Hospitals were statistically similar in terms of total volume of ED visits, percentage of ED visits resulting in admission, total number of observation‐status stays, percentage of admissions under observation status, and payer mix.

Figure 1

Observation‐Status Patients by Hospital Type

In 2011, there were a total of 136,239 observation‐status stays69,983 (51.4%) within the 14 hospitals with a dedicated OU and 66,256 (48.6%) within the 17 hospitals without. Patient care originated in the ED for 57.8% observation‐status stays in hospitals with an OU compared with 53.0% of observation‐status stays in hospitals without (P<0.001). Compared with hospitals with a dedicated OU, those without a dedicated OU had higher percentages of observation‐status patients older than 12 years and non‐Hispanic and a higher percentage of observation‐status patients with private payer type (Table 2). The 15 top‐ranking APR‐DRGs accounted for roughly half of all observation‐status stays and were relatively consistent between hospitals with and without a dedicated OU (Table 3). Procedural care was frequently associated with observation‐status stays.

Outcomes of Observation‐Status Stays

A greater percentage of observation‐status stays in hospitals with a dedicated OU experienced a same‐day discharge (Table 4). In addition, a higher percentage of discharges occurred between midnight and 11 _am in hospitals with a dedicated OU. However, overall risk‐adjusted LOS in hours (12.8 vs 12.2 hours, P=0.90) and risk‐adjusted total standardized costs ($2551 vs $2433, P=0.75) were similar between hospital types. These findings were consistent within the 1 APR‐DRGs commonly cared for by pediatric hospitalists (see Supporting Information, Appendix 1, in the online version of this article). Overall, conversion from observation to inpatient status was significantly higher in hospitals with a dedicated OU compared with hospitals without; however, this pattern was not consistent across the 10 APR‐DRGs commonly cared for by pediatric hospitalists (see Supporting Information, Appendix 1, in the online version of this article). Adjusted odds of 3‐day ED return visits and 30‐day readmissions were comparable between hospital groups.

We found similar results in sensitivity analyses comparing observation‐status stays in hospitals with a continuously open OU (open 24 hours per day, 7 days per week, for all of 2011 [n=10 hospitals]) to those without(see Supporting Information, Appendix 2, in the online version of this article). However, there were, on average, more observation‐status stays in hospitals with a continuously open OU (median 5605, IQR 42077089) than hospitals without (median 3309, IQR 26784616) (P=0.04). In contrast to our main results, conversion to inpatient status was lower in hospitals with a continuously open OU compared with hospitals without (8.52% vs 11.57%, P<0.01).

Limitations

The primary limitation to our analyses is the lack of ability to identify patients who were treated in a dedicated OU because few hospitals provided data to PHIS that allowed for the identification of the unit or location of care. Second, it is possible that some hospitals were misclassified as not having a dedicated OU based on our survey, which initially inquired about OUs that provided care to patients first treated in the ED. Therefore, OUs that exclusively care for postoperative patients or patients with scheduled treatments may be present in hospitals that we have labeled as not having a dedicated OU. This potential misclassification would bias our results toward finding no differences. Third, in any study of administrative data there is potential that diagnosis codes are incomplete or inaccurately capture the underlying reason for the episode of care. Fourth, the experiences of the free‐standing children's hospitals that contribute data to PHIS may not be generalizable to other hospitals that provide observation care to children. Finally, return care may be underestimated, as children could receive treatment at another hospital following discharge from a PHIS hospital. Care outside of PHIS hospitals would not be captured, but we do not expect this to differ for hospitals with and without dedicated OUs. It is possible that health information exchanges will permit more comprehensive analyses of care across different hospitals in the future.

Hospital Selection and Hospital Characteristics

The study sample was drawn from the 31 hospitals that reported observation‐status patient data to PHIS in 2011. Analyses were conducted in 2013, at which time 2011 was the most recent year of data. We categorized 14 hospitals as having a dedicated OU during 2011 based on information collected in 2013.⁷ To summarize briefly, we interviewed by telephone representatives of hospitals responding to an email query as to the presence of a geographically distinct OU for the care of unscheduled patients from the ED. Three of the 14 representatives reported their hospital had 2 OUs, 1 of which was a separate surgical OU. Ten OUs cared for both ED patients and patients with scheduled procedures; 8 units received patients from non‐ED sources. Hospitalists provided staffing in more than half of the OUs.

We attempted to identify administrative data that would signal care delivered in a dedicated OU using hospital charge codes reported to PHIS, but learned this was not possible due to between‐hospital variation in the specificity of the charge codes. Therefore, we were unable to determine if patient care was delivered in a dedicated OU or another setting, such as a general inpatient unit or the ED. Other hospital characteristics available from the PHIS dataset included the number of inpatient beds, ED visits, inpatient admissions, observation‐status stays, and payer mix. We calculated the percentage of ED visits resulting in admission by dividing the number of ED visits with associated inpatient or observation status by the total number of ED visits and the percentage of admissions under observation status by dividing the number of observation‐status stays by the total number of admissions under observation or inpatient status.

Visit Selection and Patient Characteristics

All observation‐status stays regardless of the point of entry into the hospital were eligible for this study. We excluded stays that were birth‐related, included intensive care, or resulted in transfer or death. Patient demographic characteristics used to describe the cohort included age, gender, race/ethnicity, and primary payer. Stays that began in the ED were identified by an emergency room charge within PHIS. Eligible stays were categorized using All Patient Refined Diagnosis Related Groups (APR‐DRGs) version 24 using the ICD‐9‐CM code‐based proprietary 3M software (3M Health Information Systems, St. Paul, MN). We determined the 15 top‐ranking APR‐DRGs among observation‐status stays in hospitals with a dedicated OU and hospitals without. Procedural stays were identified based on procedural APR‐DRGs (eg, tonsil and adenoid procedures) or the presence of an ICD‐9‐CM procedure code (eg, 331 spinal tap).

Measured Outcomes

Outcomes of observation‐status stays were determined within 4 categories: (1) LOS, (2) standardized costs, (3) conversion to inpatient status, and (4) return visits and readmissions. LOS was calculated in terms of nights spent in hospital for all stays by subtracting the discharge date from the admission date and in terms of hours for stays in the 28 hospitals that report admission and discharge hour to the PHIS database. Discharge timing was examined in 4, 6‐hour blocks starting at midnight. Standardized costs were derived from a charge master index that was created by taking the median costs from all PHIS hospitals for each charged service.[17] Standardized costs represent the estimated cost of providing any particular clinical activity but are not the cost to patients, nor do they represent the actual cost to any given hospital. This approach allows for cost comparisons across hospitals, without biases arising from using charges or from deriving costs using hospitals' ratios of costs to charges.[18] Conversion from observation to inpatient status was calculated by dividing the number of inpatient‐status stays with observation codes by the number of observation‐statusonly stays plus the number of inpatient‐status stays with observation codes. All‐cause 3‐day ED return visits and 30‐day readmissions to the same hospital were assessed using patient‐specific identifiers that allowed for tracking of ED return visits and readmissions following the index observation stay.

Data Analysis

Descriptive statistics were calculated for hospital and patient characteristics using medians and interquartile ranges (IQRs) for continuous factors and frequencies with percentages for categorical factors. Comparisons of these factors between hospitals with dedicated OUs and without were made using [2] and Wilcoxon rank sum tests as appropriate. Multivariable regression was performed using generalized linear mixed models treating hospital as a random effect and used patient age, the case‐mix index based on the APR‐DRG severity of illness, ED visit, and procedures associated with the index observation‐status stay. For continuous outcomes, we performed a log transformation on the outcome, confirmed the normality assumption, and back transformed the results. Sensitivity analyses were conducted to compare LOS, standardized costs, and conversation rates by hospital type for 10 of the 15 top‐ranking APR‐DRGs commonly cared for by pediatric hospitalists and to compare hospitals that reported the presence of an OU that was consistently open (24 hours per day, 7 days per week) and operating during the entire 2011 calendar year, and those without. Based on information gathered from the telephone interviews, hospitals with partially open OUs were similar to hospitals with continuously open OUs, such that they were included in our main analyses. All statistical analyses were performed using SAS version 9.3 (SAS Institute, Cary, NC). P values <0.05 were considered statistically significant.

Hospital Characteristics

Dedicated OUs were present in 14 of the 31 hospitals that reported observation‐status patient data to PHIS (Figure 1). Three of these hospitals had OUs that were open for 5 months or less in 2011; 1 unit opened, 1 unit closed, and 1 hospital operated a seasonal unit. The remaining 17 hospitals reported no OU that admitted unscheduled patients from the ED during 2011. Hospitals with a dedicated OU had more inpatient beds and higher median number of inpatient admissions than those without (Table 1). Hospitals were statistically similar in terms of total volume of ED visits, percentage of ED visits resulting in admission, total number of observation‐status stays, percentage of admissions under observation status, and payer mix.

Figure 1

Observation‐Status Patients by Hospital Type

In 2011, there were a total of 136,239 observation‐status stays69,983 (51.4%) within the 14 hospitals with a dedicated OU and 66,256 (48.6%) within the 17 hospitals without. Patient care originated in the ED for 57.8% observation‐status stays in hospitals with an OU compared with 53.0% of observation‐status stays in hospitals without (P<0.001). Compared with hospitals with a dedicated OU, those without a dedicated OU had higher percentages of observation‐status patients older than 12 years and non‐Hispanic and a higher percentage of observation‐status patients with private payer type (Table 2). The 15 top‐ranking APR‐DRGs accounted for roughly half of all observation‐status stays and were relatively consistent between hospitals with and without a dedicated OU (Table 3). Procedural care was frequently associated with observation‐status stays.

Outcomes of Observation‐Status Stays

A greater percentage of observation‐status stays in hospitals with a dedicated OU experienced a same‐day discharge (Table 4). In addition, a higher percentage of discharges occurred between midnight and 11 _am in hospitals with a dedicated OU. However, overall risk‐adjusted LOS in hours (12.8 vs 12.2 hours, P=0.90) and risk‐adjusted total standardized costs ($2551 vs $2433, P=0.75) were similar between hospital types. These findings were consistent within the 1 APR‐DRGs commonly cared for by pediatric hospitalists (see Supporting Information, Appendix 1, in the online version of this article). Overall, conversion from observation to inpatient status was significantly higher in hospitals with a dedicated OU compared with hospitals without; however, this pattern was not consistent across the 10 APR‐DRGs commonly cared for by pediatric hospitalists (see Supporting Information, Appendix 1, in the online version of this article). Adjusted odds of 3‐day ED return visits and 30‐day readmissions were comparable between hospital groups.

We found similar results in sensitivity analyses comparing observation‐status stays in hospitals with a continuously open OU (open 24 hours per day, 7 days per week, for all of 2011 [n=10 hospitals]) to those without(see Supporting Information, Appendix 2, in the online version of this article). However, there were, on average, more observation‐status stays in hospitals with a continuously open OU (median 5605, IQR 42077089) than hospitals without (median 3309, IQR 26784616) (P=0.04). In contrast to our main results, conversion to inpatient status was lower in hospitals with a continuously open OU compared with hospitals without (8.52% vs 11.57%, P<0.01).

Limitations

The primary limitation to our analyses is the lack of ability to identify patients who were treated in a dedicated OU because few hospitals provided data to PHIS that allowed for the identification of the unit or location of care. Second, it is possible that some hospitals were misclassified as not having a dedicated OU based on our survey, which initially inquired about OUs that provided care to patients first treated in the ED. Therefore, OUs that exclusively care for postoperative patients or patients with scheduled treatments may be present in hospitals that we have labeled as not having a dedicated OU. This potential misclassification would bias our results toward finding no differences. Third, in any study of administrative data there is potential that diagnosis codes are incomplete or inaccurately capture the underlying reason for the episode of care. Fourth, the experiences of the free‐standing children's hospitals that contribute data to PHIS may not be generalizable to other hospitals that provide observation care to children. Finally, return care may be underestimated, as children could receive treatment at another hospital following discharge from a PHIS hospital. Care outside of PHIS hospitals would not be captured, but we do not expect this to differ for hospitals with and without dedicated OUs. It is possible that health information exchanges will permit more comprehensive analyses of care across different hospitals in the future.

Hospital Selection and Hospital Characteristics

The study sample was drawn from the 31 hospitals that reported observation‐status patient data to PHIS in 2011. Analyses were conducted in 2013, at which time 2011 was the most recent year of data. We categorized 14 hospitals as having a dedicated OU during 2011 based on information collected in 2013.⁷ To summarize briefly, we interviewed by telephone representatives of hospitals responding to an email query as to the presence of a geographically distinct OU for the care of unscheduled patients from the ED. Three of the 14 representatives reported their hospital had 2 OUs, 1 of which was a separate surgical OU. Ten OUs cared for both ED patients and patients with scheduled procedures; 8 units received patients from non‐ED sources. Hospitalists provided staffing in more than half of the OUs.

We attempted to identify administrative data that would signal care delivered in a dedicated OU using hospital charge codes reported to PHIS, but learned this was not possible due to between‐hospital variation in the specificity of the charge codes. Therefore, we were unable to determine if patient care was delivered in a dedicated OU or another setting, such as a general inpatient unit or the ED. Other hospital characteristics available from the PHIS dataset included the number of inpatient beds, ED visits, inpatient admissions, observation‐status stays, and payer mix. We calculated the percentage of ED visits resulting in admission by dividing the number of ED visits with associated inpatient or observation status by the total number of ED visits and the percentage of admissions under observation status by dividing the number of observation‐status stays by the total number of admissions under observation or inpatient status.

Visit Selection and Patient Characteristics

All observation‐status stays regardless of the point of entry into the hospital were eligible for this study. We excluded stays that were birth‐related, included intensive care, or resulted in transfer or death. Patient demographic characteristics used to describe the cohort included age, gender, race/ethnicity, and primary payer. Stays that began in the ED were identified by an emergency room charge within PHIS. Eligible stays were categorized using All Patient Refined Diagnosis Related Groups (APR‐DRGs) version 24 using the ICD‐9‐CM code‐based proprietary 3M software (3M Health Information Systems, St. Paul, MN). We determined the 15 top‐ranking APR‐DRGs among observation‐status stays in hospitals with a dedicated OU and hospitals without. Procedural stays were identified based on procedural APR‐DRGs (eg, tonsil and adenoid procedures) or the presence of an ICD‐9‐CM procedure code (eg, 331 spinal tap).

Measured Outcomes

Outcomes of observation‐status stays were determined within 4 categories: (1) LOS, (2) standardized costs, (3) conversion to inpatient status, and (4) return visits and readmissions. LOS was calculated in terms of nights spent in hospital for all stays by subtracting the discharge date from the admission date and in terms of hours for stays in the 28 hospitals that report admission and discharge hour to the PHIS database. Discharge timing was examined in 4, 6‐hour blocks starting at midnight. Standardized costs were derived from a charge master index that was created by taking the median costs from all PHIS hospitals for each charged service.[17] Standardized costs represent the estimated cost of providing any particular clinical activity but are not the cost to patients, nor do they represent the actual cost to any given hospital. This approach allows for cost comparisons across hospitals, without biases arising from using charges or from deriving costs using hospitals' ratios of costs to charges.[18] Conversion from observation to inpatient status was calculated by dividing the number of inpatient‐status stays with observation codes by the number of observation‐statusonly stays plus the number of inpatient‐status stays with observation codes. All‐cause 3‐day ED return visits and 30‐day readmissions to the same hospital were assessed using patient‐specific identifiers that allowed for tracking of ED return visits and readmissions following the index observation stay.

Data Analysis

Descriptive statistics were calculated for hospital and patient characteristics using medians and interquartile ranges (IQRs) for continuous factors and frequencies with percentages for categorical factors. Comparisons of these factors between hospitals with dedicated OUs and without were made using [2] and Wilcoxon rank sum tests as appropriate. Multivariable regression was performed using generalized linear mixed models treating hospital as a random effect and used patient age, the case‐mix index based on the APR‐DRG severity of illness, ED visit, and procedures associated with the index observation‐status stay. For continuous outcomes, we performed a log transformation on the outcome, confirmed the normality assumption, and back transformed the results. Sensitivity analyses were conducted to compare LOS, standardized costs, and conversation rates by hospital type for 10 of the 15 top‐ranking APR‐DRGs commonly cared for by pediatric hospitalists and to compare hospitals that reported the presence of an OU that was consistently open (24 hours per day, 7 days per week) and operating during the entire 2011 calendar year, and those without. Based on information gathered from the telephone interviews, hospitals with partially open OUs were similar to hospitals with continuously open OUs, such that they were included in our main analyses. All statistical analyses were performed using SAS version 9.3 (SAS Institute, Cary, NC). P values <0.05 were considered statistically significant.

Hospital Characteristics

Dedicated OUs were present in 14 of the 31 hospitals that reported observation‐status patient data to PHIS (Figure 1). Three of these hospitals had OUs that were open for 5 months or less in 2011; 1 unit opened, 1 unit closed, and 1 hospital operated a seasonal unit. The remaining 17 hospitals reported no OU that admitted unscheduled patients from the ED during 2011. Hospitals with a dedicated OU had more inpatient beds and higher median number of inpatient admissions than those without (Table 1). Hospitals were statistically similar in terms of total volume of ED visits, percentage of ED visits resulting in admission, total number of observation‐status stays, percentage of admissions under observation status, and payer mix.

Figure 1

Observation‐Status Patients by Hospital Type

In 2011, there were a total of 136,239 observation‐status stays69,983 (51.4%) within the 14 hospitals with a dedicated OU and 66,256 (48.6%) within the 17 hospitals without. Patient care originated in the ED for 57.8% observation‐status stays in hospitals with an OU compared with 53.0% of observation‐status stays in hospitals without (P<0.001). Compared with hospitals with a dedicated OU, those without a dedicated OU had higher percentages of observation‐status patients older than 12 years and non‐Hispanic and a higher percentage of observation‐status patients with private payer type (Table 2). The 15 top‐ranking APR‐DRGs accounted for roughly half of all observation‐status stays and were relatively consistent between hospitals with and without a dedicated OU (Table 3). Procedural care was frequently associated with observation‐status stays.

Outcomes of Observation‐Status Stays

A greater percentage of observation‐status stays in hospitals with a dedicated OU experienced a same‐day discharge (Table 4). In addition, a higher percentage of discharges occurred between midnight and 11 _am in hospitals with a dedicated OU. However, overall risk‐adjusted LOS in hours (12.8 vs 12.2 hours, P=0.90) and risk‐adjusted total standardized costs ($2551 vs $2433, P=0.75) were similar between hospital types. These findings were consistent within the 1 APR‐DRGs commonly cared for by pediatric hospitalists (see Supporting Information, Appendix 1, in the online version of this article). Overall, conversion from observation to inpatient status was significantly higher in hospitals with a dedicated OU compared with hospitals without; however, this pattern was not consistent across the 10 APR‐DRGs commonly cared for by pediatric hospitalists (see Supporting Information, Appendix 1, in the online version of this article). Adjusted odds of 3‐day ED return visits and 30‐day readmissions were comparable between hospital groups.

We found similar results in sensitivity analyses comparing observation‐status stays in hospitals with a continuously open OU (open 24 hours per day, 7 days per week, for all of 2011 [n=10 hospitals]) to those without(see Supporting Information, Appendix 2, in the online version of this article). However, there were, on average, more observation‐status stays in hospitals with a continuously open OU (median 5605, IQR 42077089) than hospitals without (median 3309, IQR 26784616) (P=0.04). In contrast to our main results, conversion to inpatient status was lower in hospitals with a continuously open OU compared with hospitals without (8.52% vs 11.57%, P<0.01).

Limitations

The primary limitation to our analyses is the lack of ability to identify patients who were treated in a dedicated OU because few hospitals provided data to PHIS that allowed for the identification of the unit or location of care. Second, it is possible that some hospitals were misclassified as not having a dedicated OU based on our survey, which initially inquired about OUs that provided care to patients first treated in the ED. Therefore, OUs that exclusively care for postoperative patients or patients with scheduled treatments may be present in hospitals that we have labeled as not having a dedicated OU. This potential misclassification would bias our results toward finding no differences. Third, in any study of administrative data there is potential that diagnosis codes are incomplete or inaccurately capture the underlying reason for the episode of care. Fourth, the experiences of the free‐standing children's hospitals that contribute data to PHIS may not be generalizable to other hospitals that provide observation care to children. Finally, return care may be underestimated, as children could receive treatment at another hospital following discharge from a PHIS hospital. Care outside of PHIS hospitals would not be captured, but we do not expect this to differ for hospitals with and without dedicated OUs. It is possible that health information exchanges will permit more comprehensive analyses of care across different hospitals in the future.

Study Design

We used the Pediatric Health Information System (PHIS) to identify febrile infants 56 days of age who presented to the ED between January 1, 2013 and December 31, 2013. We also surveyed ED providers at participating PHIS hospitals. Informed consent was obtained from survey respondents. The institutional review board at Boston Children's Hospital approved the study protocol.

Clinical Practice Guideline Survey

We sent an electronic survey to medical directors or division directors at 37 pediatric EDs to determine whether their ED utilized a CPG for the management of the febrile young infant in 2013. If no response was received after the second attempt, we queried ED fellowship directors or other ED attending physicians at nonresponding hospitals. Survey items included the presence of a febrile young infant CPG, and if present, the year of implementation, ages targeted, and CPG content. As applicable, respondents were asked to share their CPG and/or provide the specific CPG recommendations.

We collected and managed survey data using the Research Electronic Data Capture (REDCap) electronic data capture tools hosted at Boston Children's Hospital. REDCap is a secure, Web‐based application designed to support data capture for research studies.[13]

Data Source

The PHIS database contains administrative data from 44 US children's hospitals. These hospitals, affiliated with the Children's Hospital Association, represent 85% of freestanding US children's hospitals.[14] Encrypted patient identifiers permit tracking of patients across encounters.[15] Data quality and integrity are assured jointly by the Children's Hospital Association and participating hospitals.[16] For this study, 7 hospitals were excluded due to incomplete ED data or known data‐quality issues.[17]

Patients

We identified study infants using the following International Classification of Diseases, 9th Revision (ICD‐9) admission or discharge diagnosis codes for fever as defined previously[8, 9]: 780.6, 778.4, 780.60, or 780.61. We excluded infants with a complex chronic condition[18] and those transferred from another institution, as these infants may warrant a nonstandard evaluation and/or may have incomplete data. For infants with >1 ED visit for fever during the study period, repeat visits within 3 days of an index visit were considered a revisit for the same episode of illness; visits >3 days following an index visit were considered as a new index visit.

Study Definitions

From the PHIS database, we abstracted demographic characteristics (gender, race/ethnicity), insurance status, and region where the hospital was located (using US Census categories[19]). Billing codes were used to assess whether urine, blood, and CSF testing (as defined previously[9]) were performed during the ED evaluation. To account for ED visits that spanned the midnight hour, for hospitalized patients we considered any testing or treatment occurring on the initial or second hospital day to be performed in the ED; billing code data in PHIS are based upon calendar day and do not distinguish testing performed in the ED versus inpatient setting.[8, 9] Patients billed for observation care were classified as being hospitalized.[20, 21]

We identified the presence of an SBI using ICD‐9 diagnosis codes for the following infections as described previously[9]: urinary tract infection or pyelonephritis,[22] bacteremia or sepsis, bacterial meningitis,[16] pneumonia,[23] or bacterial enteritis. To assess return visits for SBI that required inpatient management, we defined an ED revisit for an SBI as a return visit within 3 days of ED discharge[24, 25] that resulted in hospitalization with an associated ICD‐9 discharge diagnosis code for an SBI.

Hospitals charges in PHIS database were adjusted for hospital location by using the Centers for Medicare and Medicaid Services price/wage index. Costs were estimated by applying hospital‐level cost‐to‐charge ratios to charge data.[26]

Measured Exposures

The primary exposure was the presence of an ED‐based CPG for management of the febrile young infant aged 28 days and 29 to 56 days; 56 days was used as the upper age limit as all of the CPGs included infants up to this age or beyond. Six institutions utilized CPGs with different thresholds to define the age categories (eg, dichotomized at 27 or 30 days); these CPGs were classified into the aforementioned age groups to permit comparisons across standardized age groups. We classified institutions based on the presence of a CPG. To assess differences in the application of low‐risk criteria, the CPGs were further classified a priori based upon specific recommendations around laboratory testing and hospitalization, as well as ceftriaxone use for infants aged 29 to 56 days discharged from the ED. CPGs were categorized based upon whether testing, hospitalization, and ceftriaxone use were: (1) recommended for all patients, (2) recommended only if patients were classified as high risk (absence of low‐risk criteria), (3) recommended against, or (4) recommended to consider at clinician discretion.

Outcome Measures

Measured outcomes were performance of urine, blood, CSF testing, and hospitalization rate, as well as rate of ceftriaxone use for discharged infants aged 29 to 56 days, 3‐day revisits for SBI, and costs per visit, which included hospitalization costs for admitted patients.

Data Analysis

We described continuous variables using median and interquartile range or range values and categorical variables using frequencies. We compared medians using Wilcoxon rank sum and categorical variables using a [2] test. We compared rates of testing, hospitalization, ceftriaxone use, and 3‐day revisits for SBI based on the presence of a CPG, and when present, the specific CPG recommendations. Costs per visit were compared between institutions with and without CPGs and assessed separately for admitted and discharged patients. To adjust for potential confounders and clustering of patients within hospitals, we used generalized estimating equations with logistic regression to generate adjusted odd ratios (aORs) and 95% confidence intervals (CIs). Models were adjusted for geographic region, payer, race, and gender. Statistical analyses were performed by using SAS version 9.3 (SAS Institute, Cary, NC). We determined statistical significance as a 2‐tailed P value <0.05.

Febrile infants with bronchiolitis or a history of prematurity may be managed differently from full‐term febrile young infants without bronchiolitis.[6, 27] Therefore, we performed a subgroup analysis after exclusion of infants with an ICD‐9 discharge diagnosis code for bronchiolitis (466.11 and 466.19)[28] or prematurity (765).

Because our study included ED encounters in 2013, we repeated our analyses after exclusion of hospitals with CPGs implemented during the 2013 calendar year.

CPG by Institution

Thirty‐three (89.2%) of the 37 EDs surveyed completed the questionnaire. Overall, 21 (63.6%) of the 33 EDs had a CPG; 15 (45.5%) had a CPG for all infants 56 days of age, 5 (15.2%) had a CPG for infants 28 days only, and 1 (3.0%) had a CPG for infants 29 to 56 days but not 28 days of age (Figure 1). Seventeen EDs had an established CPG prior to 2013, and 4 hospitals implemented a CPG during the 2013 calendar year, 2 with CPGs for neonates 28 days and 2 with CPGs for both 28 days and 29 to 56 days of age. Hospitals with CPGs were more likely to be located in the Northeast and West regions of the United States and provide care to a higher proportion of non‐Hispanic white patients, as well as those with commercial insurance (Table 1).

Figure 1

All 20 CPGs for the febrile young infant 28 days of age recommended urine, blood, CSF testing, and hospitalization for all infants (Figure 1). Of the 16 hospitals with CPGs for febrile infants aged 29 to 56 days, all recommended urine and blood testing for all patients, except for 1 CPG, which recommended consideration of blood testing but not to obtain routinely. Hospitals varied in recommendations for CSF testing among infants aged 29 to 56 days: 8 (50%) recommended CSF testing in all patients and 8 (50%) recommended CSF testing only if the patient was high risk per defined criteria (based on history, physical examination, urine, and blood testing). In all 16 CPGs, hospitalization was recommended only for high‐risk infants. For low‐risk infants aged 2956 days being discharged from the ED, 3 hospitals recommended ceftriaxone for all, 9 recommended consideration of ceftriaxone, and 4 recommended against antibiotics (Figure 1).

Study Patients

During the study period, there were 10,415 infants 56 days old with a diagnosis of fever at the 33 participating hospitals. After exclusion of 635 (6.1%) infants with a complex chronic condition and 445 (4.3%) transferred from another institution (including 42 with a complex chronic condition), 9377 infants remained in our study cohort. Approximately one‐third of the cohort was 28 days of age and two‐thirds aged 29 to 56 days. The overall SBI rate was 8.5% but varied by age (11.9% in infants 28 days and 6.9% in infants 29 to 56 days of age) (Table 1).

CPGs and Use of Diagnostic Testing, Hospitalization Rates, Ceftriaxone Use, and Revisits for SBI

For infants 28 days of age, the presence of a CPG was not associated with urine, blood, CSF testing, or hospitalization after multivariable adjustment (Table 2). Among infants aged 29 to 56 days, urine testing did not differ based on the presence of a CPG, whereas blood testing was performed less often at the 1 hospital whose CPG recommended to consider, but not routinely obtain, testing (aOR: 0.4, 95% CI: 0.3‐0.7, P=0.001). Compared to hospitals without a CPG, CSF testing was performed less often at hospitals with CPG recommendations to only obtain CSF if high risk (aOR: 0.5, 95% CI: 0.3‐0.8, P=0.002). However, the odds of hospitalization did not differ at institutions with and without a febrile infant CPG (aOR: 0.7, 95% CI: 0.5‐1.1, P=0.10). For infants aged 29 to 56 days discharged from the ED, ceftriaxone was administered more often at hospitals with CPGs that recommended ceftriaxone for all discharged patients (aOR: 4.6, 95% CI: 2.39.3, P<0.001) and less often at hospitals whose CPGs recommended against antibiotics (aOR: 0.3, 95% CI: 0.1‐0.9, P=0.03) (Table 3). Our findings were similar in the subgroup of infants without bronchiolitis or prematurity (see Supporting Tables 1 and 2 in the online version of this article). After exclusion of hospitals with a CPG implemented during the 2013 calendar year (4 hospitals excluded in the 28 days age group and 2 hospitals excluded in the 29 to 56 days age group), infants aged 29 to 56 days cared for at a hospital with a CPG experienced a lower odds of hospitalization (aOR: 0.7, 95% CI: 0.4‐0.98, P=0.04). Otherwise, our findings in both age groups did not materially differ from the main analyses.

Three‐day revisits for SBI were similarly low at hospitals with and without CPGs among infants 28 days (1.5% vs 0.8%, P=0.44) and 29 to 56 days of age (1.4% vs 1.1%, P=0.44) and did not differ after exclusion of hospitals with a CPG implemented in 2013.

CPGs and Costs

Among infants 28 days of age, costs per visit did not differ for admitted and discharged patients based on CPG presence. The presence of an ED febrile infant CPG was associated with higher costs for both admitted and discharged infants 29 to 56 days of age (Table 4). The cost analysis did not significantly differ after exclusion of hospitals with CPGs implemented in 2013.

Study Design

We used the Pediatric Health Information System (PHIS) to identify febrile infants 56 days of age who presented to the ED between January 1, 2013 and December 31, 2013. We also surveyed ED providers at participating PHIS hospitals. Informed consent was obtained from survey respondents. The institutional review board at Boston Children's Hospital approved the study protocol.

Clinical Practice Guideline Survey

We sent an electronic survey to medical directors or division directors at 37 pediatric EDs to determine whether their ED utilized a CPG for the management of the febrile young infant in 2013. If no response was received after the second attempt, we queried ED fellowship directors or other ED attending physicians at nonresponding hospitals. Survey items included the presence of a febrile young infant CPG, and if present, the year of implementation, ages targeted, and CPG content. As applicable, respondents were asked to share their CPG and/or provide the specific CPG recommendations.

We collected and managed survey data using the Research Electronic Data Capture (REDCap) electronic data capture tools hosted at Boston Children's Hospital. REDCap is a secure, Web‐based application designed to support data capture for research studies.[13]

Data Source

The PHIS database contains administrative data from 44 US children's hospitals. These hospitals, affiliated with the Children's Hospital Association, represent 85% of freestanding US children's hospitals.[14] Encrypted patient identifiers permit tracking of patients across encounters.[15] Data quality and integrity are assured jointly by the Children's Hospital Association and participating hospitals.[16] For this study, 7 hospitals were excluded due to incomplete ED data or known data‐quality issues.[17]

Patients

We identified study infants using the following International Classification of Diseases, 9th Revision (ICD‐9) admission or discharge diagnosis codes for fever as defined previously[8, 9]: 780.6, 778.4, 780.60, or 780.61. We excluded infants with a complex chronic condition[18] and those transferred from another institution, as these infants may warrant a nonstandard evaluation and/or may have incomplete data. For infants with >1 ED visit for fever during the study period, repeat visits within 3 days of an index visit were considered a revisit for the same episode of illness; visits >3 days following an index visit were considered as a new index visit.

Study Definitions

From the PHIS database, we abstracted demographic characteristics (gender, race/ethnicity), insurance status, and region where the hospital was located (using US Census categories[19]). Billing codes were used to assess whether urine, blood, and CSF testing (as defined previously[9]) were performed during the ED evaluation. To account for ED visits that spanned the midnight hour, for hospitalized patients we considered any testing or treatment occurring on the initial or second hospital day to be performed in the ED; billing code data in PHIS are based upon calendar day and do not distinguish testing performed in the ED versus inpatient setting.[8, 9] Patients billed for observation care were classified as being hospitalized.[20, 21]

We identified the presence of an SBI using ICD‐9 diagnosis codes for the following infections as described previously[9]: urinary tract infection or pyelonephritis,[22] bacteremia or sepsis, bacterial meningitis,[16] pneumonia,[23] or bacterial enteritis. To assess return visits for SBI that required inpatient management, we defined an ED revisit for an SBI as a return visit within 3 days of ED discharge[24, 25] that resulted in hospitalization with an associated ICD‐9 discharge diagnosis code for an SBI.

Hospitals charges in PHIS database were adjusted for hospital location by using the Centers for Medicare and Medicaid Services price/wage index. Costs were estimated by applying hospital‐level cost‐to‐charge ratios to charge data.[26]

Measured Exposures

The primary exposure was the presence of an ED‐based CPG for management of the febrile young infant aged 28 days and 29 to 56 days; 56 days was used as the upper age limit as all of the CPGs included infants up to this age or beyond. Six institutions utilized CPGs with different thresholds to define the age categories (eg, dichotomized at 27 or 30 days); these CPGs were classified into the aforementioned age groups to permit comparisons across standardized age groups. We classified institutions based on the presence of a CPG. To assess differences in the application of low‐risk criteria, the CPGs were further classified a priori based upon specific recommendations around laboratory testing and hospitalization, as well as ceftriaxone use for infants aged 29 to 56 days discharged from the ED. CPGs were categorized based upon whether testing, hospitalization, and ceftriaxone use were: (1) recommended for all patients, (2) recommended only if patients were classified as high risk (absence of low‐risk criteria), (3) recommended against, or (4) recommended to consider at clinician discretion.

Outcome Measures

Measured outcomes were performance of urine, blood, CSF testing, and hospitalization rate, as well as rate of ceftriaxone use for discharged infants aged 29 to 56 days, 3‐day revisits for SBI, and costs per visit, which included hospitalization costs for admitted patients.

Data Analysis

We described continuous variables using median and interquartile range or range values and categorical variables using frequencies. We compared medians using Wilcoxon rank sum and categorical variables using a [2] test. We compared rates of testing, hospitalization, ceftriaxone use, and 3‐day revisits for SBI based on the presence of a CPG, and when present, the specific CPG recommendations. Costs per visit were compared between institutions with and without CPGs and assessed separately for admitted and discharged patients. To adjust for potential confounders and clustering of patients within hospitals, we used generalized estimating equations with logistic regression to generate adjusted odd ratios (aORs) and 95% confidence intervals (CIs). Models were adjusted for geographic region, payer, race, and gender. Statistical analyses were performed by using SAS version 9.3 (SAS Institute, Cary, NC). We determined statistical significance as a 2‐tailed P value <0.05.

Febrile infants with bronchiolitis or a history of prematurity may be managed differently from full‐term febrile young infants without bronchiolitis.[6, 27] Therefore, we performed a subgroup analysis after exclusion of infants with an ICD‐9 discharge diagnosis code for bronchiolitis (466.11 and 466.19)[28] or prematurity (765).

Because our study included ED encounters in 2013, we repeated our analyses after exclusion of hospitals with CPGs implemented during the 2013 calendar year.

CPG by Institution

Thirty‐three (89.2%) of the 37 EDs surveyed completed the questionnaire. Overall, 21 (63.6%) of the 33 EDs had a CPG; 15 (45.5%) had a CPG for all infants 56 days of age, 5 (15.2%) had a CPG for infants 28 days only, and 1 (3.0%) had a CPG for infants 29 to 56 days but not 28 days of age (Figure 1). Seventeen EDs had an established CPG prior to 2013, and 4 hospitals implemented a CPG during the 2013 calendar year, 2 with CPGs for neonates 28 days and 2 with CPGs for both 28 days and 29 to 56 days of age. Hospitals with CPGs were more likely to be located in the Northeast and West regions of the United States and provide care to a higher proportion of non‐Hispanic white patients, as well as those with commercial insurance (Table 1).

Figure 1

All 20 CPGs for the febrile young infant 28 days of age recommended urine, blood, CSF testing, and hospitalization for all infants (Figure 1). Of the 16 hospitals with CPGs for febrile infants aged 29 to 56 days, all recommended urine and blood testing for all patients, except for 1 CPG, which recommended consideration of blood testing but not to obtain routinely. Hospitals varied in recommendations for CSF testing among infants aged 29 to 56 days: 8 (50%) recommended CSF testing in all patients and 8 (50%) recommended CSF testing only if the patient was high risk per defined criteria (based on history, physical examination, urine, and blood testing). In all 16 CPGs, hospitalization was recommended only for high‐risk infants. For low‐risk infants aged 2956 days being discharged from the ED, 3 hospitals recommended ceftriaxone for all, 9 recommended consideration of ceftriaxone, and 4 recommended against antibiotics (Figure 1).

Study Patients

During the study period, there were 10,415 infants 56 days old with a diagnosis of fever at the 33 participating hospitals. After exclusion of 635 (6.1%) infants with a complex chronic condition and 445 (4.3%) transferred from another institution (including 42 with a complex chronic condition), 9377 infants remained in our study cohort. Approximately one‐third of the cohort was 28 days of age and two‐thirds aged 29 to 56 days. The overall SBI rate was 8.5% but varied by age (11.9% in infants 28 days and 6.9% in infants 29 to 56 days of age) (Table 1).

CPGs and Use of Diagnostic Testing, Hospitalization Rates, Ceftriaxone Use, and Revisits for SBI

For infants 28 days of age, the presence of a CPG was not associated with urine, blood, CSF testing, or hospitalization after multivariable adjustment (Table 2). Among infants aged 29 to 56 days, urine testing did not differ based on the presence of a CPG, whereas blood testing was performed less often at the 1 hospital whose CPG recommended to consider, but not routinely obtain, testing (aOR: 0.4, 95% CI: 0.3‐0.7, P=0.001). Compared to hospitals without a CPG, CSF testing was performed less often at hospitals with CPG recommendations to only obtain CSF if high risk (aOR: 0.5, 95% CI: 0.3‐0.8, P=0.002). However, the odds of hospitalization did not differ at institutions with and without a febrile infant CPG (aOR: 0.7, 95% CI: 0.5‐1.1, P=0.10). For infants aged 29 to 56 days discharged from the ED, ceftriaxone was administered more often at hospitals with CPGs that recommended ceftriaxone for all discharged patients (aOR: 4.6, 95% CI: 2.39.3, P<0.001) and less often at hospitals whose CPGs recommended against antibiotics (aOR: 0.3, 95% CI: 0.1‐0.9, P=0.03) (Table 3). Our findings were similar in the subgroup of infants without bronchiolitis or prematurity (see Supporting Tables 1 and 2 in the online version of this article). After exclusion of hospitals with a CPG implemented during the 2013 calendar year (4 hospitals excluded in the 28 days age group and 2 hospitals excluded in the 29 to 56 days age group), infants aged 29 to 56 days cared for at a hospital with a CPG experienced a lower odds of hospitalization (aOR: 0.7, 95% CI: 0.4‐0.98, P=0.04). Otherwise, our findings in both age groups did not materially differ from the main analyses.

Three‐day revisits for SBI were similarly low at hospitals with and without CPGs among infants 28 days (1.5% vs 0.8%, P=0.44) and 29 to 56 days of age (1.4% vs 1.1%, P=0.44) and did not differ after exclusion of hospitals with a CPG implemented in 2013.

CPGs and Costs

Among infants 28 days of age, costs per visit did not differ for admitted and discharged patients based on CPG presence. The presence of an ED febrile infant CPG was associated with higher costs for both admitted and discharged infants 29 to 56 days of age (Table 4). The cost analysis did not significantly differ after exclusion of hospitals with CPGs implemented in 2013.

Study Design

We used the Pediatric Health Information System (PHIS) to identify febrile infants 56 days of age who presented to the ED between January 1, 2013 and December 31, 2013. We also surveyed ED providers at participating PHIS hospitals. Informed consent was obtained from survey respondents. The institutional review board at Boston Children's Hospital approved the study protocol.

Clinical Practice Guideline Survey

We sent an electronic survey to medical directors or division directors at 37 pediatric EDs to determine whether their ED utilized a CPG for the management of the febrile young infant in 2013. If no response was received after the second attempt, we queried ED fellowship directors or other ED attending physicians at nonresponding hospitals. Survey items included the presence of a febrile young infant CPG, and if present, the year of implementation, ages targeted, and CPG content. As applicable, respondents were asked to share their CPG and/or provide the specific CPG recommendations.

We collected and managed survey data using the Research Electronic Data Capture (REDCap) electronic data capture tools hosted at Boston Children's Hospital. REDCap is a secure, Web‐based application designed to support data capture for research studies.[13]

Data Source

The PHIS database contains administrative data from 44 US children's hospitals. These hospitals, affiliated with the Children's Hospital Association, represent 85% of freestanding US children's hospitals.[14] Encrypted patient identifiers permit tracking of patients across encounters.[15] Data quality and integrity are assured jointly by the Children's Hospital Association and participating hospitals.[16] For this study, 7 hospitals were excluded due to incomplete ED data or known data‐quality issues.[17]

Patients

We identified study infants using the following International Classification of Diseases, 9th Revision (ICD‐9) admission or discharge diagnosis codes for fever as defined previously[8, 9]: 780.6, 778.4, 780.60, or 780.61. We excluded infants with a complex chronic condition[18] and those transferred from another institution, as these infants may warrant a nonstandard evaluation and/or may have incomplete data. For infants with >1 ED visit for fever during the study period, repeat visits within 3 days of an index visit were considered a revisit for the same episode of illness; visits >3 days following an index visit were considered as a new index visit.

Study Definitions

From the PHIS database, we abstracted demographic characteristics (gender, race/ethnicity), insurance status, and region where the hospital was located (using US Census categories[19]). Billing codes were used to assess whether urine, blood, and CSF testing (as defined previously[9]) were performed during the ED evaluation. To account for ED visits that spanned the midnight hour, for hospitalized patients we considered any testing or treatment occurring on the initial or second hospital day to be performed in the ED; billing code data in PHIS are based upon calendar day and do not distinguish testing performed in the ED versus inpatient setting.[8, 9] Patients billed for observation care were classified as being hospitalized.[20, 21]

We identified the presence of an SBI using ICD‐9 diagnosis codes for the following infections as described previously[9]: urinary tract infection or pyelonephritis,[22] bacteremia or sepsis, bacterial meningitis,[16] pneumonia,[23] or bacterial enteritis. To assess return visits for SBI that required inpatient management, we defined an ED revisit for an SBI as a return visit within 3 days of ED discharge[24, 25] that resulted in hospitalization with an associated ICD‐9 discharge diagnosis code for an SBI.

Hospitals charges in PHIS database were adjusted for hospital location by using the Centers for Medicare and Medicaid Services price/wage index. Costs were estimated by applying hospital‐level cost‐to‐charge ratios to charge data.[26]

Measured Exposures

The primary exposure was the presence of an ED‐based CPG for management of the febrile young infant aged 28 days and 29 to 56 days; 56 days was used as the upper age limit as all of the CPGs included infants up to this age or beyond. Six institutions utilized CPGs with different thresholds to define the age categories (eg, dichotomized at 27 or 30 days); these CPGs were classified into the aforementioned age groups to permit comparisons across standardized age groups. We classified institutions based on the presence of a CPG. To assess differences in the application of low‐risk criteria, the CPGs were further classified a priori based upon specific recommendations around laboratory testing and hospitalization, as well as ceftriaxone use for infants aged 29 to 56 days discharged from the ED. CPGs were categorized based upon whether testing, hospitalization, and ceftriaxone use were: (1) recommended for all patients, (2) recommended only if patients were classified as high risk (absence of low‐risk criteria), (3) recommended against, or (4) recommended to consider at clinician discretion.

Outcome Measures

Measured outcomes were performance of urine, blood, CSF testing, and hospitalization rate, as well as rate of ceftriaxone use for discharged infants aged 29 to 56 days, 3‐day revisits for SBI, and costs per visit, which included hospitalization costs for admitted patients.

Data Analysis

We described continuous variables using median and interquartile range or range values and categorical variables using frequencies. We compared medians using Wilcoxon rank sum and categorical variables using a [2] test. We compared rates of testing, hospitalization, ceftriaxone use, and 3‐day revisits for SBI based on the presence of a CPG, and when present, the specific CPG recommendations. Costs per visit were compared between institutions with and without CPGs and assessed separately for admitted and discharged patients. To adjust for potential confounders and clustering of patients within hospitals, we used generalized estimating equations with logistic regression to generate adjusted odd ratios (aORs) and 95% confidence intervals (CIs). Models were adjusted for geographic region, payer, race, and gender. Statistical analyses were performed by using SAS version 9.3 (SAS Institute, Cary, NC). We determined statistical significance as a 2‐tailed P value <0.05.

Febrile infants with bronchiolitis or a history of prematurity may be managed differently from full‐term febrile young infants without bronchiolitis.[6, 27] Therefore, we performed a subgroup analysis after exclusion of infants with an ICD‐9 discharge diagnosis code for bronchiolitis (466.11 and 466.19)[28] or prematurity (765).

Because our study included ED encounters in 2013, we repeated our analyses after exclusion of hospitals with CPGs implemented during the 2013 calendar year.

CPG by Institution

Thirty‐three (89.2%) of the 37 EDs surveyed completed the questionnaire. Overall, 21 (63.6%) of the 33 EDs had a CPG; 15 (45.5%) had a CPG for all infants 56 days of age, 5 (15.2%) had a CPG for infants 28 days only, and 1 (3.0%) had a CPG for infants 29 to 56 days but not 28 days of age (Figure 1). Seventeen EDs had an established CPG prior to 2013, and 4 hospitals implemented a CPG during the 2013 calendar year, 2 with CPGs for neonates 28 days and 2 with CPGs for both 28 days and 29 to 56 days of age. Hospitals with CPGs were more likely to be located in the Northeast and West regions of the United States and provide care to a higher proportion of non‐Hispanic white patients, as well as those with commercial insurance (Table 1).

Figure 1

All 20 CPGs for the febrile young infant 28 days of age recommended urine, blood, CSF testing, and hospitalization for all infants (Figure 1). Of the 16 hospitals with CPGs for febrile infants aged 29 to 56 days, all recommended urine and blood testing for all patients, except for 1 CPG, which recommended consideration of blood testing but not to obtain routinely. Hospitals varied in recommendations for CSF testing among infants aged 29 to 56 days: 8 (50%) recommended CSF testing in all patients and 8 (50%) recommended CSF testing only if the patient was high risk per defined criteria (based on history, physical examination, urine, and blood testing). In all 16 CPGs, hospitalization was recommended only for high‐risk infants. For low‐risk infants aged 2956 days being discharged from the ED, 3 hospitals recommended ceftriaxone for all, 9 recommended consideration of ceftriaxone, and 4 recommended against antibiotics (Figure 1).

Study Patients

During the study period, there were 10,415 infants 56 days old with a diagnosis of fever at the 33 participating hospitals. After exclusion of 635 (6.1%) infants with a complex chronic condition and 445 (4.3%) transferred from another institution (including 42 with a complex chronic condition), 9377 infants remained in our study cohort. Approximately one‐third of the cohort was 28 days of age and two‐thirds aged 29 to 56 days. The overall SBI rate was 8.5% but varied by age (11.9% in infants 28 days and 6.9% in infants 29 to 56 days of age) (Table 1).

CPGs and Use of Diagnostic Testing, Hospitalization Rates, Ceftriaxone Use, and Revisits for SBI

For infants 28 days of age, the presence of a CPG was not associated with urine, blood, CSF testing, or hospitalization after multivariable adjustment (Table 2). Among infants aged 29 to 56 days, urine testing did not differ based on the presence of a CPG, whereas blood testing was performed less often at the 1 hospital whose CPG recommended to consider, but not routinely obtain, testing (aOR: 0.4, 95% CI: 0.3‐0.7, P=0.001). Compared to hospitals without a CPG, CSF testing was performed less often at hospitals with CPG recommendations to only obtain CSF if high risk (aOR: 0.5, 95% CI: 0.3‐0.8, P=0.002). However, the odds of hospitalization did not differ at institutions with and without a febrile infant CPG (aOR: 0.7, 95% CI: 0.5‐1.1, P=0.10). For infants aged 29 to 56 days discharged from the ED, ceftriaxone was administered more often at hospitals with CPGs that recommended ceftriaxone for all discharged patients (aOR: 4.6, 95% CI: 2.39.3, P<0.001) and less often at hospitals whose CPGs recommended against antibiotics (aOR: 0.3, 95% CI: 0.1‐0.9, P=0.03) (Table 3). Our findings were similar in the subgroup of infants without bronchiolitis or prematurity (see Supporting Tables 1 and 2 in the online version of this article). After exclusion of hospitals with a CPG implemented during the 2013 calendar year (4 hospitals excluded in the 28 days age group and 2 hospitals excluded in the 29 to 56 days age group), infants aged 29 to 56 days cared for at a hospital with a CPG experienced a lower odds of hospitalization (aOR: 0.7, 95% CI: 0.4‐0.98, P=0.04). Otherwise, our findings in both age groups did not materially differ from the main analyses.