Data Sources

Data Analysis

To examine the internal validity of the Hospital Compare measures, we calculated pairwise correlation coefficients among the 14 core‐measure components, using all eligible data points. We then calculated Cronbach's , a measure of the internal consistency of scales of measures, to characterize each of the sets of Hospital Compare core measures separately (AMI, CHF, CAP). We also generated Cronbach's for a measure we called the combined core‐measures score, which we intended to be analogous to the Best Hospitals Honor Roll, defined as the AMI, CHF, and CAP measure sets scored together.

To compare Hospital Compare data with the Best Hospitals rankings (for heart and heart surgery, respiratory disorders, and the Honor Roll), we first established national quartile score cut points for each of the 3 Hospital Compare core measure sets and for the combined core measures, using all U.S. hospitals eligible for our analysis. We used quartiles to avoid the misclassification that would be more likely to occur with deciles (based on confidence intervals for the core measures provided by CMS).6

We calculated Hospital Compare scores for each institution listed as a Best Hospital in 2004 and 2005 and classified the Best Hospitals into scoring quartiles based on national score cut points (eg, if the national cutoff for AMI core measures for the top quartile was 95.2%, then a Best Hospital with an AMI score for the core‐measures set 95.2% was classified in the first [top] quartile). AMI and CHF core measure sets were used for comparison with the Best Hospitals for heart and heart surgery, the CAP core‐measure set was used for comparison with the Best Hospitals for respiratory disorders, and the combined core‐measure set was used for comparison with the Honor Roll hospitals.

Sensitivity Analyses

To investigate the effect of missing Hospital Compare data on our study findings, we conducted sensitivity analyses. We used only those institutions with complete data for the AMI, CHF, and CAP core measure sets to establish new quartile cut points and then reexamined the quartile distribution for institutions in the corresponding Best Hospitals lists. We also compared the Best Hospitals' Hospital Compare data completeness with that of all Hospital Compare institutions.

Core Performance Measures in Hospital Compare

Of 4203 hospitals that submitted core measures as part of Hospital Compare, 4126 had at least 1 core measure eligible for analysis (> 25 observations). Of these 4126 hospitals, 2165 (52.5%) had at least 1 eligible AMI core measure, and 398 (9.7%) had all 6 measures eligible for analysis; 3130 had at least 1 eligible CHF core measure (75.9%), and 289 (7.0%) had all 4 measures eligible for analysis; and 3462 (83.9%) had at least one eligible CAP core measure and 302 (7.3%) had all 4 measures eligible for analysis. For the combined core‐measure score, 2119 (51.4%) had at least 4 eligible measures, and 120 (2.9%) had all 14 measures eligible for analysis.

Pairwise correlation coefficients within each of the disease‐specific core measure sets was highest for the AMI measures, and was generally higher for measures that reflected similar clinical activities (eg, aspirin and ‐blocker at discharge for AMI care; tobacco cessation counseling for AMI, CHF, and CAP; Table 2). In general, the AMI and CHF performance measures correlated more strongly with each other than did the AMI or CHF measures with the CAP measures.

Internal consistency within each of the disease‐specific measures was moderate to strong, with Cronbach's = .83 for AMI, Cronbach's = .58 for CHF, and Cronbach's = .49 for CAP. For the combined performance measure set (all 14 core measures together), Cronbach's = .74.

Hospital Compare Scores for Institutions Listed as Best Hospitals

Best Hospitals for heart and heart surgery and for respiratory disorders in U.S. News and World Report in 2004 and 2005 exhibited a broad distribution of Hospital Compare core measure scores (Table 3). For none of the core measure sets did a majority of Best Hospitals score in the top quartile in either year.

Among the 50 hospitals identified as best for cardiac care, only 20 (40%) in the 2004 list and 15 (30%) in the 2005 list had AMI core‐measure scores in the top quartile nationally, and 14 (28%) scored below the national median in both years. Among those same 50 hospitals, only 19 (38%) had CHF core‐measure scores in the top quartile nationally in both years, whereas 17 (34%) scored below the national median in 2004 and 16 in 2005. On the CAP core measures, Best Hospitals for respiratory disorders generally scored poorly, with only 5 (10%) from the 2004 list and 7 (14%) from the 2005 list in the top quartile nationally and nearly half the institutions scoring in the bottom national quartile (Table 3).

For the 14 hospitals named to the 2004 Honor Roll of Best Hospitals, the comparison with the combined core‐measure score (AMI, CHF, and CAP together) revealed a similarly broad distribution of core measure performance. Only five hospitals scored in the top quartile, 2 in the second quartile, 5 in the third quartile, and 2 in the bottom quartile. The distribution for hospitals in the 2005 Honor Roll was similar (5‐3‐6‐2 by quartile).

Sensitivity Analyses

National quartile Hospital Compare core‐measure cut points were slightly lower (1%‐2% in absolute terms) for those institutions with complete data than for institutions overall; in other words, institutions reporting on all 17 measures were generally more likely to have somewhat lower scores. These differences were substantive enough to shift the distribution of Best Hospitals in 2004 and 2005 up to higher quartiles for the AMI and CHF Hospital Compare measures but not for the CAP measures. For example, using the complete data AMI cut points, 23 of the 50 Best Hospitals for cardiac care in 2005 scored in the top quartile, 16 in the second quartile, 6 in the third quartile, and 5 in the bottom quartile (compared with 15‐21‐10‐4; Table 3). With complete data CHF cut points, the distribution was 26, 11, 9, and 4 for the 2005 Best Hospitals for cardiac care from the top through bottom quartiles, respectively (compared with 19‐15‐12‐4; Table 3). Results for 2004 sensitivity analyses were similar.

Institutions named as Best Hospitals appeared more likely than institutions overall to have complete Hospital Compare data. Whereas fewer than 10% of institutions in Hospital Compare had complete data for the AMI, CHF, and CAP core measures, 60% of Best Hospitals for cardiac care in 2005 had complete data for AMI measures and 44% for CHF measures, whereas 32% of Best Hospitals for respiratory care had complete CAP data.

Lack of Agreement in Hospital Quality Measurement

Discordance between the Hospital Compare and Best Hospitals rating systems is not all that surprising, given that their methods of institutional assessment differ markedly. Although both approaches share the goal of allowing consumers a comparative look at institutional performance nationally, they clearly measure different aspects of hospital care.

Hospital Compare measures focus on the delivery of disease‐specific, evidence‐based practices for 3 acute medical conditions from the emergency department to discharge. In comparison, the Best Hospitals rankings emphasize the reputation and mortality data of hospitals and health systems across a variety of general and subspecialty care settings (including several in which core quality measures have not yet been developed), combined with factors related to nursing and technology availability that may also influence consumers' choices. Of note, the Best Hospitals rating approach has been criticized in the past for its strong reliance on physicians' ratings of institutional reputation, which may have little to do with functional measures of quality.7

In essence, the Hospital Compare measures indicate how hospitals perform for an average case, while Best Hospitals relies on reputation and focus on mortality to indicate how institutions perform on the toughest cases. The question at hand is: are these institutional quality measures complementary or contradictory? Our findings suggest that Hospital Compare and Best Hospitals measures offer consumers a mix of complementary and contradictory information, depending on the institution.

The ratings systems differ in other respects as well. In Hospital Compare, performance data are available for more than 4000 hospitals, which permits consumers to examine their local institutions, whereas the Best Hospitals lists offer information only on the top performers. On the other hand, the more established Best Hospitals listings have been published annually for the last 15 years,5 permitting some longitudinal evaluation of hospitals' quality consistency. Importantly, neither rating system includes measures of patient satisfaction with hospital care.

One dimension that both rating systems share is the migration of quality measurement from the local and institutional level to the national stage. Historically, health care quality measurement has been a local phenomenon, as institutions work to gain larger shares of their local markets. A few hospitals have marketed their care and services regionally or even nationally and internationally, but these institutionswhich previously primarily used their reputation rather than specific outcome metrics to reach beyond their local communitiesare a minority of U.S. hospitals.

Although Hospital Compare and Best Hospitals are both national in scope, only Hospital Compare allows consumers to understand the quality of care in most of their community hospitals and health systems. Other investigators analyzing the same data set have highlighted significant differences in hospital performance according to for‐profit status, academic status, and size (number of beds).8

However, it is not yet clear if and how hospital ratings influence consumers' health care decisions. In fact, some studies suggest that only a minority of patients are inclined to use performance reports in their decisions about health care.9, 10 Moreover, if illness is acute, the factors driving choice of hospital may be geographic proximity, bed availability, and payer contracts rather than performance measures.

These constraints on the utility of hospital quality metrics from the consumer perspective are reminders that such metrics may have other benefits. Specifically, ratings such as Hospital Compare and Best Hospitals, as well as others such as those of the Leapfrog Group11 and the Joint Commission on Accreditation of Healthcare Organizations,12 offer differing arrays of performance measures that may induce hospitals to improve their quality of care.1, 13 Institutions that score well or improve their scores over time can use such scores not only to benchmark their processes and outcomes but also to signal the comparative value of their care to the public. In the past, hospitals named to the Best Hospitals Honor Roll have trumpeted their achievements through plaques on their walls and in advertisements for their services. Whether institutions will do the same regarding their Hospital Compare scores remains to be seen.

Study Limitations

The chief limitation of this analysis is that not all hospitals reported data for the Hospital Compare core measures. We standardized the core‐measure sets for AMI, CHF, and CAP care for the number of measures reported in each set in order to include as many hospitals as possible in our analyses. Participation in Hospital Compare is voluntary (although strongly encouraged because of better Medicare reimbursement for institutions that participate), so it is possible that there was a systematic scoring bias in hospitals' incomplete reporting across all measures, that is, hospitals might not report specific core measure scores if they were particularly poor.13 That scale score medians were slightly lower for hospitals with complete data than for hospitals overall may indicate some reporting bias in the Hospital Compare data. Nevertheless, in the sensitivity analyses we performed using only those hospitals with complete data on the Hospital Compare core measures, comparisons with the Best Hospitals lists still predominantly indicated discordance between the rating systems.

Another limitation of this work is that we examined only 2 of several currently available hospital‐rating schemes. We chose to examine Hospital Compare because it is the first governmental effort to report specific hospital quality measures to the public, and we elected to look at Hospital Compare alongside the Best Hospitals lists because the latter are arguably the hospital ratings best known to the lay public.

A third potential limitation is that the Best Hospitals lists for 2004 were based in part on mortality figures and hospital survey data from 2002, which were the most recent data available at the time of the rankings; for the 2005 Best Hospitals lists, the most recent mortality and hospital survey data were collected in 2003.4 Hospital Compare scores were calculated on the basis of patients discharged in 2004, and therefore the ratings systems reflect somewhat different time frames. Nonetheless, we do not believe that this mismatch explains the extent of discordance between the 2 rating scales, particularly because there was such stability in the Best Hospital lists over the 2 years.

Data Sources

Data Analysis

To examine the internal validity of the Hospital Compare measures, we calculated pairwise correlation coefficients among the 14 core‐measure components, using all eligible data points. We then calculated Cronbach's , a measure of the internal consistency of scales of measures, to characterize each of the sets of Hospital Compare core measures separately (AMI, CHF, CAP). We also generated Cronbach's for a measure we called the combined core‐measures score, which we intended to be analogous to the Best Hospitals Honor Roll, defined as the AMI, CHF, and CAP measure sets scored together.

To compare Hospital Compare data with the Best Hospitals rankings (for heart and heart surgery, respiratory disorders, and the Honor Roll), we first established national quartile score cut points for each of the 3 Hospital Compare core measure sets and for the combined core measures, using all U.S. hospitals eligible for our analysis. We used quartiles to avoid the misclassification that would be more likely to occur with deciles (based on confidence intervals for the core measures provided by CMS).6

We calculated Hospital Compare scores for each institution listed as a Best Hospital in 2004 and 2005 and classified the Best Hospitals into scoring quartiles based on national score cut points (eg, if the national cutoff for AMI core measures for the top quartile was 95.2%, then a Best Hospital with an AMI score for the core‐measures set 95.2% was classified in the first [top] quartile). AMI and CHF core measure sets were used for comparison with the Best Hospitals for heart and heart surgery, the CAP core‐measure set was used for comparison with the Best Hospitals for respiratory disorders, and the combined core‐measure set was used for comparison with the Honor Roll hospitals.

Sensitivity Analyses

To investigate the effect of missing Hospital Compare data on our study findings, we conducted sensitivity analyses. We used only those institutions with complete data for the AMI, CHF, and CAP core measure sets to establish new quartile cut points and then reexamined the quartile distribution for institutions in the corresponding Best Hospitals lists. We also compared the Best Hospitals' Hospital Compare data completeness with that of all Hospital Compare institutions.

Core Performance Measures in Hospital Compare

Of 4203 hospitals that submitted core measures as part of Hospital Compare, 4126 had at least 1 core measure eligible for analysis (> 25 observations). Of these 4126 hospitals, 2165 (52.5%) had at least 1 eligible AMI core measure, and 398 (9.7%) had all 6 measures eligible for analysis; 3130 had at least 1 eligible CHF core measure (75.9%), and 289 (7.0%) had all 4 measures eligible for analysis; and 3462 (83.9%) had at least one eligible CAP core measure and 302 (7.3%) had all 4 measures eligible for analysis. For the combined core‐measure score, 2119 (51.4%) had at least 4 eligible measures, and 120 (2.9%) had all 14 measures eligible for analysis.

Pairwise correlation coefficients within each of the disease‐specific core measure sets was highest for the AMI measures, and was generally higher for measures that reflected similar clinical activities (eg, aspirin and ‐blocker at discharge for AMI care; tobacco cessation counseling for AMI, CHF, and CAP; Table 2). In general, the AMI and CHF performance measures correlated more strongly with each other than did the AMI or CHF measures with the CAP measures.

Internal consistency within each of the disease‐specific measures was moderate to strong, with Cronbach's = .83 for AMI, Cronbach's = .58 for CHF, and Cronbach's = .49 for CAP. For the combined performance measure set (all 14 core measures together), Cronbach's = .74.

Hospital Compare Scores for Institutions Listed as Best Hospitals

Best Hospitals for heart and heart surgery and for respiratory disorders in U.S. News and World Report in 2004 and 2005 exhibited a broad distribution of Hospital Compare core measure scores (Table 3). For none of the core measure sets did a majority of Best Hospitals score in the top quartile in either year.

Among the 50 hospitals identified as best for cardiac care, only 20 (40%) in the 2004 list and 15 (30%) in the 2005 list had AMI core‐measure scores in the top quartile nationally, and 14 (28%) scored below the national median in both years. Among those same 50 hospitals, only 19 (38%) had CHF core‐measure scores in the top quartile nationally in both years, whereas 17 (34%) scored below the national median in 2004 and 16 in 2005. On the CAP core measures, Best Hospitals for respiratory disorders generally scored poorly, with only 5 (10%) from the 2004 list and 7 (14%) from the 2005 list in the top quartile nationally and nearly half the institutions scoring in the bottom national quartile (Table 3).

For the 14 hospitals named to the 2004 Honor Roll of Best Hospitals, the comparison with the combined core‐measure score (AMI, CHF, and CAP together) revealed a similarly broad distribution of core measure performance. Only five hospitals scored in the top quartile, 2 in the second quartile, 5 in the third quartile, and 2 in the bottom quartile. The distribution for hospitals in the 2005 Honor Roll was similar (5‐3‐6‐2 by quartile).

Sensitivity Analyses

National quartile Hospital Compare core‐measure cut points were slightly lower (1%‐2% in absolute terms) for those institutions with complete data than for institutions overall; in other words, institutions reporting on all 17 measures were generally more likely to have somewhat lower scores. These differences were substantive enough to shift the distribution of Best Hospitals in 2004 and 2005 up to higher quartiles for the AMI and CHF Hospital Compare measures but not for the CAP measures. For example, using the complete data AMI cut points, 23 of the 50 Best Hospitals for cardiac care in 2005 scored in the top quartile, 16 in the second quartile, 6 in the third quartile, and 5 in the bottom quartile (compared with 15‐21‐10‐4; Table 3). With complete data CHF cut points, the distribution was 26, 11, 9, and 4 for the 2005 Best Hospitals for cardiac care from the top through bottom quartiles, respectively (compared with 19‐15‐12‐4; Table 3). Results for 2004 sensitivity analyses were similar.

Institutions named as Best Hospitals appeared more likely than institutions overall to have complete Hospital Compare data. Whereas fewer than 10% of institutions in Hospital Compare had complete data for the AMI, CHF, and CAP core measures, 60% of Best Hospitals for cardiac care in 2005 had complete data for AMI measures and 44% for CHF measures, whereas 32% of Best Hospitals for respiratory care had complete CAP data.

Lack of Agreement in Hospital Quality Measurement

Discordance between the Hospital Compare and Best Hospitals rating systems is not all that surprising, given that their methods of institutional assessment differ markedly. Although both approaches share the goal of allowing consumers a comparative look at institutional performance nationally, they clearly measure different aspects of hospital care.

Hospital Compare measures focus on the delivery of disease‐specific, evidence‐based practices for 3 acute medical conditions from the emergency department to discharge. In comparison, the Best Hospitals rankings emphasize the reputation and mortality data of hospitals and health systems across a variety of general and subspecialty care settings (including several in which core quality measures have not yet been developed), combined with factors related to nursing and technology availability that may also influence consumers' choices. Of note, the Best Hospitals rating approach has been criticized in the past for its strong reliance on physicians' ratings of institutional reputation, which may have little to do with functional measures of quality.7

In essence, the Hospital Compare measures indicate how hospitals perform for an average case, while Best Hospitals relies on reputation and focus on mortality to indicate how institutions perform on the toughest cases. The question at hand is: are these institutional quality measures complementary or contradictory? Our findings suggest that Hospital Compare and Best Hospitals measures offer consumers a mix of complementary and contradictory information, depending on the institution.

The ratings systems differ in other respects as well. In Hospital Compare, performance data are available for more than 4000 hospitals, which permits consumers to examine their local institutions, whereas the Best Hospitals lists offer information only on the top performers. On the other hand, the more established Best Hospitals listings have been published annually for the last 15 years,5 permitting some longitudinal evaluation of hospitals' quality consistency. Importantly, neither rating system includes measures of patient satisfaction with hospital care.

One dimension that both rating systems share is the migration of quality measurement from the local and institutional level to the national stage. Historically, health care quality measurement has been a local phenomenon, as institutions work to gain larger shares of their local markets. A few hospitals have marketed their care and services regionally or even nationally and internationally, but these institutionswhich previously primarily used their reputation rather than specific outcome metrics to reach beyond their local communitiesare a minority of U.S. hospitals.

Although Hospital Compare and Best Hospitals are both national in scope, only Hospital Compare allows consumers to understand the quality of care in most of their community hospitals and health systems. Other investigators analyzing the same data set have highlighted significant differences in hospital performance according to for‐profit status, academic status, and size (number of beds).8

However, it is not yet clear if and how hospital ratings influence consumers' health care decisions. In fact, some studies suggest that only a minority of patients are inclined to use performance reports in their decisions about health care.9, 10 Moreover, if illness is acute, the factors driving choice of hospital may be geographic proximity, bed availability, and payer contracts rather than performance measures.

These constraints on the utility of hospital quality metrics from the consumer perspective are reminders that such metrics may have other benefits. Specifically, ratings such as Hospital Compare and Best Hospitals, as well as others such as those of the Leapfrog Group11 and the Joint Commission on Accreditation of Healthcare Organizations,12 offer differing arrays of performance measures that may induce hospitals to improve their quality of care.1, 13 Institutions that score well or improve their scores over time can use such scores not only to benchmark their processes and outcomes but also to signal the comparative value of their care to the public. In the past, hospitals named to the Best Hospitals Honor Roll have trumpeted their achievements through plaques on their walls and in advertisements for their services. Whether institutions will do the same regarding their Hospital Compare scores remains to be seen.

Study Limitations

The chief limitation of this analysis is that not all hospitals reported data for the Hospital Compare core measures. We standardized the core‐measure sets for AMI, CHF, and CAP care for the number of measures reported in each set in order to include as many hospitals as possible in our analyses. Participation in Hospital Compare is voluntary (although strongly encouraged because of better Medicare reimbursement for institutions that participate), so it is possible that there was a systematic scoring bias in hospitals' incomplete reporting across all measures, that is, hospitals might not report specific core measure scores if they were particularly poor.13 That scale score medians were slightly lower for hospitals with complete data than for hospitals overall may indicate some reporting bias in the Hospital Compare data. Nevertheless, in the sensitivity analyses we performed using only those hospitals with complete data on the Hospital Compare core measures, comparisons with the Best Hospitals lists still predominantly indicated discordance between the rating systems.

Another limitation of this work is that we examined only 2 of several currently available hospital‐rating schemes. We chose to examine Hospital Compare because it is the first governmental effort to report specific hospital quality measures to the public, and we elected to look at Hospital Compare alongside the Best Hospitals lists because the latter are arguably the hospital ratings best known to the lay public.

A third potential limitation is that the Best Hospitals lists for 2004 were based in part on mortality figures and hospital survey data from 2002, which were the most recent data available at the time of the rankings; for the 2005 Best Hospitals lists, the most recent mortality and hospital survey data were collected in 2003.4 Hospital Compare scores were calculated on the basis of patients discharged in 2004, and therefore the ratings systems reflect somewhat different time frames. Nonetheless, we do not believe that this mismatch explains the extent of discordance between the 2 rating scales, particularly because there was such stability in the Best Hospital lists over the 2 years.

Data Sources

Data Analysis

To examine the internal validity of the Hospital Compare measures, we calculated pairwise correlation coefficients among the 14 core‐measure components, using all eligible data points. We then calculated Cronbach's , a measure of the internal consistency of scales of measures, to characterize each of the sets of Hospital Compare core measures separately (AMI, CHF, CAP). We also generated Cronbach's for a measure we called the combined core‐measures score, which we intended to be analogous to the Best Hospitals Honor Roll, defined as the AMI, CHF, and CAP measure sets scored together.

To compare Hospital Compare data with the Best Hospitals rankings (for heart and heart surgery, respiratory disorders, and the Honor Roll), we first established national quartile score cut points for each of the 3 Hospital Compare core measure sets and for the combined core measures, using all U.S. hospitals eligible for our analysis. We used quartiles to avoid the misclassification that would be more likely to occur with deciles (based on confidence intervals for the core measures provided by CMS).6

We calculated Hospital Compare scores for each institution listed as a Best Hospital in 2004 and 2005 and classified the Best Hospitals into scoring quartiles based on national score cut points (eg, if the national cutoff for AMI core measures for the top quartile was 95.2%, then a Best Hospital with an AMI score for the core‐measures set 95.2% was classified in the first [top] quartile). AMI and CHF core measure sets were used for comparison with the Best Hospitals for heart and heart surgery, the CAP core‐measure set was used for comparison with the Best Hospitals for respiratory disorders, and the combined core‐measure set was used for comparison with the Honor Roll hospitals.

Sensitivity Analyses

To investigate the effect of missing Hospital Compare data on our study findings, we conducted sensitivity analyses. We used only those institutions with complete data for the AMI, CHF, and CAP core measure sets to establish new quartile cut points and then reexamined the quartile distribution for institutions in the corresponding Best Hospitals lists. We also compared the Best Hospitals' Hospital Compare data completeness with that of all Hospital Compare institutions.

Core Performance Measures in Hospital Compare

Of 4203 hospitals that submitted core measures as part of Hospital Compare, 4126 had at least 1 core measure eligible for analysis (> 25 observations). Of these 4126 hospitals, 2165 (52.5%) had at least 1 eligible AMI core measure, and 398 (9.7%) had all 6 measures eligible for analysis; 3130 had at least 1 eligible CHF core measure (75.9%), and 289 (7.0%) had all 4 measures eligible for analysis; and 3462 (83.9%) had at least one eligible CAP core measure and 302 (7.3%) had all 4 measures eligible for analysis. For the combined core‐measure score, 2119 (51.4%) had at least 4 eligible measures, and 120 (2.9%) had all 14 measures eligible for analysis.

Pairwise correlation coefficients within each of the disease‐specific core measure sets was highest for the AMI measures, and was generally higher for measures that reflected similar clinical activities (eg, aspirin and ‐blocker at discharge for AMI care; tobacco cessation counseling for AMI, CHF, and CAP; Table 2). In general, the AMI and CHF performance measures correlated more strongly with each other than did the AMI or CHF measures with the CAP measures.

Internal consistency within each of the disease‐specific measures was moderate to strong, with Cronbach's = .83 for AMI, Cronbach's = .58 for CHF, and Cronbach's = .49 for CAP. For the combined performance measure set (all 14 core measures together), Cronbach's = .74.

Hospital Compare Scores for Institutions Listed as Best Hospitals

Best Hospitals for heart and heart surgery and for respiratory disorders in U.S. News and World Report in 2004 and 2005 exhibited a broad distribution of Hospital Compare core measure scores (Table 3). For none of the core measure sets did a majority of Best Hospitals score in the top quartile in either year.

Among the 50 hospitals identified as best for cardiac care, only 20 (40%) in the 2004 list and 15 (30%) in the 2005 list had AMI core‐measure scores in the top quartile nationally, and 14 (28%) scored below the national median in both years. Among those same 50 hospitals, only 19 (38%) had CHF core‐measure scores in the top quartile nationally in both years, whereas 17 (34%) scored below the national median in 2004 and 16 in 2005. On the CAP core measures, Best Hospitals for respiratory disorders generally scored poorly, with only 5 (10%) from the 2004 list and 7 (14%) from the 2005 list in the top quartile nationally and nearly half the institutions scoring in the bottom national quartile (Table 3).

For the 14 hospitals named to the 2004 Honor Roll of Best Hospitals, the comparison with the combined core‐measure score (AMI, CHF, and CAP together) revealed a similarly broad distribution of core measure performance. Only five hospitals scored in the top quartile, 2 in the second quartile, 5 in the third quartile, and 2 in the bottom quartile. The distribution for hospitals in the 2005 Honor Roll was similar (5‐3‐6‐2 by quartile).

Sensitivity Analyses

National quartile Hospital Compare core‐measure cut points were slightly lower (1%‐2% in absolute terms) for those institutions with complete data than for institutions overall; in other words, institutions reporting on all 17 measures were generally more likely to have somewhat lower scores. These differences were substantive enough to shift the distribution of Best Hospitals in 2004 and 2005 up to higher quartiles for the AMI and CHF Hospital Compare measures but not for the CAP measures. For example, using the complete data AMI cut points, 23 of the 50 Best Hospitals for cardiac care in 2005 scored in the top quartile, 16 in the second quartile, 6 in the third quartile, and 5 in the bottom quartile (compared with 15‐21‐10‐4; Table 3). With complete data CHF cut points, the distribution was 26, 11, 9, and 4 for the 2005 Best Hospitals for cardiac care from the top through bottom quartiles, respectively (compared with 19‐15‐12‐4; Table 3). Results for 2004 sensitivity analyses were similar.

Institutions named as Best Hospitals appeared more likely than institutions overall to have complete Hospital Compare data. Whereas fewer than 10% of institutions in Hospital Compare had complete data for the AMI, CHF, and CAP core measures, 60% of Best Hospitals for cardiac care in 2005 had complete data for AMI measures and 44% for CHF measures, whereas 32% of Best Hospitals for respiratory care had complete CAP data.

Lack of Agreement in Hospital Quality Measurement

Discordance between the Hospital Compare and Best Hospitals rating systems is not all that surprising, given that their methods of institutional assessment differ markedly. Although both approaches share the goal of allowing consumers a comparative look at institutional performance nationally, they clearly measure different aspects of hospital care.

Hospital Compare measures focus on the delivery of disease‐specific, evidence‐based practices for 3 acute medical conditions from the emergency department to discharge. In comparison, the Best Hospitals rankings emphasize the reputation and mortality data of hospitals and health systems across a variety of general and subspecialty care settings (including several in which core quality measures have not yet been developed), combined with factors related to nursing and technology availability that may also influence consumers' choices. Of note, the Best Hospitals rating approach has been criticized in the past for its strong reliance on physicians' ratings of institutional reputation, which may have little to do with functional measures of quality.7

In essence, the Hospital Compare measures indicate how hospitals perform for an average case, while Best Hospitals relies on reputation and focus on mortality to indicate how institutions perform on the toughest cases. The question at hand is: are these institutional quality measures complementary or contradictory? Our findings suggest that Hospital Compare and Best Hospitals measures offer consumers a mix of complementary and contradictory information, depending on the institution.

The ratings systems differ in other respects as well. In Hospital Compare, performance data are available for more than 4000 hospitals, which permits consumers to examine their local institutions, whereas the Best Hospitals lists offer information only on the top performers. On the other hand, the more established Best Hospitals listings have been published annually for the last 15 years,5 permitting some longitudinal evaluation of hospitals' quality consistency. Importantly, neither rating system includes measures of patient satisfaction with hospital care.

One dimension that both rating systems share is the migration of quality measurement from the local and institutional level to the national stage. Historically, health care quality measurement has been a local phenomenon, as institutions work to gain larger shares of their local markets. A few hospitals have marketed their care and services regionally or even nationally and internationally, but these institutionswhich previously primarily used their reputation rather than specific outcome metrics to reach beyond their local communitiesare a minority of U.S. hospitals.

Although Hospital Compare and Best Hospitals are both national in scope, only Hospital Compare allows consumers to understand the quality of care in most of their community hospitals and health systems. Other investigators analyzing the same data set have highlighted significant differences in hospital performance according to for‐profit status, academic status, and size (number of beds).8

However, it is not yet clear if and how hospital ratings influence consumers' health care decisions. In fact, some studies suggest that only a minority of patients are inclined to use performance reports in their decisions about health care.9, 10 Moreover, if illness is acute, the factors driving choice of hospital may be geographic proximity, bed availability, and payer contracts rather than performance measures.

These constraints on the utility of hospital quality metrics from the consumer perspective are reminders that such metrics may have other benefits. Specifically, ratings such as Hospital Compare and Best Hospitals, as well as others such as those of the Leapfrog Group11 and the Joint Commission on Accreditation of Healthcare Organizations,12 offer differing arrays of performance measures that may induce hospitals to improve their quality of care.1, 13 Institutions that score well or improve their scores over time can use such scores not only to benchmark their processes and outcomes but also to signal the comparative value of their care to the public. In the past, hospitals named to the Best Hospitals Honor Roll have trumpeted their achievements through plaques on their walls and in advertisements for their services. Whether institutions will do the same regarding their Hospital Compare scores remains to be seen.

Study Limitations

The chief limitation of this analysis is that not all hospitals reported data for the Hospital Compare core measures. We standardized the core‐measure sets for AMI, CHF, and CAP care for the number of measures reported in each set in order to include as many hospitals as possible in our analyses. Participation in Hospital Compare is voluntary (although strongly encouraged because of better Medicare reimbursement for institutions that participate), so it is possible that there was a systematic scoring bias in hospitals' incomplete reporting across all measures, that is, hospitals might not report specific core measure scores if they were particularly poor.13 That scale score medians were slightly lower for hospitals with complete data than for hospitals overall may indicate some reporting bias in the Hospital Compare data. Nevertheless, in the sensitivity analyses we performed using only those hospitals with complete data on the Hospital Compare core measures, comparisons with the Best Hospitals lists still predominantly indicated discordance between the rating systems.

Another limitation of this work is that we examined only 2 of several currently available hospital‐rating schemes. We chose to examine Hospital Compare because it is the first governmental effort to report specific hospital quality measures to the public, and we elected to look at Hospital Compare alongside the Best Hospitals lists because the latter are arguably the hospital ratings best known to the lay public.

A third potential limitation is that the Best Hospitals lists for 2004 were based in part on mortality figures and hospital survey data from 2002, which were the most recent data available at the time of the rankings; for the 2005 Best Hospitals lists, the most recent mortality and hospital survey data were collected in 2003.4 Hospital Compare scores were calculated on the basis of patients discharged in 2004, and therefore the ratings systems reflect somewhat different time frames. Nonetheless, we do not believe that this mismatch explains the extent of discordance between the 2 rating scales, particularly because there was such stability in the Best Hospital lists over the 2 years.

Design and Setting

In this prospective cohort study, the cohort was all patients admitted to the general medicine service at Cook County Hospital, a 500‐bed public teaching hospital in Chicago, Illinois, in January and February of 2006. The general medicine inpatient service is divided into 3 firms (A, B, and C), each with 4 separate teams of physicians and students. Admissions from the emergency department or other services in the hospital, such as intensive care units (which are closed and therefore staffed by separate teams of physicians), are distributed in sequence to on‐call teams from each firm. During the study period, the availability of a bedside procedure service varied by firm. Throughout the first 4 weeks, the service was available to only 1 of 3 firms (firm A). Then, during weeks 5 through 8, the service crossed over to the other 2 firms (firms B and C) and was unavailable to the original firm. Firm assignments for residents assigned to the inpatient service for all 8 weeks did not change. Of the 16 residents assigned to firm A during the first 4 weeks, when the procedure service was available, 3 remained on the wards during the second 4 weeks, when the procedure service was not available.

We chose to collect data on 4 bedside procedures: paracentesis, thoracentesis, LP, and CVC. Similar to those at other teaching hospitals, our residents informally acquire the skills to perform these procedures while assisting and being assisted by more experienced senior residents in a see one, do one, teach one apprenticeship model of learning.4 To improve the training and performance of these bedside procedures, the Department of Medicine piloted a bedside procedure service to teach procedural skills and assist residents during these procedures. Use of the service, though voluntary, was actively encouraged at residents' monthly orientation meetings and regular conferences.

One attending inpatient physician (J.A.) staffed the bedside procedure service, which was available during normal work hours on weekdays. Requests for procedures were made by general medicine residents through an online database and, after approval by the procedure service attending physician, were performed under his direct supervision. A hand‐carried ultrasound (MicroMaxx, Sonosite, Inc., Bothell, WA) that generates a 2‐dimensional gray‐scale image was used to both confirm the presence and location of fluid prior to paracentesis and thoracentesis and provide real‐time guidance during central venous catheterization. When the bedside procedure service was unavailable, residents performed bedside procedures in the usual fashion, typically without direct attending physician supervision. But if requested, an on‐call chief medical resident with access to a hand‐carried ultrasound device used by the intensive care unit was available for assistance at any time.

Subjects

The study subjects were all patients admitted to the general medical service during the 8‐week pilot period. Patients were excluded if they had been discharged before arrival on the medical wards or if they were under the care of the general medicine service for less than 6 hours before discharge or transfer to another service. We chose 6 hours because we reasoned that such brief admissions were not potential candidates for invasive bedside procedures.

Data Collection

Each morning an investigator contacted the senior residents who had admitted patients during the previous 24‐hour shift and confirmed that newly admitted patients were under the care of the general medicine service for more than 6 hours. To examine how the number of attempts may have been affected by procedures done in the emergency room or intensive care units before admission to the general medicine service, investigators also asked admitting residents whether a bedside procedure had been attempted in the 72 hours before admission. Every general medicine service resident was asked to fill out a brief data collection form after an attempt to perform any procedure on the general medical wards. In addition, chief residents asked each member of the general medicine service at mandatory sign‐out rounds at the end of each weekday whether any procedures had been attempted, and on weekend days investigators contacted senior residents from each general medicine service team.

We report on this quality assurance study, which was conducted during a pilot phase. This report has been reviewed and judged exempt by our institutional review board.

Primary OutcomeNumber of Procedure Attempts

For all bedside procedures attempted by residents on the general medical wards, investigators determined whether the residents were members of firms that were offered the bedside procedure service and, if so, whether the procedure service attending directly supervised the procedure attempt. Multiple procedure attempts of the same type were counted for an individual patient if (1) the procedure attempts did not occur during the same admissions and (2) neither the physicians attempting the procedure nor the primary indications for it were the same. Therefore, neither attempts performed after initially unsuccessful ones nor repeated procedures, such as large‐volume therapeutic paracentesis and thoracentesis, were counted twice. We reasoned that when these criteria were met, procedure attempts could be considered independently.

Secondary Outcomes

Investigators asked residents who attempted procedures to indicate whether (1) the indication for the procedure was solely diagnostic or was, at least in part, therapeutic; (2) the procedure was successful; and (3) there were any immediate major periprocedural complications. A procedure was considered to have been successfully performed if it fulfilled 2 criteria: it had to be completed during a single continuous attempt, even if multiple sites or procedure kits were used; and it had to fulfill the indication for it being done. For example, if the indication for thoracentesis was therapeutic, this procedure would be considered successful if it yielded a large enough volume of fluid to alleviate the patient's symptoms, but if the indication was diagnostic, then thoracentesis would be considered successful if it yielded enough fluid for laboratory processing. Residents were asked to report any periprocedural complications that they considered major; 2 illustrative examples were provided: a pneumothorax and severe bleeding.

Data Analyses

On the basis of earlier pilot data, we estimated that 8%10% of all admissions to the general medicine service underwent at least 1 procedure (paracentesis, thoracentesis, lumbar puncture, or central vein catheterization). We planned for a sample size of 1900 admissions, which would have 80% power to detect a clinically meaningful 50% relative increase in the mean number of bedside procedures with a double‐sided alpha error of 0.05. We used permutation tests to compare the mean number of procedures attempted between firms and bootstrap simulation to construct 95% confidence intervals for those means and the differences between and ratios of them. Fisher's exact test was used to compare proportions of successfully performed procedures and preadmission procedure attempts. All analyses were conducted with Stata Statistical Software, Release 9 (StataCorp, LP, College Station, TX).

Subjects

During this 8‐week pilot study, there were 2157 admissions to the general medicine service. Among these admissions, 216 were excluded from our study because the patients did not arrive on the medical wards or were not under the care of the general medicine service for at least 6 hours before discharge or before being transferred to another service. Of the remaining 1941 admissions, 935 were to firms with the bedside procedure service available, and 1006 were to firms without the service available (Fig. 1)

Figure 1

Primary OutcomeNumber of Procedure Attempts

Overall, 122 patients underwent 145 procedure attempts that met our criteria for independence. The mean number of procedure attempts in firms offered the bedside procedure service was 48% higher (90 versus 61 per 1000 admissions; RR 1.48, 95% CI 1.062.10; P = .030; Fig. 1). When procedures attempted on weekends and holidays were excluded, the relative increase in procedure attempts in firms offered the bedside procedure service was even higher (70 versus 43 per 1000 admissions; RR 1.63, 95% CI 1.092.49; P = .023; Fig. 1). When grouped according to whether procedure attempts occurred before or after crossover of the procedure service, the mean number of procedure attempts in firms was higher when the service was offered: firm A dropped from 84 to 70 per 1000 admissions (P = .58) after losing the service, whereas firms B and C increased from 57 to 94 per 1000 admissions (P = .025) on gaining the service. There were 40 procedure attempts performed on patients within 72 hours before admission, but there was no difference between firms in the proportions of these preadmission procedures (P = .43).

Secondary Outcomes

Table 1 shows how of each type of procedure contributed to the overall difference. Attempts of CVC and therapeutic paracentesis and thoracentesis accounted for 86% of the overall increase in procedure attempts for admissions to firms offered the bedside procedure service, whereas only 14% of this increase was a result of diagnostic procedures. There were no differences in the proportions of successfully performed procedures, whether grouped by firm (P = 1.0) or by direct supervision from the procedure service attending (P = .64; Table 2). There were 3 self‐reported major periprocedural complications; all were related to excessive bleeding from CVC attempts. Two occurred without direct supervision from the bedside procedure service attending, one hemomediastinum from an internal jugular CVC attempt and one groin hematoma from a femoral CVC attempt. The third, a groin hematoma from a femoral CVC attempt, occurred during direct supervision from the bedside procedure service attending.

Design and Setting

In this prospective cohort study, the cohort was all patients admitted to the general medicine service at Cook County Hospital, a 500‐bed public teaching hospital in Chicago, Illinois, in January and February of 2006. The general medicine inpatient service is divided into 3 firms (A, B, and C), each with 4 separate teams of physicians and students. Admissions from the emergency department or other services in the hospital, such as intensive care units (which are closed and therefore staffed by separate teams of physicians), are distributed in sequence to on‐call teams from each firm. During the study period, the availability of a bedside procedure service varied by firm. Throughout the first 4 weeks, the service was available to only 1 of 3 firms (firm A). Then, during weeks 5 through 8, the service crossed over to the other 2 firms (firms B and C) and was unavailable to the original firm. Firm assignments for residents assigned to the inpatient service for all 8 weeks did not change. Of the 16 residents assigned to firm A during the first 4 weeks, when the procedure service was available, 3 remained on the wards during the second 4 weeks, when the procedure service was not available.

We chose to collect data on 4 bedside procedures: paracentesis, thoracentesis, LP, and CVC. Similar to those at other teaching hospitals, our residents informally acquire the skills to perform these procedures while assisting and being assisted by more experienced senior residents in a see one, do one, teach one apprenticeship model of learning.4 To improve the training and performance of these bedside procedures, the Department of Medicine piloted a bedside procedure service to teach procedural skills and assist residents during these procedures. Use of the service, though voluntary, was actively encouraged at residents' monthly orientation meetings and regular conferences.

One attending inpatient physician (J.A.) staffed the bedside procedure service, which was available during normal work hours on weekdays. Requests for procedures were made by general medicine residents through an online database and, after approval by the procedure service attending physician, were performed under his direct supervision. A hand‐carried ultrasound (MicroMaxx, Sonosite, Inc., Bothell, WA) that generates a 2‐dimensional gray‐scale image was used to both confirm the presence and location of fluid prior to paracentesis and thoracentesis and provide real‐time guidance during central venous catheterization. When the bedside procedure service was unavailable, residents performed bedside procedures in the usual fashion, typically without direct attending physician supervision. But if requested, an on‐call chief medical resident with access to a hand‐carried ultrasound device used by the intensive care unit was available for assistance at any time.

Subjects

The study subjects were all patients admitted to the general medical service during the 8‐week pilot period. Patients were excluded if they had been discharged before arrival on the medical wards or if they were under the care of the general medicine service for less than 6 hours before discharge or transfer to another service. We chose 6 hours because we reasoned that such brief admissions were not potential candidates for invasive bedside procedures.

Data Collection

Each morning an investigator contacted the senior residents who had admitted patients during the previous 24‐hour shift and confirmed that newly admitted patients were under the care of the general medicine service for more than 6 hours. To examine how the number of attempts may have been affected by procedures done in the emergency room or intensive care units before admission to the general medicine service, investigators also asked admitting residents whether a bedside procedure had been attempted in the 72 hours before admission. Every general medicine service resident was asked to fill out a brief data collection form after an attempt to perform any procedure on the general medical wards. In addition, chief residents asked each member of the general medicine service at mandatory sign‐out rounds at the end of each weekday whether any procedures had been attempted, and on weekend days investigators contacted senior residents from each general medicine service team.

We report on this quality assurance study, which was conducted during a pilot phase. This report has been reviewed and judged exempt by our institutional review board.

Primary OutcomeNumber of Procedure Attempts

For all bedside procedures attempted by residents on the general medical wards, investigators determined whether the residents were members of firms that were offered the bedside procedure service and, if so, whether the procedure service attending directly supervised the procedure attempt. Multiple procedure attempts of the same type were counted for an individual patient if (1) the procedure attempts did not occur during the same admissions and (2) neither the physicians attempting the procedure nor the primary indications for it were the same. Therefore, neither attempts performed after initially unsuccessful ones nor repeated procedures, such as large‐volume therapeutic paracentesis and thoracentesis, were counted twice. We reasoned that when these criteria were met, procedure attempts could be considered independently.

Secondary Outcomes

Investigators asked residents who attempted procedures to indicate whether (1) the indication for the procedure was solely diagnostic or was, at least in part, therapeutic; (2) the procedure was successful; and (3) there were any immediate major periprocedural complications. A procedure was considered to have been successfully performed if it fulfilled 2 criteria: it had to be completed during a single continuous attempt, even if multiple sites or procedure kits were used; and it had to fulfill the indication for it being done. For example, if the indication for thoracentesis was therapeutic, this procedure would be considered successful if it yielded a large enough volume of fluid to alleviate the patient's symptoms, but if the indication was diagnostic, then thoracentesis would be considered successful if it yielded enough fluid for laboratory processing. Residents were asked to report any periprocedural complications that they considered major; 2 illustrative examples were provided: a pneumothorax and severe bleeding.

Data Analyses

On the basis of earlier pilot data, we estimated that 8%10% of all admissions to the general medicine service underwent at least 1 procedure (paracentesis, thoracentesis, lumbar puncture, or central vein catheterization). We planned for a sample size of 1900 admissions, which would have 80% power to detect a clinically meaningful 50% relative increase in the mean number of bedside procedures with a double‐sided alpha error of 0.05. We used permutation tests to compare the mean number of procedures attempted between firms and bootstrap simulation to construct 95% confidence intervals for those means and the differences between and ratios of them. Fisher's exact test was used to compare proportions of successfully performed procedures and preadmission procedure attempts. All analyses were conducted with Stata Statistical Software, Release 9 (StataCorp, LP, College Station, TX).

Subjects

During this 8‐week pilot study, there were 2157 admissions to the general medicine service. Among these admissions, 216 were excluded from our study because the patients did not arrive on the medical wards or were not under the care of the general medicine service for at least 6 hours before discharge or before being transferred to another service. Of the remaining 1941 admissions, 935 were to firms with the bedside procedure service available, and 1006 were to firms without the service available (Fig. 1)

Figure 1

Primary OutcomeNumber of Procedure Attempts

Overall, 122 patients underwent 145 procedure attempts that met our criteria for independence. The mean number of procedure attempts in firms offered the bedside procedure service was 48% higher (90 versus 61 per 1000 admissions; RR 1.48, 95% CI 1.062.10; P = .030; Fig. 1). When procedures attempted on weekends and holidays were excluded, the relative increase in procedure attempts in firms offered the bedside procedure service was even higher (70 versus 43 per 1000 admissions; RR 1.63, 95% CI 1.092.49; P = .023; Fig. 1). When grouped according to whether procedure attempts occurred before or after crossover of the procedure service, the mean number of procedure attempts in firms was higher when the service was offered: firm A dropped from 84 to 70 per 1000 admissions (P = .58) after losing the service, whereas firms B and C increased from 57 to 94 per 1000 admissions (P = .025) on gaining the service. There were 40 procedure attempts performed on patients within 72 hours before admission, but there was no difference between firms in the proportions of these preadmission procedures (P = .43).

Secondary Outcomes

Table 1 shows how of each type of procedure contributed to the overall difference. Attempts of CVC and therapeutic paracentesis and thoracentesis accounted for 86% of the overall increase in procedure attempts for admissions to firms offered the bedside procedure service, whereas only 14% of this increase was a result of diagnostic procedures. There were no differences in the proportions of successfully performed procedures, whether grouped by firm (P = 1.0) or by direct supervision from the procedure service attending (P = .64; Table 2). There were 3 self‐reported major periprocedural complications; all were related to excessive bleeding from CVC attempts. Two occurred without direct supervision from the bedside procedure service attending, one hemomediastinum from an internal jugular CVC attempt and one groin hematoma from a femoral CVC attempt. The third, a groin hematoma from a femoral CVC attempt, occurred during direct supervision from the bedside procedure service attending.

Design and Setting

In this prospective cohort study, the cohort was all patients admitted to the general medicine service at Cook County Hospital, a 500‐bed public teaching hospital in Chicago, Illinois, in January and February of 2006. The general medicine inpatient service is divided into 3 firms (A, B, and C), each with 4 separate teams of physicians and students. Admissions from the emergency department or other services in the hospital, such as intensive care units (which are closed and therefore staffed by separate teams of physicians), are distributed in sequence to on‐call teams from each firm. During the study period, the availability of a bedside procedure service varied by firm. Throughout the first 4 weeks, the service was available to only 1 of 3 firms (firm A). Then, during weeks 5 through 8, the service crossed over to the other 2 firms (firms B and C) and was unavailable to the original firm. Firm assignments for residents assigned to the inpatient service for all 8 weeks did not change. Of the 16 residents assigned to firm A during the first 4 weeks, when the procedure service was available, 3 remained on the wards during the second 4 weeks, when the procedure service was not available.

We chose to collect data on 4 bedside procedures: paracentesis, thoracentesis, LP, and CVC. Similar to those at other teaching hospitals, our residents informally acquire the skills to perform these procedures while assisting and being assisted by more experienced senior residents in a see one, do one, teach one apprenticeship model of learning.4 To improve the training and performance of these bedside procedures, the Department of Medicine piloted a bedside procedure service to teach procedural skills and assist residents during these procedures. Use of the service, though voluntary, was actively encouraged at residents' monthly orientation meetings and regular conferences.

One attending inpatient physician (J.A.) staffed the bedside procedure service, which was available during normal work hours on weekdays. Requests for procedures were made by general medicine residents through an online database and, after approval by the procedure service attending physician, were performed under his direct supervision. A hand‐carried ultrasound (MicroMaxx, Sonosite, Inc., Bothell, WA) that generates a 2‐dimensional gray‐scale image was used to both confirm the presence and location of fluid prior to paracentesis and thoracentesis and provide real‐time guidance during central venous catheterization. When the bedside procedure service was unavailable, residents performed bedside procedures in the usual fashion, typically without direct attending physician supervision. But if requested, an on‐call chief medical resident with access to a hand‐carried ultrasound device used by the intensive care unit was available for assistance at any time.

Subjects

The study subjects were all patients admitted to the general medical service during the 8‐week pilot period. Patients were excluded if they had been discharged before arrival on the medical wards or if they were under the care of the general medicine service for less than 6 hours before discharge or transfer to another service. We chose 6 hours because we reasoned that such brief admissions were not potential candidates for invasive bedside procedures.

Data Collection

Each morning an investigator contacted the senior residents who had admitted patients during the previous 24‐hour shift and confirmed that newly admitted patients were under the care of the general medicine service for more than 6 hours. To examine how the number of attempts may have been affected by procedures done in the emergency room or intensive care units before admission to the general medicine service, investigators also asked admitting residents whether a bedside procedure had been attempted in the 72 hours before admission. Every general medicine service resident was asked to fill out a brief data collection form after an attempt to perform any procedure on the general medical wards. In addition, chief residents asked each member of the general medicine service at mandatory sign‐out rounds at the end of each weekday whether any procedures had been attempted, and on weekend days investigators contacted senior residents from each general medicine service team.

We report on this quality assurance study, which was conducted during a pilot phase. This report has been reviewed and judged exempt by our institutional review board.

Primary OutcomeNumber of Procedure Attempts

For all bedside procedures attempted by residents on the general medical wards, investigators determined whether the residents were members of firms that were offered the bedside procedure service and, if so, whether the procedure service attending directly supervised the procedure attempt. Multiple procedure attempts of the same type were counted for an individual patient if (1) the procedure attempts did not occur during the same admissions and (2) neither the physicians attempting the procedure nor the primary indications for it were the same. Therefore, neither attempts performed after initially unsuccessful ones nor repeated procedures, such as large‐volume therapeutic paracentesis and thoracentesis, were counted twice. We reasoned that when these criteria were met, procedure attempts could be considered independently.

Secondary Outcomes

Investigators asked residents who attempted procedures to indicate whether (1) the indication for the procedure was solely diagnostic or was, at least in part, therapeutic; (2) the procedure was successful; and (3) there were any immediate major periprocedural complications. A procedure was considered to have been successfully performed if it fulfilled 2 criteria: it had to be completed during a single continuous attempt, even if multiple sites or procedure kits were used; and it had to fulfill the indication for it being done. For example, if the indication for thoracentesis was therapeutic, this procedure would be considered successful if it yielded a large enough volume of fluid to alleviate the patient's symptoms, but if the indication was diagnostic, then thoracentesis would be considered successful if it yielded enough fluid for laboratory processing. Residents were asked to report any periprocedural complications that they considered major; 2 illustrative examples were provided: a pneumothorax and severe bleeding.

Data Analyses

On the basis of earlier pilot data, we estimated that 8%10% of all admissions to the general medicine service underwent at least 1 procedure (paracentesis, thoracentesis, lumbar puncture, or central vein catheterization). We planned for a sample size of 1900 admissions, which would have 80% power to detect a clinically meaningful 50% relative increase in the mean number of bedside procedures with a double‐sided alpha error of 0.05. We used permutation tests to compare the mean number of procedures attempted between firms and bootstrap simulation to construct 95% confidence intervals for those means and the differences between and ratios of them. Fisher's exact test was used to compare proportions of successfully performed procedures and preadmission procedure attempts. All analyses were conducted with Stata Statistical Software, Release 9 (StataCorp, LP, College Station, TX).

Subjects

During this 8‐week pilot study, there were 2157 admissions to the general medicine service. Among these admissions, 216 were excluded from our study because the patients did not arrive on the medical wards or were not under the care of the general medicine service for at least 6 hours before discharge or before being transferred to another service. Of the remaining 1941 admissions, 935 were to firms with the bedside procedure service available, and 1006 were to firms without the service available (Fig. 1)

Figure 1

Primary OutcomeNumber of Procedure Attempts

Overall, 122 patients underwent 145 procedure attempts that met our criteria for independence. The mean number of procedure attempts in firms offered the bedside procedure service was 48% higher (90 versus 61 per 1000 admissions; RR 1.48, 95% CI 1.062.10; P = .030; Fig. 1). When procedures attempted on weekends and holidays were excluded, the relative increase in procedure attempts in firms offered the bedside procedure service was even higher (70 versus 43 per 1000 admissions; RR 1.63, 95% CI 1.092.49; P = .023; Fig. 1). When grouped according to whether procedure attempts occurred before or after crossover of the procedure service, the mean number of procedure attempts in firms was higher when the service was offered: firm A dropped from 84 to 70 per 1000 admissions (P = .58) after losing the service, whereas firms B and C increased from 57 to 94 per 1000 admissions (P = .025) on gaining the service. There were 40 procedure attempts performed on patients within 72 hours before admission, but there was no difference between firms in the proportions of these preadmission procedures (P = .43).

Secondary Outcomes

Table 1 shows how of each type of procedure contributed to the overall difference. Attempts of CVC and therapeutic paracentesis and thoracentesis accounted for 86% of the overall increase in procedure attempts for admissions to firms offered the bedside procedure service, whereas only 14% of this increase was a result of diagnostic procedures. There were no differences in the proportions of successfully performed procedures, whether grouped by firm (P = 1.0) or by direct supervision from the procedure service attending (P = .64; Table 2). There were 3 self‐reported major periprocedural complications; all were related to excessive bleeding from CVC attempts. Two occurred without direct supervision from the bedside procedure service attending, one hemomediastinum from an internal jugular CVC attempt and one groin hematoma from a femoral CVC attempt. The third, a groin hematoma from a femoral CVC attempt, occurred during direct supervision from the bedside procedure service attending.

Data and Assumptions

A 3‐ and 4‐tiered computer model (with the tiers reflecting the variables presence of lung disease, having siblings, and sibling immunization status and the fourth tier reflecting parental immunization in the NICU as a function of the immunization program) assessing influenza vaccination status of parents of a cohort of 2632 patients admitted to the New York Regional Perinatal Center NICU during the influenza season of 2003‐2004 was constructed using the viewpoint of a large multinetwork medical center predominantly serving a lower socioeconomic status population. The likelihood of influenza infection of an infant, the need for infant hospitalization, subsequent length of stay, and the need for the patient to have outpatient physician visits were based on the following clinical variables: lung disease in the infant (defined as a 28‐day‐old patient whose birth weight was less than 1500 g being oxygen dependent); having school‐age siblings, sibling vaccination status; parental vaccination status; and parental compliance. Variables of the model were based on published results when possible. For the purposes of this model, we assumed a 10% reduction in influenza infectivity for parents of children who were immunized in the absence of other confounders based on the risk of needing medical attention of children less than 6 months old for documented influenza with parental vaccination in the 9 states that make up the Emerging Infections Program Network of the Centers for Disease Control.1719 Infected patients younger than 6 months of age were also programmed to have a 10% chance of an outpatient hospitalization visit. No deaths were introduced in the cohort. An outline of the different groups into which patients were classified (before and after the influenza vaccination campaign) is outlined in Figure 1.

Figure 1

Direct Costs

Indirect Costs

For each outpatient office visit, we used the cost‐estimation scheme outlined by Yount et al.20. We assumed that 1 parent accompanied the infant and 3 hours of lost work should be accounted for. Using the U.S. Bureau of Labor and Statistics 2002 average wage of $17/hour, lost wages for each extra outpatient visits were tabulated by (number of extra MD visits per group 3 hours $17/hour).24 No travel or transportation costs were considered.

Hospitalization

For each hospitalization, we assumed 1 parent stayed with an infant at bedside during the infant's inpatient stay. We calculated the average length of stay for patients with lung disease as 8 days and for those without lung disease as 4.5 days. Calculations were obtained using the following formula: (number of infants in each group from the New York Regional Perinatal Center 2004 Database hospitalization rate of each group number of days hospitalized $17/hour 8 work hours/day 5/7 workdays/week).

Sensitivity Analysis

We evaluated the sensitivity of the model to variations in the assumptions made. We varied the sibling immunization rate from 12% to 17% and the reduction in hospitalization for parents who received influenza vaccine from 10% to 20%. A summary of variables used in the analysis is included in Table 1.

Influenza Costs Prior to Implementation of NICU‐Based Parental Vaccination

Direct and indirect costs of influenza hospitalization of the NICU graduates are summarized in Table 2. The total per‐patient cost of influenza vaccination obtained in the NICU for the 2632 patients in the source data 1 one season was $181.20. NICU patients with lung disease and siblings who were not protected from or immunized for influenza demonstrated the greatest per capita inpatient cost, $1925/patient. Vaccination of patients without lung disease who had no siblings cost $51, the same amount that it cost to vaccinate patients without lung disease who had vaccinated siblings.

Outpatient costs of influenza hospitalization based on source data revealed summarized costs for 1 season of $6.80/patient. This reflected 245 excess primary care visits at a total cost of $5569.20. The cost of excess prescriptions of the antibiotic amoxicillin because of misdiagnoses. Indirect costs secondary to parent lost work hours while attending to their infants in the hospital totaled $12,530.70. Thus, the total cost of influenza in the source population for 1 season including inpatient, outpatient, direct, and indirect costs was $188/patient.

Influenza Costs after Implementation of an NICU‐Based Parental Vaccination Program

Direct and indirect costs of influenza hospitalization for neonates with lung disease are summarized in Table 3. The introduction of parental vaccination decreased the per‐patient cost in the cohort of patients with lung disease and unprotected siblings to $1732 from $1925. This group showed the largest cost savings compared with the costs for this group prior to introduction of the campaign.

Direct and indirect costs of influenza hospitalization for infants without lung disease are summarized in Table 4. The introduction of parental vaccination to disrupt the cycle of infectious transmission to infant decreased per‐patient costs in patients whose parents and siblings received vaccinations to $45. This reduction of $6/patient was the greatest savings among all the groups in the cohort without lung disease.

Outpatient costs were reduced after the introduction of the campaign to $1.40/patient, reflecting the decrease in the number of outpatient visits from 245 to 51. Thus, the total cost of influenza in the source population after the introduction of an NICU‐based parental vaccination campaign was $200/patient. The $193/patient savings in the lung disease cohort with unprotected siblings ($1925 vs. $1732) was not sufficient to cover the increased cost of the vaccine. For this population of 2632 NICU patients, administration of NICU‐based parental influenza cost $12 extra/patient.

Financial Modeling Based on Source Data

Using the financial model, cost per patient was determined using the same estimates of incidence of the variables (ie, lung disease, siblings); only the number of enrollees in the program was varied. The relationship of cost per patient with number of NICU patients is shown in Figure 2. Cost per patient was zero at 4000 patients. Beyond that point, cost savings occurred, increasing with number of NICU admissions.

Figure 2

Estimating a 1‐day ICU admission rate of 0.5% at $3770/day reduces the required patient population for costs/patient to zero. This occurs at 3700 patients. Initially there is no added benefit with ICU admission, as the overall patient population is not large enough to support a significant ICU burden. As the population increases to 3000 patients, cost savings begin.

The relationship of variable immunization rates in siblings of the 2632 NICU patients in the source data is presented in Figure 3. Cost savings were not achieved until 37% of siblings had been immunized. A steep reduction in cost was seen as the immunization rate of siblings increased in the cohort. Marginal cost effectiveness was also increased in sibling immunization, meaning greater cost savings is achieved by immunizing a sibling of a high‐risk infant than by immunizing the parents, reflecting that siblings are more likely than parents to be vectors of disease in multichild households.

Figure 3

Data and Assumptions

A 3‐ and 4‐tiered computer model (with the tiers reflecting the variables presence of lung disease, having siblings, and sibling immunization status and the fourth tier reflecting parental immunization in the NICU as a function of the immunization program) assessing influenza vaccination status of parents of a cohort of 2632 patients admitted to the New York Regional Perinatal Center NICU during the influenza season of 2003‐2004 was constructed using the viewpoint of a large multinetwork medical center predominantly serving a lower socioeconomic status population. The likelihood of influenza infection of an infant, the need for infant hospitalization, subsequent length of stay, and the need for the patient to have outpatient physician visits were based on the following clinical variables: lung disease in the infant (defined as a 28‐day‐old patient whose birth weight was less than 1500 g being oxygen dependent); having school‐age siblings, sibling vaccination status; parental vaccination status; and parental compliance. Variables of the model were based on published results when possible. For the purposes of this model, we assumed a 10% reduction in influenza infectivity for parents of children who were immunized in the absence of other confounders based on the risk of needing medical attention of children less than 6 months old for documented influenza with parental vaccination in the 9 states that make up the Emerging Infections Program Network of the Centers for Disease Control.1719 Infected patients younger than 6 months of age were also programmed to have a 10% chance of an outpatient hospitalization visit. No deaths were introduced in the cohort. An outline of the different groups into which patients were classified (before and after the influenza vaccination campaign) is outlined in Figure 1.

Figure 1

Direct Costs

Indirect Costs

For each outpatient office visit, we used the cost‐estimation scheme outlined by Yount et al.20. We assumed that 1 parent accompanied the infant and 3 hours of lost work should be accounted for. Using the U.S. Bureau of Labor and Statistics 2002 average wage of $17/hour, lost wages for each extra outpatient visits were tabulated by (number of extra MD visits per group 3 hours $17/hour).24 No travel or transportation costs were considered.

Hospitalization

For each hospitalization, we assumed 1 parent stayed with an infant at bedside during the infant's inpatient stay. We calculated the average length of stay for patients with lung disease as 8 days and for those without lung disease as 4.5 days. Calculations were obtained using the following formula: (number of infants in each group from the New York Regional Perinatal Center 2004 Database hospitalization rate of each group number of days hospitalized $17/hour 8 work hours/day 5/7 workdays/week).

Sensitivity Analysis

We evaluated the sensitivity of the model to variations in the assumptions made. We varied the sibling immunization rate from 12% to 17% and the reduction in hospitalization for parents who received influenza vaccine from 10% to 20%. A summary of variables used in the analysis is included in Table 1.

Influenza Costs Prior to Implementation of NICU‐Based Parental Vaccination

Direct and indirect costs of influenza hospitalization of the NICU graduates are summarized in Table 2. The total per‐patient cost of influenza vaccination obtained in the NICU for the 2632 patients in the source data 1 one season was $181.20. NICU patients with lung disease and siblings who were not protected from or immunized for influenza demonstrated the greatest per capita inpatient cost, $1925/patient. Vaccination of patients without lung disease who had no siblings cost $51, the same amount that it cost to vaccinate patients without lung disease who had vaccinated siblings.

Outpatient costs of influenza hospitalization based on source data revealed summarized costs for 1 season of $6.80/patient. This reflected 245 excess primary care visits at a total cost of $5569.20. The cost of excess prescriptions of the antibiotic amoxicillin because of misdiagnoses. Indirect costs secondary to parent lost work hours while attending to their infants in the hospital totaled $12,530.70. Thus, the total cost of influenza in the source population for 1 season including inpatient, outpatient, direct, and indirect costs was $188/patient.

Influenza Costs after Implementation of an NICU‐Based Parental Vaccination Program

Direct and indirect costs of influenza hospitalization for neonates with lung disease are summarized in Table 3. The introduction of parental vaccination decreased the per‐patient cost in the cohort of patients with lung disease and unprotected siblings to $1732 from $1925. This group showed the largest cost savings compared with the costs for this group prior to introduction of the campaign.

Direct and indirect costs of influenza hospitalization for infants without lung disease are summarized in Table 4. The introduction of parental vaccination to disrupt the cycle of infectious transmission to infant decreased per‐patient costs in patients whose parents and siblings received vaccinations to $45. This reduction of $6/patient was the greatest savings among all the groups in the cohort without lung disease.

Outpatient costs were reduced after the introduction of the campaign to $1.40/patient, reflecting the decrease in the number of outpatient visits from 245 to 51. Thus, the total cost of influenza in the source population after the introduction of an NICU‐based parental vaccination campaign was $200/patient. The $193/patient savings in the lung disease cohort with unprotected siblings ($1925 vs. $1732) was not sufficient to cover the increased cost of the vaccine. For this population of 2632 NICU patients, administration of NICU‐based parental influenza cost $12 extra/patient.

Financial Modeling Based on Source Data

Using the financial model, cost per patient was determined using the same estimates of incidence of the variables (ie, lung disease, siblings); only the number of enrollees in the program was varied. The relationship of cost per patient with number of NICU patients is shown in Figure 2. Cost per patient was zero at 4000 patients. Beyond that point, cost savings occurred, increasing with number of NICU admissions.

Figure 2

Estimating a 1‐day ICU admission rate of 0.5% at $3770/day reduces the required patient population for costs/patient to zero. This occurs at 3700 patients. Initially there is no added benefit with ICU admission, as the overall patient population is not large enough to support a significant ICU burden. As the population increases to 3000 patients, cost savings begin.

The relationship of variable immunization rates in siblings of the 2632 NICU patients in the source data is presented in Figure 3. Cost savings were not achieved until 37% of siblings had been immunized. A steep reduction in cost was seen as the immunization rate of siblings increased in the cohort. Marginal cost effectiveness was also increased in sibling immunization, meaning greater cost savings is achieved by immunizing a sibling of a high‐risk infant than by immunizing the parents, reflecting that siblings are more likely than parents to be vectors of disease in multichild households.

Figure 3

Data and Assumptions

A 3‐ and 4‐tiered computer model (with the tiers reflecting the variables presence of lung disease, having siblings, and sibling immunization status and the fourth tier reflecting parental immunization in the NICU as a function of the immunization program) assessing influenza vaccination status of parents of a cohort of 2632 patients admitted to the New York Regional Perinatal Center NICU during the influenza season of 2003‐2004 was constructed using the viewpoint of a large multinetwork medical center predominantly serving a lower socioeconomic status population. The likelihood of influenza infection of an infant, the need for infant hospitalization, subsequent length of stay, and the need for the patient to have outpatient physician visits were based on the following clinical variables: lung disease in the infant (defined as a 28‐day‐old patient whose birth weight was less than 1500 g being oxygen dependent); having school‐age siblings, sibling vaccination status; parental vaccination status; and parental compliance. Variables of the model were based on published results when possible. For the purposes of this model, we assumed a 10% reduction in influenza infectivity for parents of children who were immunized in the absence of other confounders based on the risk of needing medical attention of children less than 6 months old for documented influenza with parental vaccination in the 9 states that make up the Emerging Infections Program Network of the Centers for Disease Control.1719 Infected patients younger than 6 months of age were also programmed to have a 10% chance of an outpatient hospitalization visit. No deaths were introduced in the cohort. An outline of the different groups into which patients were classified (before and after the influenza vaccination campaign) is outlined in Figure 1.

Figure 1

Direct Costs

Indirect Costs

For each outpatient office visit, we used the cost‐estimation scheme outlined by Yount et al.20. We assumed that 1 parent accompanied the infant and 3 hours of lost work should be accounted for. Using the U.S. Bureau of Labor and Statistics 2002 average wage of $17/hour, lost wages for each extra outpatient visits were tabulated by (number of extra MD visits per group 3 hours $17/hour).24 No travel or transportation costs were considered.

Hospitalization

For each hospitalization, we assumed 1 parent stayed with an infant at bedside during the infant's inpatient stay. We calculated the average length of stay for patients with lung disease as 8 days and for those without lung disease as 4.5 days. Calculations were obtained using the following formula: (number of infants in each group from the New York Regional Perinatal Center 2004 Database hospitalization rate of each group number of days hospitalized $17/hour 8 work hours/day 5/7 workdays/week).

Sensitivity Analysis

We evaluated the sensitivity of the model to variations in the assumptions made. We varied the sibling immunization rate from 12% to 17% and the reduction in hospitalization for parents who received influenza vaccine from 10% to 20%. A summary of variables used in the analysis is included in Table 1.

Influenza Costs Prior to Implementation of NICU‐Based Parental Vaccination

Direct and indirect costs of influenza hospitalization of the NICU graduates are summarized in Table 2. The total per‐patient cost of influenza vaccination obtained in the NICU for the 2632 patients in the source data 1 one season was $181.20. NICU patients with lung disease and siblings who were not protected from or immunized for influenza demonstrated the greatest per capita inpatient cost, $1925/patient. Vaccination of patients without lung disease who had no siblings cost $51, the same amount that it cost to vaccinate patients without lung disease who had vaccinated siblings.

Outpatient costs of influenza hospitalization based on source data revealed summarized costs for 1 season of $6.80/patient. This reflected 245 excess primary care visits at a total cost of $5569.20. The cost of excess prescriptions of the antibiotic amoxicillin because of misdiagnoses. Indirect costs secondary to parent lost work hours while attending to their infants in the hospital totaled $12,530.70. Thus, the total cost of influenza in the source population for 1 season including inpatient, outpatient, direct, and indirect costs was $188/patient.

Influenza Costs after Implementation of an NICU‐Based Parental Vaccination Program

Direct and indirect costs of influenza hospitalization for neonates with lung disease are summarized in Table 3. The introduction of parental vaccination decreased the per‐patient cost in the cohort of patients with lung disease and unprotected siblings to $1732 from $1925. This group showed the largest cost savings compared with the costs for this group prior to introduction of the campaign.

Direct and indirect costs of influenza hospitalization for infants without lung disease are summarized in Table 4. The introduction of parental vaccination to disrupt the cycle of infectious transmission to infant decreased per‐patient costs in patients whose parents and siblings received vaccinations to $45. This reduction of $6/patient was the greatest savings among all the groups in the cohort without lung disease.

Outpatient costs were reduced after the introduction of the campaign to $1.40/patient, reflecting the decrease in the number of outpatient visits from 245 to 51. Thus, the total cost of influenza in the source population after the introduction of an NICU‐based parental vaccination campaign was $200/patient. The $193/patient savings in the lung disease cohort with unprotected siblings ($1925 vs. $1732) was not sufficient to cover the increased cost of the vaccine. For this population of 2632 NICU patients, administration of NICU‐based parental influenza cost $12 extra/patient.

Financial Modeling Based on Source Data

Using the financial model, cost per patient was determined using the same estimates of incidence of the variables (ie, lung disease, siblings); only the number of enrollees in the program was varied. The relationship of cost per patient with number of NICU patients is shown in Figure 2. Cost per patient was zero at 4000 patients. Beyond that point, cost savings occurred, increasing with number of NICU admissions.

Figure 2

Estimating a 1‐day ICU admission rate of 0.5% at $3770/day reduces the required patient population for costs/patient to zero. This occurs at 3700 patients. Initially there is no added benefit with ICU admission, as the overall patient population is not large enough to support a significant ICU burden. As the population increases to 3000 patients, cost savings begin.

The relationship of variable immunization rates in siblings of the 2632 NICU patients in the source data is presented in Figure 3. Cost savings were not achieved until 37% of siblings had been immunized. A steep reduction in cost was seen as the immunization rate of siblings increased in the cohort. Marginal cost effectiveness was also increased in sibling immunization, meaning greater cost savings is achieved by immunizing a sibling of a high‐risk infant than by immunizing the parents, reflecting that siblings are more likely than parents to be vectors of disease in multichild households.

Figure 3

Setting and Study Population

We enrolled patients from newborn to 18 years of age who had a diagnosis compatible with neurological impairment and who received their first fundoplication for GERD between January 2005 and July 2006 at Primary Children's Medical Center (PCMC), in Salt Lake City, Utah. PCMC is a 232‐bed children's hospital in the Intermountain West owned and operated by Intermountain Healthcare, Inc., a large vertically integrated health care delivery system that serves as both the primary hospital for Salt Lake County and as the tertiary‐care hospital for 4 additional states (Wyoming, Nevada, Idaho, and Montana).25 Patients who had a previous gastrojejunal feeding tube were excluded as were patients who had a previous fundoplication, as these procedures may have biased their reported quality of life, our main outcome measure.

Patients were included in the study if they had GERD (based on clinical history or testing) that had been refractory to medical management (defined as continued gastroesophageal reflux symptoms despite antireflux medications). GERD was defined using the North American Society for Pediatric Gastroenterology, Hepatology and Nutrition (NASPGHAN) criteria.26 These include: the presence of clinical symptoms and at least 1 abnormal result from an upper gastrointestinal x‐ray series (recognizing that this test is neither sensitive nor specific for reflux), pH probe, upper gastrointestinal endoscopy with biopsy, nuclear medicine scan, or a modified barium swallow. As this was a prospective observational study, physicians were allowed to order testing as their practice dictated. Patients were excluded if they had neurological impairment but lacked objective documentation of GERD using the NASPGHAN recommendations, unless there were obvious clinical indications such as witnessed vomiting and aspiration (N = 3). No patient received a prophylactic fundoplication (fundoplication without documented GERD).

Study Design

This is an ongoing prospective longitudinal study. Patients who had a first fundoplication at PCMC were identified by the surgical service, with weekly lists shared with the research team. Patients were approached by a research assistant during that initial hospitalization to see if they met inclusion criteria for the study using data from the medical records and surgical team when necessary.

Data Variables and Sources

Indications for the fundoplication, performance and results of diagnostic testing for GERD, complications of the fundoplication, and reasons for the neurological impairment were obtained through review of the electronic and paper medical records. Mortality data, subsequent emergency department visits, and admissions to the hospital were obtained using Intermountain Healthcare's electronic data warehouse, which merges clinical, financial, and administrative data including the Utah Vital Statistics database.

Neurological impairment was defined from 2 sources: (1) clinical diagnoses as identified by providers and (2) International Classification of Diseases Codes Modified, version 9 (ICD‐9 CM) identified a priori as indicating neurological impairment.

Instruments and Study Outcomes

Functional status was measured using the WeeFIM. This instrument has been tested and shown to be valid and reliable for children more than 6 months old with neurodevelopmental disabilities including spina bifida and Down syndrome.2732 WeeFIM is a self‐administered parent instrument composed of 18 items and 6 domains (self‐care, sphincter control, transfers, locomotion, communication, and social cognition).33 The WeeFIM allows patients to be stratified into areas of function from severely impaired to normal.

The primary outcome was child quality of life as measured by the Child Health Questionnaire Parental Form 50 (CHQ‐PF50). Caregiver quality of life was measured using the Short‐Form Health Status Survey (SF‐36) and Parenting Stress Index/Short Form (PSI/SF). The CHQ‐PF50 is a self‐administered parent questionnaire of 50 questions that measures 6 domains, including physical function and abilities, pain and discomfort, general health perception, behavior, temperament and moods, and satisfaction with growth and development.34 This instrument has been tested for validity and reliability in children with cerebral palsy.35 The SF‐36 is a widely accepted measure of health status that measures 8 domains of health: physical functioning, role limitations due to physical problems, bodily pain, general health, vitality, social functioning, role limitations due to emotional problems, and mental health. The SF‐36 has been well studied and has been used to measure the effect on a caregiver's quality of life associated with caring for a chronically ill child with significant medical problems.36, 37 Higher scores in each domain of both the CHQPF50 and the SF‐36 reflect better quality of life. Caregiver stress was measured using the PSI/SF (Psychological Assessment Resources Inc, Odessa, FL).38 In the PSI/SF a parent rates the parentchild dyad on 36 items that are summarized in 3 subscales: parental distress, parentchild dysfunctional interaction, and difficult child. A parent scores each item as strongly agree, agree, not sure, disagree, or strongly disagree. The sum of the 3 subscale scores is the total stress score. Higher scores denote a greater degree of stress. The PSI/SF has been validated in several studies for caregivers of children with chronic diseases.3943

The CHQ‐PF50, SF‐36, and PSI/SF questionnaires and the WeeFIM functional status measure were administered to each study patient and caregiver in person at enrollment (baseline) and by mail 1 month after fundoplication. A follow‐up postcard reminder was mailed 1 week after the initial mailing. Second and final mailings were sent to nonresponders 3 and 5 weeks, respectively, after the initial mailing.

Secondary outcomes included rates of complications including failure of the fundoplication. Complications were defined as a subsequent emergency department visit, hospitalization, or death related to the surgery, gastrostomy tube, or aspiration pneumonia. Failures were defined as a second fundoplication or the insertion of a gastrojejunal feeding tube as nonsurgical management of recurrent GERD and/or paraesophageal hernia. Secondary outcomes were followed from the time of the first fundoplication until 1 month after surgery.

Analyses

The differential effect of the fundoplication on the quality of life measures was assessed and quantified through statistical analysis. Because the primary interest was to measure change in baseline characteristics over time, repeated‐measures models were used to compare the group relative to changes in functional status. In particular, changes from baseline values were modeled 1 month after the procedure. The Kenward‐Roger approximation of degrees of freedom was used to compute P values from the overall tests.44 Repeated‐measures models were fit to the data using restricted maximum likelihood estimation. An autoregressive covariance matrix was assumed for the multiple measurements of each individual, thus limiting the number of restrictions forced by the model on the data. The repeated‐measures models used all the available data on participants, including those who dropped out of the study. To obtain the most accurate comparison of the study group, the covariate of functional status at baseline was taken into account in the fitted models. Statistical analyses were performed with SAS statistical software (version 9.13; SAS Institute, Cary NC). Student t tests were performed for comparison of means of the quality‐of‐life domains for the study cohort compared to either the general population or specific groups of patients for comparative purposes.

The study was approved by the institutional review boards of the University of Utah Health Sciences Center and Primary Children's Medical Center.

Setting and Study Population

We enrolled patients from newborn to 18 years of age who had a diagnosis compatible with neurological impairment and who received their first fundoplication for GERD between January 2005 and July 2006 at Primary Children's Medical Center (PCMC), in Salt Lake City, Utah. PCMC is a 232‐bed children's hospital in the Intermountain West owned and operated by Intermountain Healthcare, Inc., a large vertically integrated health care delivery system that serves as both the primary hospital for Salt Lake County and as the tertiary‐care hospital for 4 additional states (Wyoming, Nevada, Idaho, and Montana).25 Patients who had a previous gastrojejunal feeding tube were excluded as were patients who had a previous fundoplication, as these procedures may have biased their reported quality of life, our main outcome measure.

Patients were included in the study if they had GERD (based on clinical history or testing) that had been refractory to medical management (defined as continued gastroesophageal reflux symptoms despite antireflux medications). GERD was defined using the North American Society for Pediatric Gastroenterology, Hepatology and Nutrition (NASPGHAN) criteria.26 These include: the presence of clinical symptoms and at least 1 abnormal result from an upper gastrointestinal x‐ray series (recognizing that this test is neither sensitive nor specific for reflux), pH probe, upper gastrointestinal endoscopy with biopsy, nuclear medicine scan, or a modified barium swallow. As this was a prospective observational study, physicians were allowed to order testing as their practice dictated. Patients were excluded if they had neurological impairment but lacked objective documentation of GERD using the NASPGHAN recommendations, unless there were obvious clinical indications such as witnessed vomiting and aspiration (N = 3). No patient received a prophylactic fundoplication (fundoplication without documented GERD).

Study Design

This is an ongoing prospective longitudinal study. Patients who had a first fundoplication at PCMC were identified by the surgical service, with weekly lists shared with the research team. Patients were approached by a research assistant during that initial hospitalization to see if they met inclusion criteria for the study using data from the medical records and surgical team when necessary.

Data Variables and Sources

Indications for the fundoplication, performance and results of diagnostic testing for GERD, complications of the fundoplication, and reasons for the neurological impairment were obtained through review of the electronic and paper medical records. Mortality data, subsequent emergency department visits, and admissions to the hospital were obtained using Intermountain Healthcare's electronic data warehouse, which merges clinical, financial, and administrative data including the Utah Vital Statistics database.

Neurological impairment was defined from 2 sources: (1) clinical diagnoses as identified by providers and (2) International Classification of Diseases Codes Modified, version 9 (ICD‐9 CM) identified a priori as indicating neurological impairment.

Instruments and Study Outcomes

Functional status was measured using the WeeFIM. This instrument has been tested and shown to be valid and reliable for children more than 6 months old with neurodevelopmental disabilities including spina bifida and Down syndrome.2732 WeeFIM is a self‐administered parent instrument composed of 18 items and 6 domains (self‐care, sphincter control, transfers, locomotion, communication, and social cognition).33 The WeeFIM allows patients to be stratified into areas of function from severely impaired to normal.

The primary outcome was child quality of life as measured by the Child Health Questionnaire Parental Form 50 (CHQ‐PF50). Caregiver quality of life was measured using the Short‐Form Health Status Survey (SF‐36) and Parenting Stress Index/Short Form (PSI/SF). The CHQ‐PF50 is a self‐administered parent questionnaire of 50 questions that measures 6 domains, including physical function and abilities, pain and discomfort, general health perception, behavior, temperament and moods, and satisfaction with growth and development.34 This instrument has been tested for validity and reliability in children with cerebral palsy.35 The SF‐36 is a widely accepted measure of health status that measures 8 domains of health: physical functioning, role limitations due to physical problems, bodily pain, general health, vitality, social functioning, role limitations due to emotional problems, and mental health. The SF‐36 has been well studied and has been used to measure the effect on a caregiver's quality of life associated with caring for a chronically ill child with significant medical problems.36, 37 Higher scores in each domain of both the CHQPF50 and the SF‐36 reflect better quality of life. Caregiver stress was measured using the PSI/SF (Psychological Assessment Resources Inc, Odessa, FL).38 In the PSI/SF a parent rates the parentchild dyad on 36 items that are summarized in 3 subscales: parental distress, parentchild dysfunctional interaction, and difficult child. A parent scores each item as strongly agree, agree, not sure, disagree, or strongly disagree. The sum of the 3 subscale scores is the total stress score. Higher scores denote a greater degree of stress. The PSI/SF has been validated in several studies for caregivers of children with chronic diseases.3943

The CHQ‐PF50, SF‐36, and PSI/SF questionnaires and the WeeFIM functional status measure were administered to each study patient and caregiver in person at enrollment (baseline) and by mail 1 month after fundoplication. A follow‐up postcard reminder was mailed 1 week after the initial mailing. Second and final mailings were sent to nonresponders 3 and 5 weeks, respectively, after the initial mailing.

Secondary outcomes included rates of complications including failure of the fundoplication. Complications were defined as a subsequent emergency department visit, hospitalization, or death related to the surgery, gastrostomy tube, or aspiration pneumonia. Failures were defined as a second fundoplication or the insertion of a gastrojejunal feeding tube as nonsurgical management of recurrent GERD and/or paraesophageal hernia. Secondary outcomes were followed from the time of the first fundoplication until 1 month after surgery.

Analyses

The differential effect of the fundoplication on the quality of life measures was assessed and quantified through statistical analysis. Because the primary interest was to measure change in baseline characteristics over time, repeated‐measures models were used to compare the group relative to changes in functional status. In particular, changes from baseline values were modeled 1 month after the procedure. The Kenward‐Roger approximation of degrees of freedom was used to compute P values from the overall tests.44 Repeated‐measures models were fit to the data using restricted maximum likelihood estimation. An autoregressive covariance matrix was assumed for the multiple measurements of each individual, thus limiting the number of restrictions forced by the model on the data. The repeated‐measures models used all the available data on participants, including those who dropped out of the study. To obtain the most accurate comparison of the study group, the covariate of functional status at baseline was taken into account in the fitted models. Statistical analyses were performed with SAS statistical software (version 9.13; SAS Institute, Cary NC). Student t tests were performed for comparison of means of the quality‐of‐life domains for the study cohort compared to either the general population or specific groups of patients for comparative purposes.

The study was approved by the institutional review boards of the University of Utah Health Sciences Center and Primary Children's Medical Center.

Setting and Study Population

We enrolled patients from newborn to 18 years of age who had a diagnosis compatible with neurological impairment and who received their first fundoplication for GERD between January 2005 and July 2006 at Primary Children's Medical Center (PCMC), in Salt Lake City, Utah. PCMC is a 232‐bed children's hospital in the Intermountain West owned and operated by Intermountain Healthcare, Inc., a large vertically integrated health care delivery system that serves as both the primary hospital for Salt Lake County and as the tertiary‐care hospital for 4 additional states (Wyoming, Nevada, Idaho, and Montana).25 Patients who had a previous gastrojejunal feeding tube were excluded as were patients who had a previous fundoplication, as these procedures may have biased their reported quality of life, our main outcome measure.

Patients were included in the study if they had GERD (based on clinical history or testing) that had been refractory to medical management (defined as continued gastroesophageal reflux symptoms despite antireflux medications). GERD was defined using the North American Society for Pediatric Gastroenterology, Hepatology and Nutrition (NASPGHAN) criteria.26 These include: the presence of clinical symptoms and at least 1 abnormal result from an upper gastrointestinal x‐ray series (recognizing that this test is neither sensitive nor specific for reflux), pH probe, upper gastrointestinal endoscopy with biopsy, nuclear medicine scan, or a modified barium swallow. As this was a prospective observational study, physicians were allowed to order testing as their practice dictated. Patients were excluded if they had neurological impairment but lacked objective documentation of GERD using the NASPGHAN recommendations, unless there were obvious clinical indications such as witnessed vomiting and aspiration (N = 3). No patient received a prophylactic fundoplication (fundoplication without documented GERD).

Study Design

This is an ongoing prospective longitudinal study. Patients who had a first fundoplication at PCMC were identified by the surgical service, with weekly lists shared with the research team. Patients were approached by a research assistant during that initial hospitalization to see if they met inclusion criteria for the study using data from the medical records and surgical team when necessary.

Data Variables and Sources

Indications for the fundoplication, performance and results of diagnostic testing for GERD, complications of the fundoplication, and reasons for the neurological impairment were obtained through review of the electronic and paper medical records. Mortality data, subsequent emergency department visits, and admissions to the hospital were obtained using Intermountain Healthcare's electronic data warehouse, which merges clinical, financial, and administrative data including the Utah Vital Statistics database.

Neurological impairment was defined from 2 sources: (1) clinical diagnoses as identified by providers and (2) International Classification of Diseases Codes Modified, version 9 (ICD‐9 CM) identified a priori as indicating neurological impairment.

Instruments and Study Outcomes

Functional status was measured using the WeeFIM. This instrument has been tested and shown to be valid and reliable for children more than 6 months old with neurodevelopmental disabilities including spina bifida and Down syndrome.2732 WeeFIM is a self‐administered parent instrument composed of 18 items and 6 domains (self‐care, sphincter control, transfers, locomotion, communication, and social cognition).33 The WeeFIM allows patients to be stratified into areas of function from severely impaired to normal.

The primary outcome was child quality of life as measured by the Child Health Questionnaire Parental Form 50 (CHQ‐PF50). Caregiver quality of life was measured using the Short‐Form Health Status Survey (SF‐36) and Parenting Stress Index/Short Form (PSI/SF). The CHQ‐PF50 is a self‐administered parent questionnaire of 50 questions that measures 6 domains, including physical function and abilities, pain and discomfort, general health perception, behavior, temperament and moods, and satisfaction with growth and development.34 This instrument has been tested for validity and reliability in children with cerebral palsy.35 The SF‐36 is a widely accepted measure of health status that measures 8 domains of health: physical functioning, role limitations due to physical problems, bodily pain, general health, vitality, social functioning, role limitations due to emotional problems, and mental health. The SF‐36 has been well studied and has been used to measure the effect on a caregiver's quality of life associated with caring for a chronically ill child with significant medical problems.36, 37 Higher scores in each domain of both the CHQPF50 and the SF‐36 reflect better quality of life. Caregiver stress was measured using the PSI/SF (Psychological Assessment Resources Inc, Odessa, FL).38 In the PSI/SF a parent rates the parentchild dyad on 36 items that are summarized in 3 subscales: parental distress, parentchild dysfunctional interaction, and difficult child. A parent scores each item as strongly agree, agree, not sure, disagree, or strongly disagree. The sum of the 3 subscale scores is the total stress score. Higher scores denote a greater degree of stress. The PSI/SF has been validated in several studies for caregivers of children with chronic diseases.3943

The CHQ‐PF50, SF‐36, and PSI/SF questionnaires and the WeeFIM functional status measure were administered to each study patient and caregiver in person at enrollment (baseline) and by mail 1 month after fundoplication. A follow‐up postcard reminder was mailed 1 week after the initial mailing. Second and final mailings were sent to nonresponders 3 and 5 weeks, respectively, after the initial mailing.

Secondary outcomes included rates of complications including failure of the fundoplication. Complications were defined as a subsequent emergency department visit, hospitalization, or death related to the surgery, gastrostomy tube, or aspiration pneumonia. Failures were defined as a second fundoplication or the insertion of a gastrojejunal feeding tube as nonsurgical management of recurrent GERD and/or paraesophageal hernia. Secondary outcomes were followed from the time of the first fundoplication until 1 month after surgery.

Analyses

The differential effect of the fundoplication on the quality of life measures was assessed and quantified through statistical analysis. Because the primary interest was to measure change in baseline characteristics over time, repeated‐measures models were used to compare the group relative to changes in functional status. In particular, changes from baseline values were modeled 1 month after the procedure. The Kenward‐Roger approximation of degrees of freedom was used to compute P values from the overall tests.44 Repeated‐measures models were fit to the data using restricted maximum likelihood estimation. An autoregressive covariance matrix was assumed for the multiple measurements of each individual, thus limiting the number of restrictions forced by the model on the data. The repeated‐measures models used all the available data on participants, including those who dropped out of the study. To obtain the most accurate comparison of the study group, the covariate of functional status at baseline was taken into account in the fitted models. Statistical analyses were performed with SAS statistical software (version 9.13; SAS Institute, Cary NC). Student t tests were performed for comparison of means of the quality‐of‐life domains for the study cohort compared to either the general population or specific groups of patients for comparative purposes.

The study was approved by the institutional review boards of the University of Utah Health Sciences Center and Primary Children's Medical Center.