Observation Protocol

Data collection was conducted by an independent, board‐certified, pediatric physician observer on alternating weeknights immediately after the day‐to‐night evening handoff had taken place. A strict observation protocol was followed. When an eligible question was asked of the participating intern, the physician observer would record the question and the time. The question source, defined as a nurse, parent/patient, or other clinical staff (eg, pharmacist, consultant) was documented, as well as the mode of questioning, defined as face to face, text page, or phone call.

The observer would then note if and when the question was answered. Once the question was answered, the observer would ask the intern if he or she had used the written handoff to provide the answer (yes or no). Our primary outcome was reported use of the written handoff. In addition, the observer directly noted if the intern looked at the written handoff tool at any time when answering a question. The intern was also asked to name any and all additional information resources used, including verbal handoff, senior resident, nursing staff, other clinicians, a patient/parent or other family member, a patient's physical exam, the EMR, the Internet, or his or her own medical or clinical knowledge.

All question and answer information was tracked using a handheld, digital, time device. In addition, the following patient data were recorded for each patient involved in a recorded question: the patient's admitting service, transfer status, and length of stay.

Data Categorization and Analysis

Content of recorded questions were categorized according to whether they involved: (1) medications (including drug allergies or levels), (2) diet or fluids, (3) laboratory values or diagnostic testing/procedures, (4) physical exam findings (eg, a distended abdomen, blood pressure, height/weight), or (5) general care‐plan questions. We also categorized time used for generating an answer as immediate (<5 minutes), delayed (>5 minutes but <1.5 hours), or deferred (any question unanswered during the time of observation).

All data were entered into a database using SPSS 16.0 Data Builder software (SPSS Inc., Chicago, IL), and statistical analyses were performed with PASW 18 (SPSS Inc.) and SAS 9.2 (SAS Institute Inc., Cary, NC) software. Observed questions were summarized according to content categories. We also described trainee and patient characteristics relevant to the questions being studied. To study risk factors for written handoff use, the outcome was dichotomized as reported use of written handoff by the intern as a resource to answer the question asked versus written handoff use was not reported by the intern as a resource to answer the question asked. We did not include observed use of the written handoff in these statistical analyses. To accommodate for patient‐ or provider‐induced correlations among observed questions, we used a generalized estimation equations approach (PROC GENMOD in SAS 9.2) to fit logistic regression models for written handoff use and permitted a nested correlation structure among the questions (ie, questions from the same patient were allowed to be correlated, and patients under the care of the same intern could have intern‐induced correlation). Univariate regression modeling was used to evaluate the effects of question, patient, and intern characteristics. Multivariate logistic regression models were used to identify independent risk factors for written handoff use. Any variable that had a P value 0.1 in univariate regression model was considered as a candidate variable in the multivariate regression model. We then used a backward elimination approach to obtain the final model, which only included variables remaining to be significant at a P<0.05 significance level. Our analysis for verbal handoff use was carried out in a similar fashion.

Questions

A total of 260 questions (ward 1: 136 questions, ward 2: 124 questions) met inclusion criteria and involved 101 different patients, with a median of 2 questions/patient (IQR: 13) and a range of 1 to 14 questions/patient. Overall, interns were asked 2.6 questions/hour (IQR: 1.44.7), with a range of 0 to 7 questions per hour; the great majority of questions (210 [82%]) were posed face to face. Types of questions recorded included medications 28% (73), diet/fluids 15% (39), laboratory or diagnostic/procedural related 22% (57), physical exam or other measurements 8.5% (22), or other general medical or patient care‐plan questions 26.5% (69) (Table 1). Examples of recorded questions are provided in Table 2.

Across the 2 study wards, 48% (49) of patients involved in questions were admitted to a general pediatric service; 27% (27) were admitted to a pediatric specialty service (including the genetics/metabolism, endocrinology, adolescent medicine, pulmonary, or toxicology admitting services); the remaining 25% (25) were admitted to a complex care service (CCS), specifically designed for patients with multisystem genetic, neurological, or congenital disorders (Table 1).[16, 17] Approximately 21% (21) of patients had been transferred to the floor from a critical care unit (Table 1).

Answers

Of the 260 recorded questions, 90% (233) had documented answers. For the 10% (27) of questions with undocumented answers, 21 were observed to be verbally deferred by the intern to the day team or another care provider (ie, other physician or nurse), and almost half (42.9% [9]) involved general care‐plan questions; the remainder involved medication (4), diet (2), diagnostic testing (5), or vital sign (1) questions. An additional 6 questions went unanswered during the observation period, and it is unknown if or when they were answered.

Of the answered questions, 90% (209) of answers were provided by trainees within 5 minutes and 9% (21) within 1.5 hours. In all, interns reported using 1 information resource to provide answers for 61% (142) of questions, at least 2 resources for 33% (76) questions, and 3 resources for 6% (15) questions.

Across both study wards, interns reported using information provided in written or verbal handoffs to answer 32.6% of questions. Interns reported using the written handoff, either alone or in combination with other information resources, to provide answers for 7.3% (17) of questions; verbal handoff, either alone or in combination with another resource (excluding written handoff), was reported as a resource for 25.3% (59) of questions. Of note, interns were directly observed to look at the written handoff when answering 21% (49) of questions.

A variety of other resources, including general medical/clinical knowledge, the EMR, and parents or other resources, were used to answer the remaining 67.4% (157) of questions. Intern general medical knowledge (ie, reports of simply knowing the answer to the question in their head[s]) was used to provide answers for 53.2% (124) of questions asked.

Unadjusted univariate regression analyses assessing predictors of written and verbal handoff use are shown in Figure 1. Multivariate logistic regression analyses showed that both dietary questions (odds ratio [OR]: 3.64, 95% confidence interval [CI]: 1.518.76; P=0.004) and interns' consecutive call night (OR: 0.29, 95% CI: 0.090.93; P=0.04) remained significant predictors of written handoff use. After adjusting for risk factors identified above, no differences in written handoff use were seen between the 2 wards.

Figure 1

Multivariate logistic regression for predictors of the verbal handoff use showed that questions regarding patients with longer lengths of stay (OR: 1.97, 95% CI: 1.023.8; P=0.04), those regarding general care plans (OR: 2.07, 95% CI: 1.133.78; P=0.02), as well as those asked by clinical staff (OR: 1.95, 95 CI: 1.043.66; P=0.04), remained significant predictors of reported verbal handoff use.

Observation Protocol

Data collection was conducted by an independent, board‐certified, pediatric physician observer on alternating weeknights immediately after the day‐to‐night evening handoff had taken place. A strict observation protocol was followed. When an eligible question was asked of the participating intern, the physician observer would record the question and the time. The question source, defined as a nurse, parent/patient, or other clinical staff (eg, pharmacist, consultant) was documented, as well as the mode of questioning, defined as face to face, text page, or phone call.

The observer would then note if and when the question was answered. Once the question was answered, the observer would ask the intern if he or she had used the written handoff to provide the answer (yes or no). Our primary outcome was reported use of the written handoff. In addition, the observer directly noted if the intern looked at the written handoff tool at any time when answering a question. The intern was also asked to name any and all additional information resources used, including verbal handoff, senior resident, nursing staff, other clinicians, a patient/parent or other family member, a patient's physical exam, the EMR, the Internet, or his or her own medical or clinical knowledge.

All question and answer information was tracked using a handheld, digital, time device. In addition, the following patient data were recorded for each patient involved in a recorded question: the patient's admitting service, transfer status, and length of stay.

Data Categorization and Analysis

Content of recorded questions were categorized according to whether they involved: (1) medications (including drug allergies or levels), (2) diet or fluids, (3) laboratory values or diagnostic testing/procedures, (4) physical exam findings (eg, a distended abdomen, blood pressure, height/weight), or (5) general care‐plan questions. We also categorized time used for generating an answer as immediate (<5 minutes), delayed (>5 minutes but <1.5 hours), or deferred (any question unanswered during the time of observation).

All data were entered into a database using SPSS 16.0 Data Builder software (SPSS Inc., Chicago, IL), and statistical analyses were performed with PASW 18 (SPSS Inc.) and SAS 9.2 (SAS Institute Inc., Cary, NC) software. Observed questions were summarized according to content categories. We also described trainee and patient characteristics relevant to the questions being studied. To study risk factors for written handoff use, the outcome was dichotomized as reported use of written handoff by the intern as a resource to answer the question asked versus written handoff use was not reported by the intern as a resource to answer the question asked. We did not include observed use of the written handoff in these statistical analyses. To accommodate for patient‐ or provider‐induced correlations among observed questions, we used a generalized estimation equations approach (PROC GENMOD in SAS 9.2) to fit logistic regression models for written handoff use and permitted a nested correlation structure among the questions (ie, questions from the same patient were allowed to be correlated, and patients under the care of the same intern could have intern‐induced correlation). Univariate regression modeling was used to evaluate the effects of question, patient, and intern characteristics. Multivariate logistic regression models were used to identify independent risk factors for written handoff use. Any variable that had a P value 0.1 in univariate regression model was considered as a candidate variable in the multivariate regression model. We then used a backward elimination approach to obtain the final model, which only included variables remaining to be significant at a P<0.05 significance level. Our analysis for verbal handoff use was carried out in a similar fashion.

Questions

A total of 260 questions (ward 1: 136 questions, ward 2: 124 questions) met inclusion criteria and involved 101 different patients, with a median of 2 questions/patient (IQR: 13) and a range of 1 to 14 questions/patient. Overall, interns were asked 2.6 questions/hour (IQR: 1.44.7), with a range of 0 to 7 questions per hour; the great majority of questions (210 [82%]) were posed face to face. Types of questions recorded included medications 28% (73), diet/fluids 15% (39), laboratory or diagnostic/procedural related 22% (57), physical exam or other measurements 8.5% (22), or other general medical or patient care‐plan questions 26.5% (69) (Table 1). Examples of recorded questions are provided in Table 2.

Across the 2 study wards, 48% (49) of patients involved in questions were admitted to a general pediatric service; 27% (27) were admitted to a pediatric specialty service (including the genetics/metabolism, endocrinology, adolescent medicine, pulmonary, or toxicology admitting services); the remaining 25% (25) were admitted to a complex care service (CCS), specifically designed for patients with multisystem genetic, neurological, or congenital disorders (Table 1).[16, 17] Approximately 21% (21) of patients had been transferred to the floor from a critical care unit (Table 1).

Answers

Of the 260 recorded questions, 90% (233) had documented answers. For the 10% (27) of questions with undocumented answers, 21 were observed to be verbally deferred by the intern to the day team or another care provider (ie, other physician or nurse), and almost half (42.9% [9]) involved general care‐plan questions; the remainder involved medication (4), diet (2), diagnostic testing (5), or vital sign (1) questions. An additional 6 questions went unanswered during the observation period, and it is unknown if or when they were answered.

Of the answered questions, 90% (209) of answers were provided by trainees within 5 minutes and 9% (21) within 1.5 hours. In all, interns reported using 1 information resource to provide answers for 61% (142) of questions, at least 2 resources for 33% (76) questions, and 3 resources for 6% (15) questions.

Across both study wards, interns reported using information provided in written or verbal handoffs to answer 32.6% of questions. Interns reported using the written handoff, either alone or in combination with other information resources, to provide answers for 7.3% (17) of questions; verbal handoff, either alone or in combination with another resource (excluding written handoff), was reported as a resource for 25.3% (59) of questions. Of note, interns were directly observed to look at the written handoff when answering 21% (49) of questions.

A variety of other resources, including general medical/clinical knowledge, the EMR, and parents or other resources, were used to answer the remaining 67.4% (157) of questions. Intern general medical knowledge (ie, reports of simply knowing the answer to the question in their head[s]) was used to provide answers for 53.2% (124) of questions asked.

Unadjusted univariate regression analyses assessing predictors of written and verbal handoff use are shown in Figure 1. Multivariate logistic regression analyses showed that both dietary questions (odds ratio [OR]: 3.64, 95% confidence interval [CI]: 1.518.76; P=0.004) and interns' consecutive call night (OR: 0.29, 95% CI: 0.090.93; P=0.04) remained significant predictors of written handoff use. After adjusting for risk factors identified above, no differences in written handoff use were seen between the 2 wards.

Figure 1

Multivariate logistic regression for predictors of the verbal handoff use showed that questions regarding patients with longer lengths of stay (OR: 1.97, 95% CI: 1.023.8; P=0.04), those regarding general care plans (OR: 2.07, 95% CI: 1.133.78; P=0.02), as well as those asked by clinical staff (OR: 1.95, 95 CI: 1.043.66; P=0.04), remained significant predictors of reported verbal handoff use.

Observation Protocol

Data collection was conducted by an independent, board‐certified, pediatric physician observer on alternating weeknights immediately after the day‐to‐night evening handoff had taken place. A strict observation protocol was followed. When an eligible question was asked of the participating intern, the physician observer would record the question and the time. The question source, defined as a nurse, parent/patient, or other clinical staff (eg, pharmacist, consultant) was documented, as well as the mode of questioning, defined as face to face, text page, or phone call.

The observer would then note if and when the question was answered. Once the question was answered, the observer would ask the intern if he or she had used the written handoff to provide the answer (yes or no). Our primary outcome was reported use of the written handoff. In addition, the observer directly noted if the intern looked at the written handoff tool at any time when answering a question. The intern was also asked to name any and all additional information resources used, including verbal handoff, senior resident, nursing staff, other clinicians, a patient/parent or other family member, a patient's physical exam, the EMR, the Internet, or his or her own medical or clinical knowledge.

All question and answer information was tracked using a handheld, digital, time device. In addition, the following patient data were recorded for each patient involved in a recorded question: the patient's admitting service, transfer status, and length of stay.

Data Categorization and Analysis

Content of recorded questions were categorized according to whether they involved: (1) medications (including drug allergies or levels), (2) diet or fluids, (3) laboratory values or diagnostic testing/procedures, (4) physical exam findings (eg, a distended abdomen, blood pressure, height/weight), or (5) general care‐plan questions. We also categorized time used for generating an answer as immediate (<5 minutes), delayed (>5 minutes but <1.5 hours), or deferred (any question unanswered during the time of observation).

All data were entered into a database using SPSS 16.0 Data Builder software (SPSS Inc., Chicago, IL), and statistical analyses were performed with PASW 18 (SPSS Inc.) and SAS 9.2 (SAS Institute Inc., Cary, NC) software. Observed questions were summarized according to content categories. We also described trainee and patient characteristics relevant to the questions being studied. To study risk factors for written handoff use, the outcome was dichotomized as reported use of written handoff by the intern as a resource to answer the question asked versus written handoff use was not reported by the intern as a resource to answer the question asked. We did not include observed use of the written handoff in these statistical analyses. To accommodate for patient‐ or provider‐induced correlations among observed questions, we used a generalized estimation equations approach (PROC GENMOD in SAS 9.2) to fit logistic regression models for written handoff use and permitted a nested correlation structure among the questions (ie, questions from the same patient were allowed to be correlated, and patients under the care of the same intern could have intern‐induced correlation). Univariate regression modeling was used to evaluate the effects of question, patient, and intern characteristics. Multivariate logistic regression models were used to identify independent risk factors for written handoff use. Any variable that had a P value 0.1 in univariate regression model was considered as a candidate variable in the multivariate regression model. We then used a backward elimination approach to obtain the final model, which only included variables remaining to be significant at a P<0.05 significance level. Our analysis for verbal handoff use was carried out in a similar fashion.

Questions

A total of 260 questions (ward 1: 136 questions, ward 2: 124 questions) met inclusion criteria and involved 101 different patients, with a median of 2 questions/patient (IQR: 13) and a range of 1 to 14 questions/patient. Overall, interns were asked 2.6 questions/hour (IQR: 1.44.7), with a range of 0 to 7 questions per hour; the great majority of questions (210 [82%]) were posed face to face. Types of questions recorded included medications 28% (73), diet/fluids 15% (39), laboratory or diagnostic/procedural related 22% (57), physical exam or other measurements 8.5% (22), or other general medical or patient care‐plan questions 26.5% (69) (Table 1). Examples of recorded questions are provided in Table 2.

Across the 2 study wards, 48% (49) of patients involved in questions were admitted to a general pediatric service; 27% (27) were admitted to a pediatric specialty service (including the genetics/metabolism, endocrinology, adolescent medicine, pulmonary, or toxicology admitting services); the remaining 25% (25) were admitted to a complex care service (CCS), specifically designed for patients with multisystem genetic, neurological, or congenital disorders (Table 1).[16, 17] Approximately 21% (21) of patients had been transferred to the floor from a critical care unit (Table 1).

Answers

Of the 260 recorded questions, 90% (233) had documented answers. For the 10% (27) of questions with undocumented answers, 21 were observed to be verbally deferred by the intern to the day team or another care provider (ie, other physician or nurse), and almost half (42.9% [9]) involved general care‐plan questions; the remainder involved medication (4), diet (2), diagnostic testing (5), or vital sign (1) questions. An additional 6 questions went unanswered during the observation period, and it is unknown if or when they were answered.

Of the answered questions, 90% (209) of answers were provided by trainees within 5 minutes and 9% (21) within 1.5 hours. In all, interns reported using 1 information resource to provide answers for 61% (142) of questions, at least 2 resources for 33% (76) questions, and 3 resources for 6% (15) questions.

Across both study wards, interns reported using information provided in written or verbal handoffs to answer 32.6% of questions. Interns reported using the written handoff, either alone or in combination with other information resources, to provide answers for 7.3% (17) of questions; verbal handoff, either alone or in combination with another resource (excluding written handoff), was reported as a resource for 25.3% (59) of questions. Of note, interns were directly observed to look at the written handoff when answering 21% (49) of questions.

A variety of other resources, including general medical/clinical knowledge, the EMR, and parents or other resources, were used to answer the remaining 67.4% (157) of questions. Intern general medical knowledge (ie, reports of simply knowing the answer to the question in their head[s]) was used to provide answers for 53.2% (124) of questions asked.

Unadjusted univariate regression analyses assessing predictors of written and verbal handoff use are shown in Figure 1. Multivariate logistic regression analyses showed that both dietary questions (odds ratio [OR]: 3.64, 95% confidence interval [CI]: 1.518.76; P=0.004) and interns' consecutive call night (OR: 0.29, 95% CI: 0.090.93; P=0.04) remained significant predictors of written handoff use. After adjusting for risk factors identified above, no differences in written handoff use were seen between the 2 wards.

Figure 1

Multivariate logistic regression for predictors of the verbal handoff use showed that questions regarding patients with longer lengths of stay (OR: 1.97, 95% CI: 1.023.8; P=0.04), those regarding general care plans (OR: 2.07, 95% CI: 1.133.78; P=0.02), as well as those asked by clinical staff (OR: 1.95, 95 CI: 1.043.66; P=0.04), remained significant predictors of reported verbal handoff use.

Hospital Eligibility for VBP

All acute care hospitals in the United States (excluding Maryland) that are not psychiatric hospitals, rehabilitation hospitals, long‐term care facilities, children's hospitals, or cancer hospitals are eligible to participate in VBP in FY 2013 (full eligibility criteria is outlined in Table 1). For FY 2013, CMS chose to incentivize measures in just 2 care domains: (1) clinical processes of care and (2) patient experience of care. To be eligible for VBP in FY 2013, a hospital must report at least 10 cases each in at least 4 of 12 measures included in the clinical processes of care domain (Table 2), and/or must have at least 100 completed Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS). Designed and validated by CMS, the HCAHPS survey provides hospitals with a standardized instrument for gathering information about patient satisfaction with, and perspectives on, their hospital care.[12] HCAHPS will be used to assess 8 patient experience of care measures (Table 3).

Participation in the program is mandatory for eligible hospitals, and CMS estimates that more than 3000 facilities across the United States will participate in FY 2013. Roughly $850 million dollars in VBP incentives will be paid out to these participating hospitals in FY 2013. The program is being financed through a 1% across‐the‐board reduction in FY 2013 diagnosis‐related group (DRG)‐based inpatient payments to participating hospitals. On December 20, 2012, CMS publically announced FY 2013 VBP incentives for all participating hospitals. Each hospital's incentive is retroactive and based on its performance between July 1, 2011 and March 31, 2012.

All data used for calculating VBP incentives is reported to CMS through its Hospital Inpatient Quality Reporting (Hospital IQR) Program, a national program instituted in 2003 that rewards hospitals for reporting designated quality measures. As of 2007, approximately 95% of eligible US hospitals were using the Hospital IQR program.[1] Measures evaluated via chart abstracts and surveys reflect a hospital's performance for its entire patient population, whereas measures assessed with claims data reflect hospital performance only for Medicare patients.

Evaluation of Hospitals

In FY 2013, hospital VBP incentive payments will be based entirely on performance in 2 domains: (1) clinical processes of care (weighted 70%) and (2) patient experience of care (weighted 30%). For each domain, CMS will evaluate each hospital's improvement over time as well as achievement compared to other hospitals in the VBP program. By assessing and rewarding both achievement and improvement, CMS will ensure that lower‐performing hospitals will still be rewarded for making substantial improvements in quality. To evaluate the first metricimprovement over timeCMS will compare a hospital's performance during a given reporting period with its baseline performance 2 years prior to this block of time. A hospital receives improvement points for improving its performance over time. To assess the second metricachievement compared to other hospitals in the VBP programCMS will compare each hospital's performance during a reporting period with the baseline performance (eg, performance 2 years prior to reporting period) of all other hospitals in the VBP program. A hospital is awarded achievement points if its performance exceeds the 50th percentile of all hospitals during the baseline performance period. Improvement scores range from 0 to 9, whereas achievement scores range from 0 to 10. The greater of a hospital's improvement and achievement scores on each VBP measure are used to calculate each hospital's total earned clinical care domain score and total earned HCAHPS base score. Hospitals that lack baseline performance data, which is required to assess improvement, will be evaluated solely on the basis of achievement points.[1] The total earned clinical care domain score is multiplied by 70% to reach the clinical care domain's contribution to a hospital's total performance score.

Each hospital's total patient experience domain, or HCAHPS performance, score consists of 2 components: a total earned HCAHPS base score as described above and a consistency score. The consistency score evaluates the reliability of a hospital's performance across all 8 patient experience of care measures (Table 3). If a hospital is above the 50th percentile of all hospital scores during the baseline period on all 8 measures, then it receives 100% of its consistency points. If a hospital is at the 0 percentile for a given measure, then it receives 0 consistency points for all measures. This provision promotes consistency by harshly penalizing hospitals with extremely poor performance on any 1 specific measure. If 1 or more measures are between the 0 and 50th percentiles, then it will receive a consistency score that takes into account how many measures were below the 50th percentile and their distance from this threshold. Each hospital's total HCAHPS performance score (the sum of total earned HCAHPS base points and consistency points) is then multiplied by 30% to arrive at the patient experience of care domain's contribution to a hospital's total performance score.

Importantly, CMS excluded from its VBP initiative 10 clinical process measures reported in the Hospital IQR Program because they are topped out; that is, almost all hospitals already perform them at very high rates (Table 4). Examples of these topped out process measures include administration of aspirin to all patients with AMI on arrival at the hospital; counseling of patients with AMI, CHF, and pneumonia about smoking cessation; and prescribing angiotensin‐converting enzyme inhibitors or angiotensin receptor blockers to patients with CHF and left ventricular dysfunction.[1]

Calculation of VBP Incentives and Public Reporting

A hospital's total performance score for FY 2013 is equal to the sum of 70% of its clinical care domain score and 30% of its total HCAHPS performance score. This total performance score is entered into a linear mathematical formula to calculate each hospital's incentive payment. CMS projects that VBP will lead to a net increase in Medicare payments for one‐half of hospitals and a net decrease in payments for the other half of participating facilities.[1]

In December 2012, CMS publicly disclosed information about the initial performance of each hospital in the VBP program. Reported information included: (1) hospital performance for each applicable performance measure, (2) hospital performance by disease condition or procedure, and (3) hospital's total performance score. Initial analyses of this performance data revealed that 1557 hospitals will receive bonus payments under VBP in FY 2013, whereas 1427 hospitals will lose money under this program. Treasure Valley Hospital, a 10‐bed physician‐owned hospital in Boise, Idaho, will receive a 0.83% increase in Medicare payments, the largest payment increase under VBP in 2013. Conversely, Auburn Community Hospital in upstate New York, will suffer the most severe payment reduction: 0.9% per Medicare admission. The penalty will cost Auburn Hospital about $100,000, which is slightly more than 0.1% of its yearly $85 million operating budget.[13] For almost two‐thirds of participating hospitals, FY 2013 Medicare payments will change by <0.25%.[13] Additional information about VBP payments for FY 2013, including the number of hospitals who received VBP incentives and the size and range of these payments, is now accessible to the public through CMS' Hospital Compare Web site (http://www.hospitalcompare.hhs.gov).

Rigorous and Continuous Evaluation of VBP Programs

The main premise of using VBP to incentivize hospitals to deliver high‐quality cost‐effective care is that the process measures used to determine hospital quality do impact patient outcomes. However, it is already well established that improvements in measures of process quality are not always associated with improvements in patient outcomes.[14, 15, 16] Moreover, incentivizing specific process measures encourages hospitals to shift resources away from other aspects of care delivery, which may have ambiguous, or even deleterious, effects on patient outcomes. Although incentives ideally push hospitals to shift resources away from low‐quality care toward high‐quality care, in practice this is not always the case. Hospital resources may instead be drawn away from areas that are not yet incented by VBP, but for which improvements in quality of care are desperately needed. The same empirical focus behind using VBP to incentivize hospitals to improve patient outcomes efficiently should be used to evaluate whether VBP is continually meeting its stated goals: reducing overall patient morbidity and mortality and improving patient satisfaction at ideally lower cost. The experience of the US education system with public policies designed to improve student testing performance may serve as a cautionary example here. Such policies, which provide financial rewards to schools whose students perform well on standardized tests, can indeed raise testing performance. However, these policies also lead educators to teach to the test, and to neglect important topics that are not tested on standardized exams.[17]

Prioritization of Process Measures

As payment incentives for VBP currently stand, process measures are weighted equally regardless of the clinical benefits they generate and the resources required to achieve improvements in process quality. For instance, 2 process measures, continuing home ‐blocker medications for patients with coronary artery disease undergoing surgery and early percutaneous coronary intervention for patients with AMI, may be weighted equally as process measures although both their clinical benefits and the costs of implementation are very different. Some hospitals responding to VBP incentives may choose to invest in areas where their ability to earn VBP incentive payments is high and the costs of improvement are low, although those areas may not be where interventions are most needed because clinical outcomes could be most improved. Recognizing that process measures have heterogeneous benefits and costs of implementation is important when prioritizing their reimbursement in VBP.

Measuring Improvements in Hospital Quality

Tying hospital financial compensation to hospital quality implies that measures of hospital quality should be robust. To incentivize hospitals to improve quality not only relative to other hospitals but to themselves in the past, the VBP program has established a baseline performance for each hospital. Each hospital is compared to its baseline performance in subsequent evaluation periods. Thus, properly measuring a hospital's baseline performance is important. During a given baseline period, some hospitals may have better or worse outcomes than their steady state due to random variation alone. Some hospitals deemed to have a low baseline will experience improvements in quality that are not related to active efforts to improve quality but through chance alone. Similarly, some hospitals deemed to have a high baseline will experience reductions in quality through chance. Of course, neither of these changes should be subject to differences in reimbursement because they do not reflect actual organizational changes made by the hospitals. The VBP program has made significant efforts to address this issue by requiring participating hospitals to have a large enough sample of cases such that estimated rates of process quality adherence meet a reliability threshold (ie, are likely to be consistent over time rather than vary substantially through chance alone). However, not all process measures exhibit high reliability, particularly those for which adverse events are rare (eg, foreign objects retained after surgery, air embolisms, and blood incompatibility). Ultimately, CMS's decision to balance the need for statistically reliable data with the goal of including as many hospitals as possible in the VBP program will require ongoing reevaluation of this issue.

Choosing Hospital Comparators Appropriately

In the current VBP program, hospitals will be evaluated in part by how they compare to hospitals nationally. However, studies of regional variation in healthcare have demonstrated large variations in practice patterns across the United States,[18, 19, 20] raising the question of whether hospitals should, at least initially, be compared to hospitals in the same geographic area. Although the ultimate goal of VBP should be to hold hospitals to a national standard, local practice patterns are not easily modified within 1‐ to 2‐year timeframes. Initially comparing hospitals to a national rather than local standard may unfairly penalize hospitals that are relative underperformers nationally but overperformers regionally. Although CMS's policy to reward improvement within hospitals over time mitigates issues arising from a cross‐sectional comparison of hospitals, the issue still remains if many hospitals within a region not only underperform relative to other hospitals nationally but also fail to demonstrate improvement. More broadly, this issue extends to differences across hospitals in factors that impact their ability to meet VBP goals. These factors may include, for example, hospital size, profitability, patient case and insurance mix, and presence of an electronic medical record. Comparing hospitals with vastly different abilities to achieve VBP goals and improve quickly may amount to inequitable policy.

Continual Evaluation of Topped‐Out Measures

Process measures that are met at high rates at nearly all hospitals are not used in evaluations by CMS for VBP. An assumption underlying CMS' decision to not reward hospitals for achieving these topped‐out measures is that once physicians and hospitals make cognitive and system‐level improvements that improve process quality, these gains will persist after the incentive is removed. Thus, CMS hopes and anticipates that although performance incentives will make it easier for well‐meaning physicians to learn to do the right thing, doctors will continue to do the right things for patients after these incentives are removed.[21, 22] Although this assumption may generally be accurate, it is important to continue to evaluate whether measures that are currently topped out continue to remain adequately performed, because rewarding new quality measures will necessarily lead hospitals to reallocate resources away from other clinical activities. Although we hope that the continued public reporting of topped‐out measures will prevent declines in performance on these measures, policy makers and clinicians should be aware that the lack of financial incentives for topped‐out measures may result in declines in quality. To this point, an analysis of 35 Kaiser Permanente facilities from 1997 to 2007 demonstrated that the removal of financial incentives for diabetic retinopathy and cervical cancer screening was associated with subsequent declines in performance of 3% and 1.6% per year, respectively.[23]

Will VBP Incentives Be Large Enough to Change Practice Patterns?

The VBP Program's ability to influence change depends, at least in part, on how the incentives offered under this program compare to the magnitude of the investments that hospitals must make to achieve a given reward. In general, larger incentives are necessary to motivate more significant changes in behavior or to influence organizations to invest the resources needed to achieve change. The incentives offered under VBP in FY 2013 are quite modest. Almost two‐thirds of participating hospitals will see their FY 2013 Medicare revenues change by <0.25%, roughly $125,000 at most.[13, 24] Although these incentives may motivate hospitals that can improve performance and achievement with very modest investments, they may have little impact on organizations that need to make significant upfront investments in care processes to achieve sustainable improvements in care quality. As CMS increases the size of VBP incentives over the next 2 to 4 years, it will also hold hospitals accountable for a broader and increasingly complex set of outcomes. Improving these outcomes may require investments in areas such as information technology and process improvement that far surpass the VBP incentive reward.

Moreover, prior research suggests that financial incentives like those available under VBP may contribute only slightly to performance improvements when public reporting already exists. For example, in a 2‐year study of 613 US hospitals implementing pay‐for‐performance plus public reporting or public reporting only, pay for performance plus public reporting was associated with only a 2.6% to 4.1% increase in a composite measure of quality when compared to hospitals with public reporting only.[9] Similarly, a study of 54 hospitals participating in the CMS pay for performance pilot initiative found no significant improvement in quality of care or outcomes for AMI when compared to 446 control hospitals.[25] A long‐term analysis of pay for performance in the Medicare Premier Hospital Quality Incentive Demonstration found that participation in the program had no discernible effect on 30‐day mortality rates.[10] Finally, a study of physician medical groups contracting with a large network healthcare maintenance organization found that the implementation of pay for performance did not result in major before and after improvements in clinical quality compared to a control group of medical groups.[26]

Hospital Eligibility for VBP

All acute care hospitals in the United States (excluding Maryland) that are not psychiatric hospitals, rehabilitation hospitals, long‐term care facilities, children's hospitals, or cancer hospitals are eligible to participate in VBP in FY 2013 (full eligibility criteria is outlined in Table 1). For FY 2013, CMS chose to incentivize measures in just 2 care domains: (1) clinical processes of care and (2) patient experience of care. To be eligible for VBP in FY 2013, a hospital must report at least 10 cases each in at least 4 of 12 measures included in the clinical processes of care domain (Table 2), and/or must have at least 100 completed Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS). Designed and validated by CMS, the HCAHPS survey provides hospitals with a standardized instrument for gathering information about patient satisfaction with, and perspectives on, their hospital care.[12] HCAHPS will be used to assess 8 patient experience of care measures (Table 3).

Participation in the program is mandatory for eligible hospitals, and CMS estimates that more than 3000 facilities across the United States will participate in FY 2013. Roughly $850 million dollars in VBP incentives will be paid out to these participating hospitals in FY 2013. The program is being financed through a 1% across‐the‐board reduction in FY 2013 diagnosis‐related group (DRG)‐based inpatient payments to participating hospitals. On December 20, 2012, CMS publically announced FY 2013 VBP incentives for all participating hospitals. Each hospital's incentive is retroactive and based on its performance between July 1, 2011 and March 31, 2012.

All data used for calculating VBP incentives is reported to CMS through its Hospital Inpatient Quality Reporting (Hospital IQR) Program, a national program instituted in 2003 that rewards hospitals for reporting designated quality measures. As of 2007, approximately 95% of eligible US hospitals were using the Hospital IQR program.[1] Measures evaluated via chart abstracts and surveys reflect a hospital's performance for its entire patient population, whereas measures assessed with claims data reflect hospital performance only for Medicare patients.

Evaluation of Hospitals

In FY 2013, hospital VBP incentive payments will be based entirely on performance in 2 domains: (1) clinical processes of care (weighted 70%) and (2) patient experience of care (weighted 30%). For each domain, CMS will evaluate each hospital's improvement over time as well as achievement compared to other hospitals in the VBP program. By assessing and rewarding both achievement and improvement, CMS will ensure that lower‐performing hospitals will still be rewarded for making substantial improvements in quality. To evaluate the first metricimprovement over timeCMS will compare a hospital's performance during a given reporting period with its baseline performance 2 years prior to this block of time. A hospital receives improvement points for improving its performance over time. To assess the second metricachievement compared to other hospitals in the VBP programCMS will compare each hospital's performance during a reporting period with the baseline performance (eg, performance 2 years prior to reporting period) of all other hospitals in the VBP program. A hospital is awarded achievement points if its performance exceeds the 50th percentile of all hospitals during the baseline performance period. Improvement scores range from 0 to 9, whereas achievement scores range from 0 to 10. The greater of a hospital's improvement and achievement scores on each VBP measure are used to calculate each hospital's total earned clinical care domain score and total earned HCAHPS base score. Hospitals that lack baseline performance data, which is required to assess improvement, will be evaluated solely on the basis of achievement points.[1] The total earned clinical care domain score is multiplied by 70% to reach the clinical care domain's contribution to a hospital's total performance score.

Each hospital's total patient experience domain, or HCAHPS performance, score consists of 2 components: a total earned HCAHPS base score as described above and a consistency score. The consistency score evaluates the reliability of a hospital's performance across all 8 patient experience of care measures (Table 3). If a hospital is above the 50th percentile of all hospital scores during the baseline period on all 8 measures, then it receives 100% of its consistency points. If a hospital is at the 0 percentile for a given measure, then it receives 0 consistency points for all measures. This provision promotes consistency by harshly penalizing hospitals with extremely poor performance on any 1 specific measure. If 1 or more measures are between the 0 and 50th percentiles, then it will receive a consistency score that takes into account how many measures were below the 50th percentile and their distance from this threshold. Each hospital's total HCAHPS performance score (the sum of total earned HCAHPS base points and consistency points) is then multiplied by 30% to arrive at the patient experience of care domain's contribution to a hospital's total performance score.

Importantly, CMS excluded from its VBP initiative 10 clinical process measures reported in the Hospital IQR Program because they are topped out; that is, almost all hospitals already perform them at very high rates (Table 4). Examples of these topped out process measures include administration of aspirin to all patients with AMI on arrival at the hospital; counseling of patients with AMI, CHF, and pneumonia about smoking cessation; and prescribing angiotensin‐converting enzyme inhibitors or angiotensin receptor blockers to patients with CHF and left ventricular dysfunction.[1]

Calculation of VBP Incentives and Public Reporting

A hospital's total performance score for FY 2013 is equal to the sum of 70% of its clinical care domain score and 30% of its total HCAHPS performance score. This total performance score is entered into a linear mathematical formula to calculate each hospital's incentive payment. CMS projects that VBP will lead to a net increase in Medicare payments for one‐half of hospitals and a net decrease in payments for the other half of participating facilities.[1]

In December 2012, CMS publicly disclosed information about the initial performance of each hospital in the VBP program. Reported information included: (1) hospital performance for each applicable performance measure, (2) hospital performance by disease condition or procedure, and (3) hospital's total performance score. Initial analyses of this performance data revealed that 1557 hospitals will receive bonus payments under VBP in FY 2013, whereas 1427 hospitals will lose money under this program. Treasure Valley Hospital, a 10‐bed physician‐owned hospital in Boise, Idaho, will receive a 0.83% increase in Medicare payments, the largest payment increase under VBP in 2013. Conversely, Auburn Community Hospital in upstate New York, will suffer the most severe payment reduction: 0.9% per Medicare admission. The penalty will cost Auburn Hospital about $100,000, which is slightly more than 0.1% of its yearly $85 million operating budget.[13] For almost two‐thirds of participating hospitals, FY 2013 Medicare payments will change by <0.25%.[13] Additional information about VBP payments for FY 2013, including the number of hospitals who received VBP incentives and the size and range of these payments, is now accessible to the public through CMS' Hospital Compare Web site (http://www.hospitalcompare.hhs.gov).

Rigorous and Continuous Evaluation of VBP Programs

The main premise of using VBP to incentivize hospitals to deliver high‐quality cost‐effective care is that the process measures used to determine hospital quality do impact patient outcomes. However, it is already well established that improvements in measures of process quality are not always associated with improvements in patient outcomes.[14, 15, 16] Moreover, incentivizing specific process measures encourages hospitals to shift resources away from other aspects of care delivery, which may have ambiguous, or even deleterious, effects on patient outcomes. Although incentives ideally push hospitals to shift resources away from low‐quality care toward high‐quality care, in practice this is not always the case. Hospital resources may instead be drawn away from areas that are not yet incented by VBP, but for which improvements in quality of care are desperately needed. The same empirical focus behind using VBP to incentivize hospitals to improve patient outcomes efficiently should be used to evaluate whether VBP is continually meeting its stated goals: reducing overall patient morbidity and mortality and improving patient satisfaction at ideally lower cost. The experience of the US education system with public policies designed to improve student testing performance may serve as a cautionary example here. Such policies, which provide financial rewards to schools whose students perform well on standardized tests, can indeed raise testing performance. However, these policies also lead educators to teach to the test, and to neglect important topics that are not tested on standardized exams.[17]

Prioritization of Process Measures

As payment incentives for VBP currently stand, process measures are weighted equally regardless of the clinical benefits they generate and the resources required to achieve improvements in process quality. For instance, 2 process measures, continuing home ‐blocker medications for patients with coronary artery disease undergoing surgery and early percutaneous coronary intervention for patients with AMI, may be weighted equally as process measures although both their clinical benefits and the costs of implementation are very different. Some hospitals responding to VBP incentives may choose to invest in areas where their ability to earn VBP incentive payments is high and the costs of improvement are low, although those areas may not be where interventions are most needed because clinical outcomes could be most improved. Recognizing that process measures have heterogeneous benefits and costs of implementation is important when prioritizing their reimbursement in VBP.

Measuring Improvements in Hospital Quality

Tying hospital financial compensation to hospital quality implies that measures of hospital quality should be robust. To incentivize hospitals to improve quality not only relative to other hospitals but to themselves in the past, the VBP program has established a baseline performance for each hospital. Each hospital is compared to its baseline performance in subsequent evaluation periods. Thus, properly measuring a hospital's baseline performance is important. During a given baseline period, some hospitals may have better or worse outcomes than their steady state due to random variation alone. Some hospitals deemed to have a low baseline will experience improvements in quality that are not related to active efforts to improve quality but through chance alone. Similarly, some hospitals deemed to have a high baseline will experience reductions in quality through chance. Of course, neither of these changes should be subject to differences in reimbursement because they do not reflect actual organizational changes made by the hospitals. The VBP program has made significant efforts to address this issue by requiring participating hospitals to have a large enough sample of cases such that estimated rates of process quality adherence meet a reliability threshold (ie, are likely to be consistent over time rather than vary substantially through chance alone). However, not all process measures exhibit high reliability, particularly those for which adverse events are rare (eg, foreign objects retained after surgery, air embolisms, and blood incompatibility). Ultimately, CMS's decision to balance the need for statistically reliable data with the goal of including as many hospitals as possible in the VBP program will require ongoing reevaluation of this issue.

Choosing Hospital Comparators Appropriately

In the current VBP program, hospitals will be evaluated in part by how they compare to hospitals nationally. However, studies of regional variation in healthcare have demonstrated large variations in practice patterns across the United States,[18, 19, 20] raising the question of whether hospitals should, at least initially, be compared to hospitals in the same geographic area. Although the ultimate goal of VBP should be to hold hospitals to a national standard, local practice patterns are not easily modified within 1‐ to 2‐year timeframes. Initially comparing hospitals to a national rather than local standard may unfairly penalize hospitals that are relative underperformers nationally but overperformers regionally. Although CMS's policy to reward improvement within hospitals over time mitigates issues arising from a cross‐sectional comparison of hospitals, the issue still remains if many hospitals within a region not only underperform relative to other hospitals nationally but also fail to demonstrate improvement. More broadly, this issue extends to differences across hospitals in factors that impact their ability to meet VBP goals. These factors may include, for example, hospital size, profitability, patient case and insurance mix, and presence of an electronic medical record. Comparing hospitals with vastly different abilities to achieve VBP goals and improve quickly may amount to inequitable policy.

Continual Evaluation of Topped‐Out Measures

Process measures that are met at high rates at nearly all hospitals are not used in evaluations by CMS for VBP. An assumption underlying CMS' decision to not reward hospitals for achieving these topped‐out measures is that once physicians and hospitals make cognitive and system‐level improvements that improve process quality, these gains will persist after the incentive is removed. Thus, CMS hopes and anticipates that although performance incentives will make it easier for well‐meaning physicians to learn to do the right thing, doctors will continue to do the right things for patients after these incentives are removed.[21, 22] Although this assumption may generally be accurate, it is important to continue to evaluate whether measures that are currently topped out continue to remain adequately performed, because rewarding new quality measures will necessarily lead hospitals to reallocate resources away from other clinical activities. Although we hope that the continued public reporting of topped‐out measures will prevent declines in performance on these measures, policy makers and clinicians should be aware that the lack of financial incentives for topped‐out measures may result in declines in quality. To this point, an analysis of 35 Kaiser Permanente facilities from 1997 to 2007 demonstrated that the removal of financial incentives for diabetic retinopathy and cervical cancer screening was associated with subsequent declines in performance of 3% and 1.6% per year, respectively.[23]

Will VBP Incentives Be Large Enough to Change Practice Patterns?

The VBP Program's ability to influence change depends, at least in part, on how the incentives offered under this program compare to the magnitude of the investments that hospitals must make to achieve a given reward. In general, larger incentives are necessary to motivate more significant changes in behavior or to influence organizations to invest the resources needed to achieve change. The incentives offered under VBP in FY 2013 are quite modest. Almost two‐thirds of participating hospitals will see their FY 2013 Medicare revenues change by <0.25%, roughly $125,000 at most.[13, 24] Although these incentives may motivate hospitals that can improve performance and achievement with very modest investments, they may have little impact on organizations that need to make significant upfront investments in care processes to achieve sustainable improvements in care quality. As CMS increases the size of VBP incentives over the next 2 to 4 years, it will also hold hospitals accountable for a broader and increasingly complex set of outcomes. Improving these outcomes may require investments in areas such as information technology and process improvement that far surpass the VBP incentive reward.

Moreover, prior research suggests that financial incentives like those available under VBP may contribute only slightly to performance improvements when public reporting already exists. For example, in a 2‐year study of 613 US hospitals implementing pay‐for‐performance plus public reporting or public reporting only, pay for performance plus public reporting was associated with only a 2.6% to 4.1% increase in a composite measure of quality when compared to hospitals with public reporting only.[9] Similarly, a study of 54 hospitals participating in the CMS pay for performance pilot initiative found no significant improvement in quality of care or outcomes for AMI when compared to 446 control hospitals.[25] A long‐term analysis of pay for performance in the Medicare Premier Hospital Quality Incentive Demonstration found that participation in the program had no discernible effect on 30‐day mortality rates.[10] Finally, a study of physician medical groups contracting with a large network healthcare maintenance organization found that the implementation of pay for performance did not result in major before and after improvements in clinical quality compared to a control group of medical groups.[26]

Hospital Eligibility for VBP

All acute care hospitals in the United States (excluding Maryland) that are not psychiatric hospitals, rehabilitation hospitals, long‐term care facilities, children's hospitals, or cancer hospitals are eligible to participate in VBP in FY 2013 (full eligibility criteria is outlined in Table 1). For FY 2013, CMS chose to incentivize measures in just 2 care domains: (1) clinical processes of care and (2) patient experience of care. To be eligible for VBP in FY 2013, a hospital must report at least 10 cases each in at least 4 of 12 measures included in the clinical processes of care domain (Table 2), and/or must have at least 100 completed Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS). Designed and validated by CMS, the HCAHPS survey provides hospitals with a standardized instrument for gathering information about patient satisfaction with, and perspectives on, their hospital care.[12] HCAHPS will be used to assess 8 patient experience of care measures (Table 3).

Participation in the program is mandatory for eligible hospitals, and CMS estimates that more than 3000 facilities across the United States will participate in FY 2013. Roughly $850 million dollars in VBP incentives will be paid out to these participating hospitals in FY 2013. The program is being financed through a 1% across‐the‐board reduction in FY 2013 diagnosis‐related group (DRG)‐based inpatient payments to participating hospitals. On December 20, 2012, CMS publically announced FY 2013 VBP incentives for all participating hospitals. Each hospital's incentive is retroactive and based on its performance between July 1, 2011 and March 31, 2012.

All data used for calculating VBP incentives is reported to CMS through its Hospital Inpatient Quality Reporting (Hospital IQR) Program, a national program instituted in 2003 that rewards hospitals for reporting designated quality measures. As of 2007, approximately 95% of eligible US hospitals were using the Hospital IQR program.[1] Measures evaluated via chart abstracts and surveys reflect a hospital's performance for its entire patient population, whereas measures assessed with claims data reflect hospital performance only for Medicare patients.

Evaluation of Hospitals

In FY 2013, hospital VBP incentive payments will be based entirely on performance in 2 domains: (1) clinical processes of care (weighted 70%) and (2) patient experience of care (weighted 30%). For each domain, CMS will evaluate each hospital's improvement over time as well as achievement compared to other hospitals in the VBP program. By assessing and rewarding both achievement and improvement, CMS will ensure that lower‐performing hospitals will still be rewarded for making substantial improvements in quality. To evaluate the first metricimprovement over timeCMS will compare a hospital's performance during a given reporting period with its baseline performance 2 years prior to this block of time. A hospital receives improvement points for improving its performance over time. To assess the second metricachievement compared to other hospitals in the VBP programCMS will compare each hospital's performance during a reporting period with the baseline performance (eg, performance 2 years prior to reporting period) of all other hospitals in the VBP program. A hospital is awarded achievement points if its performance exceeds the 50th percentile of all hospitals during the baseline performance period. Improvement scores range from 0 to 9, whereas achievement scores range from 0 to 10. The greater of a hospital's improvement and achievement scores on each VBP measure are used to calculate each hospital's total earned clinical care domain score and total earned HCAHPS base score. Hospitals that lack baseline performance data, which is required to assess improvement, will be evaluated solely on the basis of achievement points.[1] The total earned clinical care domain score is multiplied by 70% to reach the clinical care domain's contribution to a hospital's total performance score.

Each hospital's total patient experience domain, or HCAHPS performance, score consists of 2 components: a total earned HCAHPS base score as described above and a consistency score. The consistency score evaluates the reliability of a hospital's performance across all 8 patient experience of care measures (Table 3). If a hospital is above the 50th percentile of all hospital scores during the baseline period on all 8 measures, then it receives 100% of its consistency points. If a hospital is at the 0 percentile for a given measure, then it receives 0 consistency points for all measures. This provision promotes consistency by harshly penalizing hospitals with extremely poor performance on any 1 specific measure. If 1 or more measures are between the 0 and 50th percentiles, then it will receive a consistency score that takes into account how many measures were below the 50th percentile and their distance from this threshold. Each hospital's total HCAHPS performance score (the sum of total earned HCAHPS base points and consistency points) is then multiplied by 30% to arrive at the patient experience of care domain's contribution to a hospital's total performance score.

Importantly, CMS excluded from its VBP initiative 10 clinical process measures reported in the Hospital IQR Program because they are topped out; that is, almost all hospitals already perform them at very high rates (Table 4). Examples of these topped out process measures include administration of aspirin to all patients with AMI on arrival at the hospital; counseling of patients with AMI, CHF, and pneumonia about smoking cessation; and prescribing angiotensin‐converting enzyme inhibitors or angiotensin receptor blockers to patients with CHF and left ventricular dysfunction.[1]

Calculation of VBP Incentives and Public Reporting

A hospital's total performance score for FY 2013 is equal to the sum of 70% of its clinical care domain score and 30% of its total HCAHPS performance score. This total performance score is entered into a linear mathematical formula to calculate each hospital's incentive payment. CMS projects that VBP will lead to a net increase in Medicare payments for one‐half of hospitals and a net decrease in payments for the other half of participating facilities.[1]

In December 2012, CMS publicly disclosed information about the initial performance of each hospital in the VBP program. Reported information included: (1) hospital performance for each applicable performance measure, (2) hospital performance by disease condition or procedure, and (3) hospital's total performance score. Initial analyses of this performance data revealed that 1557 hospitals will receive bonus payments under VBP in FY 2013, whereas 1427 hospitals will lose money under this program. Treasure Valley Hospital, a 10‐bed physician‐owned hospital in Boise, Idaho, will receive a 0.83% increase in Medicare payments, the largest payment increase under VBP in 2013. Conversely, Auburn Community Hospital in upstate New York, will suffer the most severe payment reduction: 0.9% per Medicare admission. The penalty will cost Auburn Hospital about $100,000, which is slightly more than 0.1% of its yearly $85 million operating budget.[13] For almost two‐thirds of participating hospitals, FY 2013 Medicare payments will change by <0.25%.[13] Additional information about VBP payments for FY 2013, including the number of hospitals who received VBP incentives and the size and range of these payments, is now accessible to the public through CMS' Hospital Compare Web site (http://www.hospitalcompare.hhs.gov).

Rigorous and Continuous Evaluation of VBP Programs

The main premise of using VBP to incentivize hospitals to deliver high‐quality cost‐effective care is that the process measures used to determine hospital quality do impact patient outcomes. However, it is already well established that improvements in measures of process quality are not always associated with improvements in patient outcomes.[14, 15, 16] Moreover, incentivizing specific process measures encourages hospitals to shift resources away from other aspects of care delivery, which may have ambiguous, or even deleterious, effects on patient outcomes. Although incentives ideally push hospitals to shift resources away from low‐quality care toward high‐quality care, in practice this is not always the case. Hospital resources may instead be drawn away from areas that are not yet incented by VBP, but for which improvements in quality of care are desperately needed. The same empirical focus behind using VBP to incentivize hospitals to improve patient outcomes efficiently should be used to evaluate whether VBP is continually meeting its stated goals: reducing overall patient morbidity and mortality and improving patient satisfaction at ideally lower cost. The experience of the US education system with public policies designed to improve student testing performance may serve as a cautionary example here. Such policies, which provide financial rewards to schools whose students perform well on standardized tests, can indeed raise testing performance. However, these policies also lead educators to teach to the test, and to neglect important topics that are not tested on standardized exams.[17]

Prioritization of Process Measures

As payment incentives for VBP currently stand, process measures are weighted equally regardless of the clinical benefits they generate and the resources required to achieve improvements in process quality. For instance, 2 process measures, continuing home ‐blocker medications for patients with coronary artery disease undergoing surgery and early percutaneous coronary intervention for patients with AMI, may be weighted equally as process measures although both their clinical benefits and the costs of implementation are very different. Some hospitals responding to VBP incentives may choose to invest in areas where their ability to earn VBP incentive payments is high and the costs of improvement are low, although those areas may not be where interventions are most needed because clinical outcomes could be most improved. Recognizing that process measures have heterogeneous benefits and costs of implementation is important when prioritizing their reimbursement in VBP.

Measuring Improvements in Hospital Quality

Tying hospital financial compensation to hospital quality implies that measures of hospital quality should be robust. To incentivize hospitals to improve quality not only relative to other hospitals but to themselves in the past, the VBP program has established a baseline performance for each hospital. Each hospital is compared to its baseline performance in subsequent evaluation periods. Thus, properly measuring a hospital's baseline performance is important. During a given baseline period, some hospitals may have better or worse outcomes than their steady state due to random variation alone. Some hospitals deemed to have a low baseline will experience improvements in quality that are not related to active efforts to improve quality but through chance alone. Similarly, some hospitals deemed to have a high baseline will experience reductions in quality through chance. Of course, neither of these changes should be subject to differences in reimbursement because they do not reflect actual organizational changes made by the hospitals. The VBP program has made significant efforts to address this issue by requiring participating hospitals to have a large enough sample of cases such that estimated rates of process quality adherence meet a reliability threshold (ie, are likely to be consistent over time rather than vary substantially through chance alone). However, not all process measures exhibit high reliability, particularly those for which adverse events are rare (eg, foreign objects retained after surgery, air embolisms, and blood incompatibility). Ultimately, CMS's decision to balance the need for statistically reliable data with the goal of including as many hospitals as possible in the VBP program will require ongoing reevaluation of this issue.

Choosing Hospital Comparators Appropriately

In the current VBP program, hospitals will be evaluated in part by how they compare to hospitals nationally. However, studies of regional variation in healthcare have demonstrated large variations in practice patterns across the United States,[18, 19, 20] raising the question of whether hospitals should, at least initially, be compared to hospitals in the same geographic area. Although the ultimate goal of VBP should be to hold hospitals to a national standard, local practice patterns are not easily modified within 1‐ to 2‐year timeframes. Initially comparing hospitals to a national rather than local standard may unfairly penalize hospitals that are relative underperformers nationally but overperformers regionally. Although CMS's policy to reward improvement within hospitals over time mitigates issues arising from a cross‐sectional comparison of hospitals, the issue still remains if many hospitals within a region not only underperform relative to other hospitals nationally but also fail to demonstrate improvement. More broadly, this issue extends to differences across hospitals in factors that impact their ability to meet VBP goals. These factors may include, for example, hospital size, profitability, patient case and insurance mix, and presence of an electronic medical record. Comparing hospitals with vastly different abilities to achieve VBP goals and improve quickly may amount to inequitable policy.

Continual Evaluation of Topped‐Out Measures

Process measures that are met at high rates at nearly all hospitals are not used in evaluations by CMS for VBP. An assumption underlying CMS' decision to not reward hospitals for achieving these topped‐out measures is that once physicians and hospitals make cognitive and system‐level improvements that improve process quality, these gains will persist after the incentive is removed. Thus, CMS hopes and anticipates that although performance incentives will make it easier for well‐meaning physicians to learn to do the right thing, doctors will continue to do the right things for patients after these incentives are removed.[21, 22] Although this assumption may generally be accurate, it is important to continue to evaluate whether measures that are currently topped out continue to remain adequately performed, because rewarding new quality measures will necessarily lead hospitals to reallocate resources away from other clinical activities. Although we hope that the continued public reporting of topped‐out measures will prevent declines in performance on these measures, policy makers and clinicians should be aware that the lack of financial incentives for topped‐out measures may result in declines in quality. To this point, an analysis of 35 Kaiser Permanente facilities from 1997 to 2007 demonstrated that the removal of financial incentives for diabetic retinopathy and cervical cancer screening was associated with subsequent declines in performance of 3% and 1.6% per year, respectively.[23]

Will VBP Incentives Be Large Enough to Change Practice Patterns?

The VBP Program's ability to influence change depends, at least in part, on how the incentives offered under this program compare to the magnitude of the investments that hospitals must make to achieve a given reward. In general, larger incentives are necessary to motivate more significant changes in behavior or to influence organizations to invest the resources needed to achieve change. The incentives offered under VBP in FY 2013 are quite modest. Almost two‐thirds of participating hospitals will see their FY 2013 Medicare revenues change by <0.25%, roughly $125,000 at most.[13, 24] Although these incentives may motivate hospitals that can improve performance and achievement with very modest investments, they may have little impact on organizations that need to make significant upfront investments in care processes to achieve sustainable improvements in care quality. As CMS increases the size of VBP incentives over the next 2 to 4 years, it will also hold hospitals accountable for a broader and increasingly complex set of outcomes. Improving these outcomes may require investments in areas such as information technology and process improvement that far surpass the VBP incentive reward.

Moreover, prior research suggests that financial incentives like those available under VBP may contribute only slightly to performance improvements when public reporting already exists. For example, in a 2‐year study of 613 US hospitals implementing pay‐for‐performance plus public reporting or public reporting only, pay for performance plus public reporting was associated with only a 2.6% to 4.1% increase in a composite measure of quality when compared to hospitals with public reporting only.[9] Similarly, a study of 54 hospitals participating in the CMS pay for performance pilot initiative found no significant improvement in quality of care or outcomes for AMI when compared to 446 control hospitals.[25] A long‐term analysis of pay for performance in the Medicare Premier Hospital Quality Incentive Demonstration found that participation in the program had no discernible effect on 30‐day mortality rates.[10] Finally, a study of physician medical groups contracting with a large network healthcare maintenance organization found that the implementation of pay for performance did not result in major before and after improvements in clinical quality compared to a control group of medical groups.[26]

Study Design

We performed a case‐crossover study of children who experienced clinical deterioration. An alternative to the matched case‐control design, the case‐crossover design involves longitudinal within‐subject comparisons exclusively of case subjects such that an individual serves as his or her own control. It is most effective when studying intermittent exposures that result in transient changes in the risk of an acute event,[15, 16, 17] making it appropriate for our study.

Using the case‐crossover design, we compared a discrete time period in close proximity to the deterioration event, called the hazard interval, with earlier time periods in the hospitalization, called the control intervals.[15, 16, 17] In our primary analysis (Figure 1B), we defined the durations of these intervals as follows: We first censored the 2 hours immediately preceding the clinical deterioration event (hours 0 to 2). We made this decision a priori to exclude medications used after deterioration was recognized and resuscitation had already begun. The 12‐hour period immediately preceding the censored interval was the hazard interval (hours 2 to 14). Each 12‐hour period immediately preceding the hazard interval was a control interval (hours 14 to 26, 26 to 38, 38 to 50, and 50 to 62). Depending on the child's length of stay prior to the deterioration event, each hazard interval had 14 control intervals for comparison. In sensitivity analysis, we altered the durations of these intervals (see below).

Figure 1

Study Setting and Participants

We performed this study among children age <18 years who experienced clinical deterioration between January 1, 2005, and December 31, 2008, after being hospitalized on a general medical or surgical unit at The Children's Hospital of Philadelphia for 24 hours. Clinical deterioration was a composite outcome defined as cardiopulmonary arrest (CPA), acute respiratory compromise (ARC), or urgent ICU transfer. Cardiopulmonary arrest events required either pulselessness or a pulse with inadequate perfusion treated with chest compressions and/or defibrillation. Acute respiratory compromise events required respiratory insufficiency treated with bag‐valve‐mask or invasive airway interventions. Urgent ICU transfers included 1 of the following outcomes in the 12 hours after transfer: death, CPA, intubation, initiation of noninvasive ventilation, or administration of a vasoactive medication infusion used for the treatment of shock. Time zero was the time of the CPA/ARC, or the time at which the child arrived in the ICU for urgent transfers. These subjects also served as the cases for a previously published case‐control study evaluating different risk factors for deterioration.[18] The institutional review board of The Children's Hospital of Philadelphia approved the study.

At the time of the study, the hospital did not have a formal RRS. An immediate‐response code‐blue team was available throughout the study period for emergencies occurring outside the ICU. Physicians could also page the pediatric ICU fellow to discuss patients who did not require immediate assistance from the code‐blue team but were clinically deteriorating. There were no established triggering criteria.

Medication Exposures

Intravenous (IV) medications administered in the 72 hours prior to clinical deterioration were considered the exposures of interest. Each medication was included in 1 therapeutic classes assigned in the hospital's formulary (Lexicomp, Hudson, OH).[19] In order to determine which therapeutic classes to evaluate, we performed a power calculation using the sampsi_mcc package for Stata 12 (StataCorp, College Station, TX). We estimated that we would have 3 matched control intervals per hazard interval. We found that, in order to detect a minimum odds ratio of 3.0 with 80% power, a therapeutic class had to be administered in 5% of control periods. All therapeutic classes meeting that requirement were included in the analysis and are listed in Table 1. (See lists of the individual medications comprising each class in the Supporting Information, Tables 124, in the online version of this article.)

Data Collection

Data were abstracted from the electronic medication administration record (Sunrise Clinical Manager; Allscripts, Chicago, IL) into a database. For each subject, we recorded the name and time of administration of each IV medication given in the 72 hours preceding deterioration, as well as demographic, event, and hospitalization characteristics.

Statistical Analysis

We used univariable conditional logistic regression to evaluate the association between each therapeutic class and the composite outcome of clinical deterioration in the primary analysis. Because cases serve as their own controls in the case‐crossover design, this method inherently adjusts for all subject‐specific time‐invariant confounding variables, such as patient demographics, disease, and hospital‐ward characteristics.[15]

Sensitivity Analysis

Our primary analysis used a 2‐hour censored interval and 12‐hour hazard and control intervals. Excluding the censored interval from analysis was a conservative approach that we chose because our goal was to identify therapeutic classes associated with deterioration during a phase in which adverse outcomes may be prevented with early intervention. In order to test whether our findings were stable across different lengths of censored, hazard, and control intervals, we performed a sensitivity analysis, also using conditional logistic regression, on all therapeutic classes that were significant (P<0.05) in primary analysis. In 6 iterations of the sensitivity analysis, we varied the length of the hazard and control intervals between 8 and 12 hours, and the length of the censored interval between 0 and 4 hours (Figure 1AF). In a seventh iteration, we used a variant of the primary analysis in which we censored the first control interval (Figure 1G).

Primary Analysis

A total of 141 hazard intervals and 487 control intervals were included in the primary analysis, the results of which are shown in Table 3. Among the antimicrobial therapeutic classes, glycopeptide antibiotics (vancomycin), anaerobic antibiotics, third‐generation and fourth‐generation cephalosporins, and aminoglycoside antibiotics were significant. All of the anti‐inflammatory therapeutic classes, including systemic corticosteroids, anti‐inflammatory agents, and adrenal corticosteroids, were significant. All of the sedatives, hypnotics, and antianxiety therapeutic classes, including sedatives, benzodiazepines, hypnotics, and antianxiety agents, were significant. Among the narcotic analgesic therapeutic classes, only 1 class, narcotic analgesics (full opioid agonists), was significant. None of the gastrointestinal therapeutic classes were significant. Among the classes classified as other, loop diuretics and antidotes to hypersensitivity reactions (diphenhydramine) were significant.

Sensitivity Analysis

Of the 14 classes that were significant in primary analysis, we carried 9 forward to sensitivity analysis. The 5 that were not carried forward overlapped substantially with other classes that were carried forward. The decision of which overlapping class to carry forward was based upon (1) parsimony and (2) clinical relevance. This is described briefly in the footnotes to Table 3 (see Supporting information in the online version of this article for a full description of this process). Figure 2 presents the odds ratios and their 95% confidence intervals for the sensitivity analysis of each therapeutic class that was significant in primary analysis. Loop diuretics remained significantly associated with deterioration in all 7 iterations. Glycopeptide antibiotics (vancomycin), third‐generation and fourth‐generation cephalosporins, systemic corticosteroids, and benzodiazepines were significant in 6. Anaerobic antibiotics and narcotic analgesics (full opioid agonists) were significant in 5, and aminoglycoside antibiotics and antidotes to hypersensitivity reactions (diphenhydramine) in 4.

Figure 2

Study Design

We performed a case‐crossover study of children who experienced clinical deterioration. An alternative to the matched case‐control design, the case‐crossover design involves longitudinal within‐subject comparisons exclusively of case subjects such that an individual serves as his or her own control. It is most effective when studying intermittent exposures that result in transient changes in the risk of an acute event,[15, 16, 17] making it appropriate for our study.

Using the case‐crossover design, we compared a discrete time period in close proximity to the deterioration event, called the hazard interval, with earlier time periods in the hospitalization, called the control intervals.[15, 16, 17] In our primary analysis (Figure 1B), we defined the durations of these intervals as follows: We first censored the 2 hours immediately preceding the clinical deterioration event (hours 0 to 2). We made this decision a priori to exclude medications used after deterioration was recognized and resuscitation had already begun. The 12‐hour period immediately preceding the censored interval was the hazard interval (hours 2 to 14). Each 12‐hour period immediately preceding the hazard interval was a control interval (hours 14 to 26, 26 to 38, 38 to 50, and 50 to 62). Depending on the child's length of stay prior to the deterioration event, each hazard interval had 14 control intervals for comparison. In sensitivity analysis, we altered the durations of these intervals (see below).

Figure 1

Study Setting and Participants

We performed this study among children age <18 years who experienced clinical deterioration between January 1, 2005, and December 31, 2008, after being hospitalized on a general medical or surgical unit at The Children's Hospital of Philadelphia for 24 hours. Clinical deterioration was a composite outcome defined as cardiopulmonary arrest (CPA), acute respiratory compromise (ARC), or urgent ICU transfer. Cardiopulmonary arrest events required either pulselessness or a pulse with inadequate perfusion treated with chest compressions and/or defibrillation. Acute respiratory compromise events required respiratory insufficiency treated with bag‐valve‐mask or invasive airway interventions. Urgent ICU transfers included 1 of the following outcomes in the 12 hours after transfer: death, CPA, intubation, initiation of noninvasive ventilation, or administration of a vasoactive medication infusion used for the treatment of shock. Time zero was the time of the CPA/ARC, or the time at which the child arrived in the ICU for urgent transfers. These subjects also served as the cases for a previously published case‐control study evaluating different risk factors for deterioration.[18] The institutional review board of The Children's Hospital of Philadelphia approved the study.

At the time of the study, the hospital did not have a formal RRS. An immediate‐response code‐blue team was available throughout the study period for emergencies occurring outside the ICU. Physicians could also page the pediatric ICU fellow to discuss patients who did not require immediate assistance from the code‐blue team but were clinically deteriorating. There were no established triggering criteria.

Medication Exposures

Intravenous (IV) medications administered in the 72 hours prior to clinical deterioration were considered the exposures of interest. Each medication was included in 1 therapeutic classes assigned in the hospital's formulary (Lexicomp, Hudson, OH).[19] In order to determine which therapeutic classes to evaluate, we performed a power calculation using the sampsi_mcc package for Stata 12 (StataCorp, College Station, TX). We estimated that we would have 3 matched control intervals per hazard interval. We found that, in order to detect a minimum odds ratio of 3.0 with 80% power, a therapeutic class had to be administered in 5% of control periods. All therapeutic classes meeting that requirement were included in the analysis and are listed in Table 1. (See lists of the individual medications comprising each class in the Supporting Information, Tables 124, in the online version of this article.)

Data Collection

Data were abstracted from the electronic medication administration record (Sunrise Clinical Manager; Allscripts, Chicago, IL) into a database. For each subject, we recorded the name and time of administration of each IV medication given in the 72 hours preceding deterioration, as well as demographic, event, and hospitalization characteristics.

Statistical Analysis

We used univariable conditional logistic regression to evaluate the association between each therapeutic class and the composite outcome of clinical deterioration in the primary analysis. Because cases serve as their own controls in the case‐crossover design, this method inherently adjusts for all subject‐specific time‐invariant confounding variables, such as patient demographics, disease, and hospital‐ward characteristics.[15]

Sensitivity Analysis

Our primary analysis used a 2‐hour censored interval and 12‐hour hazard and control intervals. Excluding the censored interval from analysis was a conservative approach that we chose because our goal was to identify therapeutic classes associated with deterioration during a phase in which adverse outcomes may be prevented with early intervention. In order to test whether our findings were stable across different lengths of censored, hazard, and control intervals, we performed a sensitivity analysis, also using conditional logistic regression, on all therapeutic classes that were significant (P<0.05) in primary analysis. In 6 iterations of the sensitivity analysis, we varied the length of the hazard and control intervals between 8 and 12 hours, and the length of the censored interval between 0 and 4 hours (Figure 1AF). In a seventh iteration, we used a variant of the primary analysis in which we censored the first control interval (Figure 1G).

Primary Analysis

A total of 141 hazard intervals and 487 control intervals were included in the primary analysis, the results of which are shown in Table 3. Among the antimicrobial therapeutic classes, glycopeptide antibiotics (vancomycin), anaerobic antibiotics, third‐generation and fourth‐generation cephalosporins, and aminoglycoside antibiotics were significant. All of the anti‐inflammatory therapeutic classes, including systemic corticosteroids, anti‐inflammatory agents, and adrenal corticosteroids, were significant. All of the sedatives, hypnotics, and antianxiety therapeutic classes, including sedatives, benzodiazepines, hypnotics, and antianxiety agents, were significant. Among the narcotic analgesic therapeutic classes, only 1 class, narcotic analgesics (full opioid agonists), was significant. None of the gastrointestinal therapeutic classes were significant. Among the classes classified as other, loop diuretics and antidotes to hypersensitivity reactions (diphenhydramine) were significant.

Sensitivity Analysis

Of the 14 classes that were significant in primary analysis, we carried 9 forward to sensitivity analysis. The 5 that were not carried forward overlapped substantially with other classes that were carried forward. The decision of which overlapping class to carry forward was based upon (1) parsimony and (2) clinical relevance. This is described briefly in the footnotes to Table 3 (see Supporting information in the online version of this article for a full description of this process). Figure 2 presents the odds ratios and their 95% confidence intervals for the sensitivity analysis of each therapeutic class that was significant in primary analysis. Loop diuretics remained significantly associated with deterioration in all 7 iterations. Glycopeptide antibiotics (vancomycin), third‐generation and fourth‐generation cephalosporins, systemic corticosteroids, and benzodiazepines were significant in 6. Anaerobic antibiotics and narcotic analgesics (full opioid agonists) were significant in 5, and aminoglycoside antibiotics and antidotes to hypersensitivity reactions (diphenhydramine) in 4.

Figure 2

Study Design

We performed a case‐crossover study of children who experienced clinical deterioration. An alternative to the matched case‐control design, the case‐crossover design involves longitudinal within‐subject comparisons exclusively of case subjects such that an individual serves as his or her own control. It is most effective when studying intermittent exposures that result in transient changes in the risk of an acute event,[15, 16, 17] making it appropriate for our study.

Using the case‐crossover design, we compared a discrete time period in close proximity to the deterioration event, called the hazard interval, with earlier time periods in the hospitalization, called the control intervals.[15, 16, 17] In our primary analysis (Figure 1B), we defined the durations of these intervals as follows: We first censored the 2 hours immediately preceding the clinical deterioration event (hours 0 to 2). We made this decision a priori to exclude medications used after deterioration was recognized and resuscitation had already begun. The 12‐hour period immediately preceding the censored interval was the hazard interval (hours 2 to 14). Each 12‐hour period immediately preceding the hazard interval was a control interval (hours 14 to 26, 26 to 38, 38 to 50, and 50 to 62). Depending on the child's length of stay prior to the deterioration event, each hazard interval had 14 control intervals for comparison. In sensitivity analysis, we altered the durations of these intervals (see below).

Figure 1

Study Setting and Participants

We performed this study among children age <18 years who experienced clinical deterioration between January 1, 2005, and December 31, 2008, after being hospitalized on a general medical or surgical unit at The Children's Hospital of Philadelphia for 24 hours. Clinical deterioration was a composite outcome defined as cardiopulmonary arrest (CPA), acute respiratory compromise (ARC), or urgent ICU transfer. Cardiopulmonary arrest events required either pulselessness or a pulse with inadequate perfusion treated with chest compressions and/or defibrillation. Acute respiratory compromise events required respiratory insufficiency treated with bag‐valve‐mask or invasive airway interventions. Urgent ICU transfers included 1 of the following outcomes in the 12 hours after transfer: death, CPA, intubation, initiation of noninvasive ventilation, or administration of a vasoactive medication infusion used for the treatment of shock. Time zero was the time of the CPA/ARC, or the time at which the child arrived in the ICU for urgent transfers. These subjects also served as the cases for a previously published case‐control study evaluating different risk factors for deterioration.[18] The institutional review board of The Children's Hospital of Philadelphia approved the study.

At the time of the study, the hospital did not have a formal RRS. An immediate‐response code‐blue team was available throughout the study period for emergencies occurring outside the ICU. Physicians could also page the pediatric ICU fellow to discuss patients who did not require immediate assistance from the code‐blue team but were clinically deteriorating. There were no established triggering criteria.

Medication Exposures

Intravenous (IV) medications administered in the 72 hours prior to clinical deterioration were considered the exposures of interest. Each medication was included in 1 therapeutic classes assigned in the hospital's formulary (Lexicomp, Hudson, OH).[19] In order to determine which therapeutic classes to evaluate, we performed a power calculation using the sampsi_mcc package for Stata 12 (StataCorp, College Station, TX). We estimated that we would have 3 matched control intervals per hazard interval. We found that, in order to detect a minimum odds ratio of 3.0 with 80% power, a therapeutic class had to be administered in 5% of control periods. All therapeutic classes meeting that requirement were included in the analysis and are listed in Table 1. (See lists of the individual medications comprising each class in the Supporting Information, Tables 124, in the online version of this article.)

Data Collection

Data were abstracted from the electronic medication administration record (Sunrise Clinical Manager; Allscripts, Chicago, IL) into a database. For each subject, we recorded the name and time of administration of each IV medication given in the 72 hours preceding deterioration, as well as demographic, event, and hospitalization characteristics.

Statistical Analysis

We used univariable conditional logistic regression to evaluate the association between each therapeutic class and the composite outcome of clinical deterioration in the primary analysis. Because cases serve as their own controls in the case‐crossover design, this method inherently adjusts for all subject‐specific time‐invariant confounding variables, such as patient demographics, disease, and hospital‐ward characteristics.[15]

Sensitivity Analysis

Our primary analysis used a 2‐hour censored interval and 12‐hour hazard and control intervals. Excluding the censored interval from analysis was a conservative approach that we chose because our goal was to identify therapeutic classes associated with deterioration during a phase in which adverse outcomes may be prevented with early intervention. In order to test whether our findings were stable across different lengths of censored, hazard, and control intervals, we performed a sensitivity analysis, also using conditional logistic regression, on all therapeutic classes that were significant (P<0.05) in primary analysis. In 6 iterations of the sensitivity analysis, we varied the length of the hazard and control intervals between 8 and 12 hours, and the length of the censored interval between 0 and 4 hours (Figure 1AF). In a seventh iteration, we used a variant of the primary analysis in which we censored the first control interval (Figure 1G).

Primary Analysis

A total of 141 hazard intervals and 487 control intervals were included in the primary analysis, the results of which are shown in Table 3. Among the antimicrobial therapeutic classes, glycopeptide antibiotics (vancomycin), anaerobic antibiotics, third‐generation and fourth‐generation cephalosporins, and aminoglycoside antibiotics were significant. All of the anti‐inflammatory therapeutic classes, including systemic corticosteroids, anti‐inflammatory agents, and adrenal corticosteroids, were significant. All of the sedatives, hypnotics, and antianxiety therapeutic classes, including sedatives, benzodiazepines, hypnotics, and antianxiety agents, were significant. Among the narcotic analgesic therapeutic classes, only 1 class, narcotic analgesics (full opioid agonists), was significant. None of the gastrointestinal therapeutic classes were significant. Among the classes classified as other, loop diuretics and antidotes to hypersensitivity reactions (diphenhydramine) were significant.

Sensitivity Analysis

Of the 14 classes that were significant in primary analysis, we carried 9 forward to sensitivity analysis. The 5 that were not carried forward overlapped substantially with other classes that were carried forward. The decision of which overlapping class to carry forward was based upon (1) parsimony and (2) clinical relevance. This is described briefly in the footnotes to Table 3 (see Supporting information in the online version of this article for a full description of this process). Figure 2 presents the odds ratios and their 95% confidence intervals for the sensitivity analysis of each therapeutic class that was significant in primary analysis. Loop diuretics remained significantly associated with deterioration in all 7 iterations. Glycopeptide antibiotics (vancomycin), third‐generation and fourth‐generation cephalosporins, systemic corticosteroids, and benzodiazepines were significant in 6. Anaerobic antibiotics and narcotic analgesics (full opioid agonists) were significant in 5, and aminoglycoside antibiotics and antidotes to hypersensitivity reactions (diphenhydramine) in 4.

Figure 2

Patients and Setting

We conducted a retrospective cohort study of consecutive adult patients discharged from medicine services at Brigham and Women's Hospital (BWH), a 747‐bed tertiary referral center and teaching hospital, between July 1, 2009 and June 30, 2010. Most patients are cared for by resident housestaff teams at BWH (approximately 25% are cared for by physician assistants working directly with attending physicians), and approximately half receive primary care in the Partners system, which has a shared electronic medical record (EMR). Outpatient mental health services are provided by Partners‐associated mental health professionals including those at McLean Hospital and MassHealth (Medicaid)‐associated sites through the Massachusetts Behavioral Health Partnership. Exclusion criteria were death in the hospital or discharge to another acute care facility. We also excluded patients who left against medical advice (AMA). The study protocol was approved by the Partners Institutional Review Board.

Outcome

The primary outcomes were ACR and PAR within 30 days of discharge. First, we identified all 30‐day readmissions to BWH or to 2 other hospitals in the Partners Healthcare Network (previous studies have shown that 80% of all readmitted patients are readmitted to 1 of these 3 hospitals).[12] For patients with multiple readmissions, only the first readmission was included in the dataset.

To find potentially avoidable readmissions, administrative and billing data for these patients were processed using the SQLape (SQLape s.a.r.l., Corseaux, Switzerland) algorithm, which identifies PAR by excluding patients who undergo planned follow‐up treatment (such as a cycle of planned chemotherapy) or are readmitted for conditions unrelated in any way to the index hospitalization.[13, 14] Common complications of treatment are categorized as potentially avoidable, such as development of a deep venous thrombosis, a decubitus ulcer after prolonged bed rest, or bleeding complications after starting anticoagulation. Although the algorithm identifies theoretically preventable readmissions, the algorithm does not quantify how preventable they are, and these are thus referred to as potentially avoidable. This is similar to other admission metrics, such as the Agency for Healthcare Research and Quality's prevention quality indicators, which are created from a list of ambulatory care‐sensitive conditions.[15] SQLape has the advantage of being a specific tool for readmissions. Patients with 30‐day readmissions identified by SQLape as planned or unlikely to be avoidable were excluded in the PAR analysis, although still included in ACR analysis. In each case, the comparison group is patients without any readmission.

Predictors

Our predictors of interest included the overall prevalence of a psychiatric diagnosis or diagnosis of substance abuse, the presence of specific psychiatric diagnoses, and prescription of psychiatric medications to help assess the independent contribution of these comorbidities to readmission risk.

We used a combination of easily obtainable inpatient and outpatient clinical and administrative data to identify relevant patients. Patients were considered likely to be psychiatrically ill if they: (1) had a psychiatric diagnosis on their Partners outpatient EMR problem list and were prescribed a medication to treat that condition as an outpatient, or (2) had an International Classification of Diseases, 9th Revision diagnosis of a psychiatric illness at hospital discharge. Patients were considered to have moderate probability of disease if they: (1) had a psychiatric diagnosis on their outpatient problem list, or (2) were prescribed a medication intended to treat a psychiatric condition as an outpatient. Patients were considered unlikely to have psychiatric disease if none of these criteria were met. Patients were considered likely to have a substance abuse disorder if they had this diagnosis on their outpatient EMR, or were prescribed a medication to treat this condition (eg, buprenorphine/naloxone), or received inpatient consultation from a substance abuse treatment team during their inpatient hospitalization, and were considered unlikely if none of these were true. We also evaluated individual categories of psychiatric illness (schizophrenia, depression, anxiety, bipolar disorder) and of psychotropic medications (antidepressants, antipsychotics, anxiolytics).

Potential Confounders

Data on potential confounders, based on prior literature,[16, 17] collected at the index admission were derived from electronic administrative, clinical, and billing sources, including the Brigham Integrated Computer System and the Partners Clinical Data Repository. They included patient age, gender, ethnicity, primary language, marital status, insurance status, living situation prior to admission, discharge location, length of stay, Elixhauser comorbidity index,[18] total number of medications prescribed, and number of prior admissions and ED visits in the prior year.

Statistical Analysis

Bivariate comparisons of each of the predictors of ACR and PAR risk (ie, patients with a 30‐day ACR or PAR vs those not readmitted within 30 days) were conducted using ² trend tests for ordinal predictors (eg, likelihood of psychiatric disease), and ² or Fisher exact test for dichotomous predictors (eg, receipt of inpatient substance abuse counseling).

We then used multivariate logistic regression analysis to adjust for all of the potential confounders noted above, entering each variable related to psychiatric illness into the model separately (eg, likely psychiatric illness, number of psychiatric medications). In a secondary analysis, we removed potentially collinear variables from the final model; as this did not alter the results, the full model is presented. We also conducted a secondary analysis where we included patients who left against medical advice (AMA), which also did not alter the results. Two‐sided P values <0.05 were considered significant, and all analyses were performed using the SAS version 9.2 (SAS Institute, Inc., Cary, NC).

All‐Cause Readmissions

After exclusion of 997 patients who died, were discharged to skilled nursing or rehabilitation facilities, or left AMA, 6987 patients were included (Figure 1). Of these, 1260 had a readmission (18%). Approximately half were considered unlikely to be psychiatrically ill, 22% were considered moderately likely, and 29% likely (Table 2).