Original Research

Hospital-acquired venous thromboembolism (VTE) continues to be a critical quality challenge for U.S. hospitals,¹ and high-risk patients are often not adequately prophylaxed. Use of VTE prophylaxis (VTEP) varies as widely as 26% to 85% of patients in various studies, as does patient outcomes and care expenditures.^2-6 The 9th edition of the American College of Chest Physicians (CHEST) guidelines⁷ recommend the Padua Prediction Score (PPS) to select individual patients who may be at high risk for venous thromboembolism (VTE) and could benefit from thromboprophylaxis. Use of the manually calculated PPS to select patients for thromboprophylaxis has been shown to help decrease 30-day and 90-day mortality associated with VTE events after hospitalization to medical services.⁸ However, the PPS requires time-consuming manual calculation by a provider, who may be focused on more immediate aspects of patient care and several other risk scores competing for his attention, potentially decreasing its use.

Other risk scores that use only discrete scalar data, such as vital signs and lab results to predict early recognition of sepsis, have been successfully automated and implemented within electronic health records (EHRs).^9-11 Successful automation of scores requiring input of diagnoses, recent medical events, and current clinical status such as the PPS remains difficult.¹² Data representing these characteristics are more prone to error, and harder to translate clearly into a single data field than discrete elements like heart rate, potentially impacting validity of the calculated result.¹³ To improve usage of guideline based VTE risk assessment and decrease physician burden, we developed an algorithm called Automated Padua Prediction Score (APPS) that automatically calculates the PPS using only EHR data available within prior encounters and the first 4 hours of admission, a similar timeframe to when admitting providers would be entering orders. Our goal was to assess if an automatically calculated version of the PPS, a score that depends on criteria more complex than vital signs and labs, would accurately assess risk for hospital-acquired VTE when compared to traditional manual calculation of the Padua Prediction Score by a provider.

Site Description and Ethics

The study was conducted at University of California, San Francisco Medical Center, a 790-bed academic hospital; its Institutional Review Board approved the study and collection of data via chart review. Handling of patient information complied with the Health Insurance Portability and Accountability Act of 1996.

Patient Inclusion

Adult patients admitted to a medical or surgical service between July 1, 2012 and April 1, 2014 were included in the study if they were candidates for VTEP, defined as: length of stay (LOS) greater than 2 days, not on hospice care, not pregnant at admission, no present on admission VTE diagnosis, no known contraindications to prophylaxis (eg, gastrointestinal bleed), and were not receiving therapeutic doses of warfarin, low molecular weight heparins, heparin, or novel anticoagulants prior to admission.

Data Sources

Clinical variables were extracted from the EHR’s enterprise data warehouse (EDW) by SQL Server query (Microsoft, Redmond, Washington) and deposited in a secure database. Chart review was conducted by a trained researcher (Mr. Jacolbia) using the EHR and a standardized protocol. Findings were recorded using REDCap (REDCap Consortium, Vanderbilt University, Nashville, Tennessee). The specific ICD-9, procedure, and lab codes used to determine each criterion of APPS are available in the Appendix.

Creation of the Automated Padua Prediction Score (APPS)

We developed APPS from the original 11 criteria that comprise the Padua Prediction Score: active cancer, previous VTE (excluding superficial vein thrombosis), reduced mobility, known thrombophilic condition, recent (1 month or less) trauma and/or surgery, age 70 years or older, heart and/or respiratory failure, acute myocardial infarction and/or ischemic stroke, acute infection and/or rheumatologic disorder, body mass index (BMI) 30 or higher, and ongoing hormonal treatment.¹³ APPS has the same scoring methodology as PPS: criteria are weighted from 1 to 3 points and summed with a maximum score of 20, representing highest risk of VTE. To automate the score calculation from data routinely available in the EHR, APPS checks pre-selected structured data fields for specific values within laboratory results, orders, nursing flowsheets and claims. Claims data included all ICD-9 and procedure codes used for billing purposes. If any of the predetermined data elements are found, then the specific criterion is considered positive; otherwise, it is scored as negative. The creators of the PPS were consulted in the generation of these data queries to replicate the original standards for deeming a criterion positive. The automated calculation required no use of natural language processing.

Characterization of Study Population

We recorded patient demographics (age, race, gender, BMI), LOS, and rate of hospital-acquired VTE. These patients were separated into 2 cohorts determined by the VTE prophylaxis they received. The risk profile of patients who received pharmacologic prophylaxis was hypothesized to be inherently different from those who had not. To evaluate APPS within this heterogeneous cohort, patients were divided into 2 major categories: pharmacologic vs. no pharmacologic prophylaxis. If they had a completed order or medication administration record on the institution’s approved formulary for pharmacologic VTEP, they were considered to have received pharmacologic prophylaxis. If they had only a completed order for usage of mechanical prophylaxis (sequential compression devices) or no evidence of any form of VTEP, they were considered to have received no pharmacologic prophylaxis. Patients with evidence of both pharmacologic and mechanical were placed in the pharmacologic prophylaxis group. To ensure that automated designation of prophylaxis group was accurate, we reviewed 40 randomly chosen charts because prior researchers were able to achieve sensitivity and specificity greater than 90% with that sample size.¹⁴

The primary outcome of hospital-acquired VTE was defined as an ICD-9 code for VTE (specific codes are found in the Appendix) paired with a “present on admission = no” flag on that encounter’s hospital billing data, abstracted from the EDW. A previous study at this institution used the same methodology and found 212/226 (94%) of patients with a VTE ICD-9 code on claim had evidence of a hospital-acquired VTE event upon chart review.¹⁴ Chart review was also completed to ensure that the primary outcome of newly discovered hospital-acquired VTE was differentiated from chronic VTE or history of VTE. Theoretically, ICD-9 codes and other data elements treat chronic VTE, history of VTE, and hospital-acquired VTE as distinct diagnoses, but it was unclear if this was true in our dataset. For 75 randomly selected cases of presumed hospital-acquired VTE, charts were reviewed for evidence that confirmed newly found VTE during that encounter.

Validation of APPS through Comparison to Manual Calculation of the Original PPS

To compare our automated calculation to standard clinical practice, we manually calculated the PPS through chart review within the first 2 days of admission on 300 random patients, a subsample of the entire study cohort. The largest study we could find had manually calculated the PPS of 1,080 hospitalized patients with a mean PPS of 4.86 (standard deviation [SD], 2.26).¹⁵ One researcher (Mr. Jacolbia) accessed the EHR with all patient information available to physicians, including admission notes, orders, labs, flowsheets, past medical history, and all prior encounters to calculate and record the PPS. To limit potential score bias, 2 authors (Drs. Elias and Davies) assessed 30 randomly selected charts from the cohort of 300. The standardized chart review protocol mimicked a physician’s approach to determine if a patient met a criterion, such as concluding if he/she had active cancer by examining medication lists for chemotherapy, procedure notes for radiation, and recent diagnoses on problem lists. After the original PPS was manually calculated, APPS was automatically calculated for the same 300 patients. We intended to characterize similarities and differences between APPS and manual calculation prior to investigating APPS’ predictive capacity for the entire study population, because it would not be feasible to manually calculate the PPS for all 30,726 patients.

Statistical Analysis

For the 75 randomly selected cases of presumed hospital-acquired VTE, the number of cases was chosen by powering our analysis to find a difference in proportion of 20% with 90% power, α = 0.05 (two-sided). We conducted χ² tests on the entire study cohort to determine if there were significant differences in demographics, LOS, and incidence of hospital-acquired VTE by prophylaxis received. For both the pharmacologic and the no pharmacologic prophylaxis groups, we conducted 2-sample Student t tests to determine significant differences in demographics and LOS between patients who experienced a hospital-acquired VTE and those who did not.

For the comparison of our automated calculation to standard clinical practice, we manually calculated the PPS through chart review within the first 2 days of admission on a subsample of 300 random patients. We powered our analysis to detect a difference in mean PPS from 4.86 to 4.36, enough to alter the point value, with 90% power and α = 0.05 (two-sided) and found 300 patients to be comfortably above the required sample size. We compared APPS and manual calculation in the 300-patient cohort using: 2-sample Student t tests to compare mean scores, χ² tests to compare the frequency with which criteria were positive, and receiver operating characteristic (ROC) curves to determine capacity to predict a hospital-acquired VTE event. Pearson’s correlation was also completed to assess score agreement between APPS and manual calculation on a per-patient basis. After comparing automated calculation of APPS to manual chart review on the same 300 patients, we used APPS to calculate scores for the entire study cohort (n = 30,726). We calculated the mean of APPS by prophylaxis group and whether hospital-acquired VTE had occurred. We analyzed APPS’ ROC curve statistics by prophylaxis group to determine its overall predictive capacity in our study population. Lastly, we computed the time required to calculate APPS per patient. Statistical analyses were conducted using SPSS Statistics (IBM, Armonk, New York) and Python 2.7 (Python Software Foundation, Beaverton, Oregon); 95% confidence intervals (CI) and (SD) were reported when appropriate.

Among the 30,726 unique patients in our entire cohort (all patients admitted during the time period who met the study criteria), we found 6574 (21.4%) on pharmacologic (with or without mechanical) prophylaxis, 13,511 (44.0%) on mechanical only, and 10,641 (34.6%) on no prophylaxis. χ² tests found no significant differences in demographics, LOS, or incidence of hospital-acquired VTE between the patients who received mechanical prophylaxis only and those who received no prophylaxis (Table 1). Similarly, there were no differences in these characteristics in patients receiving pharmacologic prophylaxis with or without the addition of mechanical prophylaxis. Designation of prophylaxis group by manual chart review vs. our automated process was found to agree in categorization for 39/40 (97.5%) sampled encounters. When comparing the cohort that received pharmacologic prophylaxis against the cohort that did not, there were significant differences in racial distribution, sex, BMI, and average LOS as shown in Table 1. Those who received pharmacologic prophylaxis were found to be significantly older than those who did not (62.7 years versus 53.2 years, P < 0.001), more likely to be male (50.6% vs, 42.4%, P < 0.001), more likely to have hospital-acquired VTE (2.2% vs. 0.5%, P < 0.001), and to have a shorter LOS (7.1 days vs. 9.8, P < 0.001).

Within the cohort group receiving pharmacologic prophylaxis (n = 6574), hospital-acquired VTE occurred in patients who were significantly younger (58.2 years vs. 62.8 years, P = 0.003) with a greater LOS (23.8 days vs. 6.7, P < 0.001) than those without. Within the group receiving no pharmacologic prophylaxis (n = 24,152), hospital-acquired VTE occurred in patients who were significantly older (57.1 years vs. 53.2 years, P = 0.014) with more than twice the LOS (20.2 days vs. 9.7 days, P < 0.001) compared to those without. Sixty-six of 75 (88%) randomly selected patients in which new VTE was identified by the automated electronic query had this diagnosis confirmed during manual chart review.

As shown in Table 2, automated calculation on a subsample of 300 randomly selected patients using APPS had a mean of 5.5 (SD, 2.9) while manual calculation of the original PPS on the same patients had a mean of 5.1 (SD, 2.6). There was no significant difference in mean between manual calculation and APPS (P = 0.073). There were, however, significant differences in how often individual criteria were considered present. The largest contributors to the difference in scores between APPS and manual calculation were “prior VTE” (positive, 16% vs. 8.3%, respectively) and “reduced mobility” (positive, 74.3% vs. 66%, respectively) as shown in Table 2. In the subsample, there were a total of 6 (2.0%) hospital-acquired VTE events. APPS’ automated calculation had an AUC = 0.79 (CI, 0.63-0.95) that was significant (P = 0.016) with a cutoff value of 5. Chart review’s manual calculation of the PPS had an AUC = 0.76 (CI 0.61-0.91) that was also significant (P = 0.029).

Distribution of Patient Characteristics in Cohort

Automated calculation of APPS using EHR data from prior encounters and the first 4 hours of admission was predictive of in-hospital VTE. APPS performed as well as traditional manual score calculation of the PPS. It was able to do so with no physician input, significantly lessening the burden of calculation and potentially increasing frequency of data-driven VTE risk assessment.

While automated calculation of certain scores is becoming more common, risk calculators that require data beyond vital signs and lab results have lagged,^16-19 in part because of uncertainty about 2 issues. The first is whether EHR data accurately represent the current clinical picture. The second is if a machine-interpretable algorithm to determine a clinical status (eg, “active cancer”) would be similar to a doctor’s perception of that same concept. We attempted to better understand these 2 challenges through developing APPS. Concerning accuracy, EHR data correctly represent the clinical scenario: designations of VTEP and hospital-acquired VTE were accurate in approximately 90% of reviewed cases. Regarding the second concern, when comparing APPS to manual calculation, we found significant differences (P < 0.001) in how often 8 of the 11 criteria were positive, yet no significant difference in overall score and similar predictive capacity. Manual calculation appeared more likely to find data in the index encounter or in structured data. For example, “active cancer” may be documented only in a physician’s note, easily accounted for during a physician’s calculation but missed by APPS looking only for structured data. In contrast, automated calculation found historic criteria, such as “prior VTE” or “known thrombophilic condition,” positive more often. If the patient is being admitted for a problem unrelated to blood clots, the physician may have little time or interest to look through hundreds of EHR documents to discover a 2-year-old VTE. As patients’ records become larger and denser, more historic data can become buried and forgotten. While the 2 scores differ on individual criteria, they are similarly predictive and able to bifurcate the at-risk population to those who should and should not receive pharmacologic prophylaxis.

Disclosures

Dr. Auerbach was supported by NHLBI K24HL098372 during the period of this study. Dr. Khanna, who is an implementation scientist at the University of California San Francisco Center for Digital Health Innovation, is the principal inventor of CareWeb, and may benefit financially from its commercialization. The other authors report no financial conflicts of interest.

Hospital-acquired venous thromboembolism (VTE) continues to be a critical quality challenge for U.S. hospitals,¹ and high-risk patients are often not adequately prophylaxed. Use of VTE prophylaxis (VTEP) varies as widely as 26% to 85% of patients in various studies, as does patient outcomes and care expenditures.^2-6 The 9th edition of the American College of Chest Physicians (CHEST) guidelines⁷ recommend the Padua Prediction Score (PPS) to select individual patients who may be at high risk for venous thromboembolism (VTE) and could benefit from thromboprophylaxis. Use of the manually calculated PPS to select patients for thromboprophylaxis has been shown to help decrease 30-day and 90-day mortality associated with VTE events after hospitalization to medical services.⁸ However, the PPS requires time-consuming manual calculation by a provider, who may be focused on more immediate aspects of patient care and several other risk scores competing for his attention, potentially decreasing its use.

Other risk scores that use only discrete scalar data, such as vital signs and lab results to predict early recognition of sepsis, have been successfully automated and implemented within electronic health records (EHRs).^9-11 Successful automation of scores requiring input of diagnoses, recent medical events, and current clinical status such as the PPS remains difficult.¹² Data representing these characteristics are more prone to error, and harder to translate clearly into a single data field than discrete elements like heart rate, potentially impacting validity of the calculated result.¹³ To improve usage of guideline based VTE risk assessment and decrease physician burden, we developed an algorithm called Automated Padua Prediction Score (APPS) that automatically calculates the PPS using only EHR data available within prior encounters and the first 4 hours of admission, a similar timeframe to when admitting providers would be entering orders. Our goal was to assess if an automatically calculated version of the PPS, a score that depends on criteria more complex than vital signs and labs, would accurately assess risk for hospital-acquired VTE when compared to traditional manual calculation of the Padua Prediction Score by a provider.

Site Description and Ethics

The study was conducted at University of California, San Francisco Medical Center, a 790-bed academic hospital; its Institutional Review Board approved the study and collection of data via chart review. Handling of patient information complied with the Health Insurance Portability and Accountability Act of 1996.

Patient Inclusion

Adult patients admitted to a medical or surgical service between July 1, 2012 and April 1, 2014 were included in the study if they were candidates for VTEP, defined as: length of stay (LOS) greater than 2 days, not on hospice care, not pregnant at admission, no present on admission VTE diagnosis, no known contraindications to prophylaxis (eg, gastrointestinal bleed), and were not receiving therapeutic doses of warfarin, low molecular weight heparins, heparin, or novel anticoagulants prior to admission.

Data Sources

Clinical variables were extracted from the EHR’s enterprise data warehouse (EDW) by SQL Server query (Microsoft, Redmond, Washington) and deposited in a secure database. Chart review was conducted by a trained researcher (Mr. Jacolbia) using the EHR and a standardized protocol. Findings were recorded using REDCap (REDCap Consortium, Vanderbilt University, Nashville, Tennessee). The specific ICD-9, procedure, and lab codes used to determine each criterion of APPS are available in the Appendix.

Creation of the Automated Padua Prediction Score (APPS)

We developed APPS from the original 11 criteria that comprise the Padua Prediction Score: active cancer, previous VTE (excluding superficial vein thrombosis), reduced mobility, known thrombophilic condition, recent (1 month or less) trauma and/or surgery, age 70 years or older, heart and/or respiratory failure, acute myocardial infarction and/or ischemic stroke, acute infection and/or rheumatologic disorder, body mass index (BMI) 30 or higher, and ongoing hormonal treatment.¹³ APPS has the same scoring methodology as PPS: criteria are weighted from 1 to 3 points and summed with a maximum score of 20, representing highest risk of VTE. To automate the score calculation from data routinely available in the EHR, APPS checks pre-selected structured data fields for specific values within laboratory results, orders, nursing flowsheets and claims. Claims data included all ICD-9 and procedure codes used for billing purposes. If any of the predetermined data elements are found, then the specific criterion is considered positive; otherwise, it is scored as negative. The creators of the PPS were consulted in the generation of these data queries to replicate the original standards for deeming a criterion positive. The automated calculation required no use of natural language processing.

Characterization of Study Population

We recorded patient demographics (age, race, gender, BMI), LOS, and rate of hospital-acquired VTE. These patients were separated into 2 cohorts determined by the VTE prophylaxis they received. The risk profile of patients who received pharmacologic prophylaxis was hypothesized to be inherently different from those who had not. To evaluate APPS within this heterogeneous cohort, patients were divided into 2 major categories: pharmacologic vs. no pharmacologic prophylaxis. If they had a completed order or medication administration record on the institution’s approved formulary for pharmacologic VTEP, they were considered to have received pharmacologic prophylaxis. If they had only a completed order for usage of mechanical prophylaxis (sequential compression devices) or no evidence of any form of VTEP, they were considered to have received no pharmacologic prophylaxis. Patients with evidence of both pharmacologic and mechanical were placed in the pharmacologic prophylaxis group. To ensure that automated designation of prophylaxis group was accurate, we reviewed 40 randomly chosen charts because prior researchers were able to achieve sensitivity and specificity greater than 90% with that sample size.¹⁴

The primary outcome of hospital-acquired VTE was defined as an ICD-9 code for VTE (specific codes are found in the Appendix) paired with a “present on admission = no” flag on that encounter’s hospital billing data, abstracted from the EDW. A previous study at this institution used the same methodology and found 212/226 (94%) of patients with a VTE ICD-9 code on claim had evidence of a hospital-acquired VTE event upon chart review.¹⁴ Chart review was also completed to ensure that the primary outcome of newly discovered hospital-acquired VTE was differentiated from chronic VTE or history of VTE. Theoretically, ICD-9 codes and other data elements treat chronic VTE, history of VTE, and hospital-acquired VTE as distinct diagnoses, but it was unclear if this was true in our dataset. For 75 randomly selected cases of presumed hospital-acquired VTE, charts were reviewed for evidence that confirmed newly found VTE during that encounter.

Validation of APPS through Comparison to Manual Calculation of the Original PPS

To compare our automated calculation to standard clinical practice, we manually calculated the PPS through chart review within the first 2 days of admission on 300 random patients, a subsample of the entire study cohort. The largest study we could find had manually calculated the PPS of 1,080 hospitalized patients with a mean PPS of 4.86 (standard deviation [SD], 2.26).¹⁵ One researcher (Mr. Jacolbia) accessed the EHR with all patient information available to physicians, including admission notes, orders, labs, flowsheets, past medical history, and all prior encounters to calculate and record the PPS. To limit potential score bias, 2 authors (Drs. Elias and Davies) assessed 30 randomly selected charts from the cohort of 300. The standardized chart review protocol mimicked a physician’s approach to determine if a patient met a criterion, such as concluding if he/she had active cancer by examining medication lists for chemotherapy, procedure notes for radiation, and recent diagnoses on problem lists. After the original PPS was manually calculated, APPS was automatically calculated for the same 300 patients. We intended to characterize similarities and differences between APPS and manual calculation prior to investigating APPS’ predictive capacity for the entire study population, because it would not be feasible to manually calculate the PPS for all 30,726 patients.

Statistical Analysis

For the 75 randomly selected cases of presumed hospital-acquired VTE, the number of cases was chosen by powering our analysis to find a difference in proportion of 20% with 90% power, α = 0.05 (two-sided). We conducted χ² tests on the entire study cohort to determine if there were significant differences in demographics, LOS, and incidence of hospital-acquired VTE by prophylaxis received. For both the pharmacologic and the no pharmacologic prophylaxis groups, we conducted 2-sample Student t tests to determine significant differences in demographics and LOS between patients who experienced a hospital-acquired VTE and those who did not.

For the comparison of our automated calculation to standard clinical practice, we manually calculated the PPS through chart review within the first 2 days of admission on a subsample of 300 random patients. We powered our analysis to detect a difference in mean PPS from 4.86 to 4.36, enough to alter the point value, with 90% power and α = 0.05 (two-sided) and found 300 patients to be comfortably above the required sample size. We compared APPS and manual calculation in the 300-patient cohort using: 2-sample Student t tests to compare mean scores, χ² tests to compare the frequency with which criteria were positive, and receiver operating characteristic (ROC) curves to determine capacity to predict a hospital-acquired VTE event. Pearson’s correlation was also completed to assess score agreement between APPS and manual calculation on a per-patient basis. After comparing automated calculation of APPS to manual chart review on the same 300 patients, we used APPS to calculate scores for the entire study cohort (n = 30,726). We calculated the mean of APPS by prophylaxis group and whether hospital-acquired VTE had occurred. We analyzed APPS’ ROC curve statistics by prophylaxis group to determine its overall predictive capacity in our study population. Lastly, we computed the time required to calculate APPS per patient. Statistical analyses were conducted using SPSS Statistics (IBM, Armonk, New York) and Python 2.7 (Python Software Foundation, Beaverton, Oregon); 95% confidence intervals (CI) and (SD) were reported when appropriate.

Among the 30,726 unique patients in our entire cohort (all patients admitted during the time period who met the study criteria), we found 6574 (21.4%) on pharmacologic (with or without mechanical) prophylaxis, 13,511 (44.0%) on mechanical only, and 10,641 (34.6%) on no prophylaxis. χ² tests found no significant differences in demographics, LOS, or incidence of hospital-acquired VTE between the patients who received mechanical prophylaxis only and those who received no prophylaxis (Table 1). Similarly, there were no differences in these characteristics in patients receiving pharmacologic prophylaxis with or without the addition of mechanical prophylaxis. Designation of prophylaxis group by manual chart review vs. our automated process was found to agree in categorization for 39/40 (97.5%) sampled encounters. When comparing the cohort that received pharmacologic prophylaxis against the cohort that did not, there were significant differences in racial distribution, sex, BMI, and average LOS as shown in Table 1. Those who received pharmacologic prophylaxis were found to be significantly older than those who did not (62.7 years versus 53.2 years, P < 0.001), more likely to be male (50.6% vs, 42.4%, P < 0.001), more likely to have hospital-acquired VTE (2.2% vs. 0.5%, P < 0.001), and to have a shorter LOS (7.1 days vs. 9.8, P < 0.001).

Within the cohort group receiving pharmacologic prophylaxis (n = 6574), hospital-acquired VTE occurred in patients who were significantly younger (58.2 years vs. 62.8 years, P = 0.003) with a greater LOS (23.8 days vs. 6.7, P < 0.001) than those without. Within the group receiving no pharmacologic prophylaxis (n = 24,152), hospital-acquired VTE occurred in patients who were significantly older (57.1 years vs. 53.2 years, P = 0.014) with more than twice the LOS (20.2 days vs. 9.7 days, P < 0.001) compared to those without. Sixty-six of 75 (88%) randomly selected patients in which new VTE was identified by the automated electronic query had this diagnosis confirmed during manual chart review.

As shown in Table 2, automated calculation on a subsample of 300 randomly selected patients using APPS had a mean of 5.5 (SD, 2.9) while manual calculation of the original PPS on the same patients had a mean of 5.1 (SD, 2.6). There was no significant difference in mean between manual calculation and APPS (P = 0.073). There were, however, significant differences in how often individual criteria were considered present. The largest contributors to the difference in scores between APPS and manual calculation were “prior VTE” (positive, 16% vs. 8.3%, respectively) and “reduced mobility” (positive, 74.3% vs. 66%, respectively) as shown in Table 2. In the subsample, there were a total of 6 (2.0%) hospital-acquired VTE events. APPS’ automated calculation had an AUC = 0.79 (CI, 0.63-0.95) that was significant (P = 0.016) with a cutoff value of 5. Chart review’s manual calculation of the PPS had an AUC = 0.76 (CI 0.61-0.91) that was also significant (P = 0.029).

Distribution of Patient Characteristics in Cohort

Automated calculation of APPS using EHR data from prior encounters and the first 4 hours of admission was predictive of in-hospital VTE. APPS performed as well as traditional manual score calculation of the PPS. It was able to do so with no physician input, significantly lessening the burden of calculation and potentially increasing frequency of data-driven VTE risk assessment.

While automated calculation of certain scores is becoming more common, risk calculators that require data beyond vital signs and lab results have lagged,^16-19 in part because of uncertainty about 2 issues. The first is whether EHR data accurately represent the current clinical picture. The second is if a machine-interpretable algorithm to determine a clinical status (eg, “active cancer”) would be similar to a doctor’s perception of that same concept. We attempted to better understand these 2 challenges through developing APPS. Concerning accuracy, EHR data correctly represent the clinical scenario: designations of VTEP and hospital-acquired VTE were accurate in approximately 90% of reviewed cases. Regarding the second concern, when comparing APPS to manual calculation, we found significant differences (P < 0.001) in how often 8 of the 11 criteria were positive, yet no significant difference in overall score and similar predictive capacity. Manual calculation appeared more likely to find data in the index encounter or in structured data. For example, “active cancer” may be documented only in a physician’s note, easily accounted for during a physician’s calculation but missed by APPS looking only for structured data. In contrast, automated calculation found historic criteria, such as “prior VTE” or “known thrombophilic condition,” positive more often. If the patient is being admitted for a problem unrelated to blood clots, the physician may have little time or interest to look through hundreds of EHR documents to discover a 2-year-old VTE. As patients’ records become larger and denser, more historic data can become buried and forgotten. While the 2 scores differ on individual criteria, they are similarly predictive and able to bifurcate the at-risk population to those who should and should not receive pharmacologic prophylaxis.

Disclosures

Dr. Auerbach was supported by NHLBI K24HL098372 during the period of this study. Dr. Khanna, who is an implementation scientist at the University of California San Francisco Center for Digital Health Innovation, is the principal inventor of CareWeb, and may benefit financially from its commercialization. The other authors report no financial conflicts of interest.

Hospital-acquired venous thromboembolism (VTE) continues to be a critical quality challenge for U.S. hospitals,¹ and high-risk patients are often not adequately prophylaxed. Use of VTE prophylaxis (VTEP) varies as widely as 26% to 85% of patients in various studies, as does patient outcomes and care expenditures.^2-6 The 9th edition of the American College of Chest Physicians (CHEST) guidelines⁷ recommend the Padua Prediction Score (PPS) to select individual patients who may be at high risk for venous thromboembolism (VTE) and could benefit from thromboprophylaxis. Use of the manually calculated PPS to select patients for thromboprophylaxis has been shown to help decrease 30-day and 90-day mortality associated with VTE events after hospitalization to medical services.⁸ However, the PPS requires time-consuming manual calculation by a provider, who may be focused on more immediate aspects of patient care and several other risk scores competing for his attention, potentially decreasing its use.

Other risk scores that use only discrete scalar data, such as vital signs and lab results to predict early recognition of sepsis, have been successfully automated and implemented within electronic health records (EHRs).^9-11 Successful automation of scores requiring input of diagnoses, recent medical events, and current clinical status such as the PPS remains difficult.¹² Data representing these characteristics are more prone to error, and harder to translate clearly into a single data field than discrete elements like heart rate, potentially impacting validity of the calculated result.¹³ To improve usage of guideline based VTE risk assessment and decrease physician burden, we developed an algorithm called Automated Padua Prediction Score (APPS) that automatically calculates the PPS using only EHR data available within prior encounters and the first 4 hours of admission, a similar timeframe to when admitting providers would be entering orders. Our goal was to assess if an automatically calculated version of the PPS, a score that depends on criteria more complex than vital signs and labs, would accurately assess risk for hospital-acquired VTE when compared to traditional manual calculation of the Padua Prediction Score by a provider.

Site Description and Ethics

The study was conducted at University of California, San Francisco Medical Center, a 790-bed academic hospital; its Institutional Review Board approved the study and collection of data via chart review. Handling of patient information complied with the Health Insurance Portability and Accountability Act of 1996.

Patient Inclusion

Adult patients admitted to a medical or surgical service between July 1, 2012 and April 1, 2014 were included in the study if they were candidates for VTEP, defined as: length of stay (LOS) greater than 2 days, not on hospice care, not pregnant at admission, no present on admission VTE diagnosis, no known contraindications to prophylaxis (eg, gastrointestinal bleed), and were not receiving therapeutic doses of warfarin, low molecular weight heparins, heparin, or novel anticoagulants prior to admission.

Data Sources

Clinical variables were extracted from the EHR’s enterprise data warehouse (EDW) by SQL Server query (Microsoft, Redmond, Washington) and deposited in a secure database. Chart review was conducted by a trained researcher (Mr. Jacolbia) using the EHR and a standardized protocol. Findings were recorded using REDCap (REDCap Consortium, Vanderbilt University, Nashville, Tennessee). The specific ICD-9, procedure, and lab codes used to determine each criterion of APPS are available in the Appendix.

Creation of the Automated Padua Prediction Score (APPS)

We developed APPS from the original 11 criteria that comprise the Padua Prediction Score: active cancer, previous VTE (excluding superficial vein thrombosis), reduced mobility, known thrombophilic condition, recent (1 month or less) trauma and/or surgery, age 70 years or older, heart and/or respiratory failure, acute myocardial infarction and/or ischemic stroke, acute infection and/or rheumatologic disorder, body mass index (BMI) 30 or higher, and ongoing hormonal treatment.¹³ APPS has the same scoring methodology as PPS: criteria are weighted from 1 to 3 points and summed with a maximum score of 20, representing highest risk of VTE. To automate the score calculation from data routinely available in the EHR, APPS checks pre-selected structured data fields for specific values within laboratory results, orders, nursing flowsheets and claims. Claims data included all ICD-9 and procedure codes used for billing purposes. If any of the predetermined data elements are found, then the specific criterion is considered positive; otherwise, it is scored as negative. The creators of the PPS were consulted in the generation of these data queries to replicate the original standards for deeming a criterion positive. The automated calculation required no use of natural language processing.

Characterization of Study Population

We recorded patient demographics (age, race, gender, BMI), LOS, and rate of hospital-acquired VTE. These patients were separated into 2 cohorts determined by the VTE prophylaxis they received. The risk profile of patients who received pharmacologic prophylaxis was hypothesized to be inherently different from those who had not. To evaluate APPS within this heterogeneous cohort, patients were divided into 2 major categories: pharmacologic vs. no pharmacologic prophylaxis. If they had a completed order or medication administration record on the institution’s approved formulary for pharmacologic VTEP, they were considered to have received pharmacologic prophylaxis. If they had only a completed order for usage of mechanical prophylaxis (sequential compression devices) or no evidence of any form of VTEP, they were considered to have received no pharmacologic prophylaxis. Patients with evidence of both pharmacologic and mechanical were placed in the pharmacologic prophylaxis group. To ensure that automated designation of prophylaxis group was accurate, we reviewed 40 randomly chosen charts because prior researchers were able to achieve sensitivity and specificity greater than 90% with that sample size.¹⁴

The primary outcome of hospital-acquired VTE was defined as an ICD-9 code for VTE (specific codes are found in the Appendix) paired with a “present on admission = no” flag on that encounter’s hospital billing data, abstracted from the EDW. A previous study at this institution used the same methodology and found 212/226 (94%) of patients with a VTE ICD-9 code on claim had evidence of a hospital-acquired VTE event upon chart review.¹⁴ Chart review was also completed to ensure that the primary outcome of newly discovered hospital-acquired VTE was differentiated from chronic VTE or history of VTE. Theoretically, ICD-9 codes and other data elements treat chronic VTE, history of VTE, and hospital-acquired VTE as distinct diagnoses, but it was unclear if this was true in our dataset. For 75 randomly selected cases of presumed hospital-acquired VTE, charts were reviewed for evidence that confirmed newly found VTE during that encounter.

Validation of APPS through Comparison to Manual Calculation of the Original PPS

To compare our automated calculation to standard clinical practice, we manually calculated the PPS through chart review within the first 2 days of admission on 300 random patients, a subsample of the entire study cohort. The largest study we could find had manually calculated the PPS of 1,080 hospitalized patients with a mean PPS of 4.86 (standard deviation [SD], 2.26).¹⁵ One researcher (Mr. Jacolbia) accessed the EHR with all patient information available to physicians, including admission notes, orders, labs, flowsheets, past medical history, and all prior encounters to calculate and record the PPS. To limit potential score bias, 2 authors (Drs. Elias and Davies) assessed 30 randomly selected charts from the cohort of 300. The standardized chart review protocol mimicked a physician’s approach to determine if a patient met a criterion, such as concluding if he/she had active cancer by examining medication lists for chemotherapy, procedure notes for radiation, and recent diagnoses on problem lists. After the original PPS was manually calculated, APPS was automatically calculated for the same 300 patients. We intended to characterize similarities and differences between APPS and manual calculation prior to investigating APPS’ predictive capacity for the entire study population, because it would not be feasible to manually calculate the PPS for all 30,726 patients.

Statistical Analysis

For the 75 randomly selected cases of presumed hospital-acquired VTE, the number of cases was chosen by powering our analysis to find a difference in proportion of 20% with 90% power, α = 0.05 (two-sided). We conducted χ² tests on the entire study cohort to determine if there were significant differences in demographics, LOS, and incidence of hospital-acquired VTE by prophylaxis received. For both the pharmacologic and the no pharmacologic prophylaxis groups, we conducted 2-sample Student t tests to determine significant differences in demographics and LOS between patients who experienced a hospital-acquired VTE and those who did not.

For the comparison of our automated calculation to standard clinical practice, we manually calculated the PPS through chart review within the first 2 days of admission on a subsample of 300 random patients. We powered our analysis to detect a difference in mean PPS from 4.86 to 4.36, enough to alter the point value, with 90% power and α = 0.05 (two-sided) and found 300 patients to be comfortably above the required sample size. We compared APPS and manual calculation in the 300-patient cohort using: 2-sample Student t tests to compare mean scores, χ² tests to compare the frequency with which criteria were positive, and receiver operating characteristic (ROC) curves to determine capacity to predict a hospital-acquired VTE event. Pearson’s correlation was also completed to assess score agreement between APPS and manual calculation on a per-patient basis. After comparing automated calculation of APPS to manual chart review on the same 300 patients, we used APPS to calculate scores for the entire study cohort (n = 30,726). We calculated the mean of APPS by prophylaxis group and whether hospital-acquired VTE had occurred. We analyzed APPS’ ROC curve statistics by prophylaxis group to determine its overall predictive capacity in our study population. Lastly, we computed the time required to calculate APPS per patient. Statistical analyses were conducted using SPSS Statistics (IBM, Armonk, New York) and Python 2.7 (Python Software Foundation, Beaverton, Oregon); 95% confidence intervals (CI) and (SD) were reported when appropriate.

Among the 30,726 unique patients in our entire cohort (all patients admitted during the time period who met the study criteria), we found 6574 (21.4%) on pharmacologic (with or without mechanical) prophylaxis, 13,511 (44.0%) on mechanical only, and 10,641 (34.6%) on no prophylaxis. χ² tests found no significant differences in demographics, LOS, or incidence of hospital-acquired VTE between the patients who received mechanical prophylaxis only and those who received no prophylaxis (Table 1). Similarly, there were no differences in these characteristics in patients receiving pharmacologic prophylaxis with or without the addition of mechanical prophylaxis. Designation of prophylaxis group by manual chart review vs. our automated process was found to agree in categorization for 39/40 (97.5%) sampled encounters. When comparing the cohort that received pharmacologic prophylaxis against the cohort that did not, there were significant differences in racial distribution, sex, BMI, and average LOS as shown in Table 1. Those who received pharmacologic prophylaxis were found to be significantly older than those who did not (62.7 years versus 53.2 years, P < 0.001), more likely to be male (50.6% vs, 42.4%, P < 0.001), more likely to have hospital-acquired VTE (2.2% vs. 0.5%, P < 0.001), and to have a shorter LOS (7.1 days vs. 9.8, P < 0.001).

Within the cohort group receiving pharmacologic prophylaxis (n = 6574), hospital-acquired VTE occurred in patients who were significantly younger (58.2 years vs. 62.8 years, P = 0.003) with a greater LOS (23.8 days vs. 6.7, P < 0.001) than those without. Within the group receiving no pharmacologic prophylaxis (n = 24,152), hospital-acquired VTE occurred in patients who were significantly older (57.1 years vs. 53.2 years, P = 0.014) with more than twice the LOS (20.2 days vs. 9.7 days, P < 0.001) compared to those without. Sixty-six of 75 (88%) randomly selected patients in which new VTE was identified by the automated electronic query had this diagnosis confirmed during manual chart review.

As shown in Table 2, automated calculation on a subsample of 300 randomly selected patients using APPS had a mean of 5.5 (SD, 2.9) while manual calculation of the original PPS on the same patients had a mean of 5.1 (SD, 2.6). There was no significant difference in mean between manual calculation and APPS (P = 0.073). There were, however, significant differences in how often individual criteria were considered present. The largest contributors to the difference in scores between APPS and manual calculation were “prior VTE” (positive, 16% vs. 8.3%, respectively) and “reduced mobility” (positive, 74.3% vs. 66%, respectively) as shown in Table 2. In the subsample, there were a total of 6 (2.0%) hospital-acquired VTE events. APPS’ automated calculation had an AUC = 0.79 (CI, 0.63-0.95) that was significant (P = 0.016) with a cutoff value of 5. Chart review’s manual calculation of the PPS had an AUC = 0.76 (CI 0.61-0.91) that was also significant (P = 0.029).

Distribution of Patient Characteristics in Cohort

Automated calculation of APPS using EHR data from prior encounters and the first 4 hours of admission was predictive of in-hospital VTE. APPS performed as well as traditional manual score calculation of the PPS. It was able to do so with no physician input, significantly lessening the burden of calculation and potentially increasing frequency of data-driven VTE risk assessment.

While automated calculation of certain scores is becoming more common, risk calculators that require data beyond vital signs and lab results have lagged,^16-19 in part because of uncertainty about 2 issues. The first is whether EHR data accurately represent the current clinical picture. The second is if a machine-interpretable algorithm to determine a clinical status (eg, “active cancer”) would be similar to a doctor’s perception of that same concept. We attempted to better understand these 2 challenges through developing APPS. Concerning accuracy, EHR data correctly represent the clinical scenario: designations of VTEP and hospital-acquired VTE were accurate in approximately 90% of reviewed cases. Regarding the second concern, when comparing APPS to manual calculation, we found significant differences (P < 0.001) in how often 8 of the 11 criteria were positive, yet no significant difference in overall score and similar predictive capacity. Manual calculation appeared more likely to find data in the index encounter or in structured data. For example, “active cancer” may be documented only in a physician’s note, easily accounted for during a physician’s calculation but missed by APPS looking only for structured data. In contrast, automated calculation found historic criteria, such as “prior VTE” or “known thrombophilic condition,” positive more often. If the patient is being admitted for a problem unrelated to blood clots, the physician may have little time or interest to look through hundreds of EHR documents to discover a 2-year-old VTE. As patients’ records become larger and denser, more historic data can become buried and forgotten. While the 2 scores differ on individual criteria, they are similarly predictive and able to bifurcate the at-risk population to those who should and should not receive pharmacologic prophylaxis.

Disclosures

Dr. Auerbach was supported by NHLBI K24HL098372 during the period of this study. Dr. Khanna, who is an implementation scientist at the University of California San Francisco Center for Digital Health Innovation, is the principal inventor of CareWeb, and may benefit financially from its commercialization. The other authors report no financial conflicts of interest.

Approximately 20% of hospitalized Medicare beneficiaries in the U.S. are discharged to skilled nursing facilities (SNFs) for post-acute care,^1,2 and 23.5% of these patients are readmitted within 30 days.³ Because hospital readmissions are costly and associated with worse outcomes,^4,5 30-day readmission rates are considered a quality indicator,⁶ and there are financial penalties for hospitals with higher than expected rates.⁷ As a result, hospitals invest substantial resources in programs to reduce readmissions.^8-10 The SNFs represent an attractive target for readmission reduction efforts, since SNFs contribute a disproportionate share of readmissions.^3,4 Because SNF patients are in a monitored environment with high medication adherence, risk factors for readmission likely differ between patients discharged to SNFs and those sent home. For example, 1 study showed that among heart failure patients with cognitive impairment, those discharged to SNFs had lower readmissions during the first 20 days, likely due to better medication adherence.¹¹ Patients discharged to SNFs generally have more complex illnesses, lower functional status, and higher 1-year mortality than patients discharged to the community.^12,13 Despite this, SNF patients might have infrequent contact with physicians. Federal regulations require only that patients discharged to SNFs need to be seen within 30 days and then at least once every 30 days thereafter.¹⁴ According to the 2014 Office of Inspector General report, one-third of Medicare beneficiaries in SNFs experience adverse events from substandard treatment, inadequate resident monitoring and failure or delay of necessary care, most of which are thought to be preventable.¹⁵

To address this issue, the Cleveland Clinic developed a program called “Connected Care SNF,” in which hospital-employed physicians and advanced practice professionals visit patients in selected SNFs 4 to 5 times per week, for the purpose of reducing preventable readmissions. The aim of this study was to assess whether the program reduced 30-day readmissions, and to identify which patients benefited most from the program.

Setting and Intervention

The Cleveland Clinic main campus is a tertiary academic medical center with 1400 beds and approximately 50,000 admissions per year. In late 2012, the Cleveland Clinic implemented the Connected Care SNF program, wherein Cleveland Clinic physicians regularly visited patients who were discharged from the Cleveland Clinic main campus to 7 regional SNFs. Beginning in December 2012, these 7 high-volume referral SNFs that were not part of the Cleveland Clinic Health System (CCHS) agreed to participate in the program, which focused on reducing avoidable hospital readmissions and delivering quality care (Table 1). The Connected Care team, comprised of 2 geriatricians (1 of whom was also a palliative medicine specialist), 1 internist, 1 family physician, and 5 advanced practice professionals (nurse practitioners and physician assistants), provided medical services at the participating SNFs. These providers aimed to see patients 4 to 5 times per week, were available on site during working hours, and provided telephone coverage at nights and on weekends. All providers had access to hospital electronic medical records and could communicate with the discharging physician and with specialists familiar with the patient as needed. Prior to the admission, providers were informed about patient arrival and, at the time of admission to the SNF, providers reviewed medications and discussed goals of care with patients and their families. In the SNF, providers worked closely with staff members to deliver medications and timely treatment. They also met monthly with multidisciplinary teams for continuous quality improvement and to review outcomes. Patients at Connected Care SNFs who had their own physicians, including most long-stay and some short-stay residents, did not receive the Connected Care intervention. They constituted less than 10% of the patients discharged from Cleveland Clinic main campus.

Study Design and Population

We reviewed administrative and clinical data from a retrospective cohort of patients discharged to SNF from the Cleveland Clinic main campus from January 1, 2011 to December 31, 2014. We included all patients who were discharged to an SNF during the study period. Our main outcome measure was 30-day all-cause readmissions to any hospital in the Cleveland Clinic Health System (CCHS), including the main campus and 8 regional community hospitals. Study patients were followed until January 30, 2015 to capture 30-day readmissions. According to 2012 Medicare data, of CCHS patients who were readmitted within 30 days, 83% of pneumonia, 81% of major joint replacement, 72% of heart failure and 57% of acute myocardial infarction patients were readmitted to a CCHS facility. As the Cleveland Clinic main campus attracts cardiac patients from a 100+-mile radius, they may be more likely to seek care readmission near home and are not reflective of CCHS patients overall. Because we did not have access to readmissions data from non-CCHS hospitals, we excluded patients who were discharged to SNFs beyond a 25-mile radius from the main campus, where they may be more likely to utilize non-CCHS hospitals for acute hospitalization. We also excluded patients discharged to non-CCHS hospital-based SNFs, which may refer readmissions to their own hospital system. Because the Connected Care program began in December 2012, the years 2011-2012 served as the baseline period. The intervention was conducted at 7 SNFs. All other SNFs within the 25-mile radius were included as controls, except for 3 hospital-based SNFs that would be unlikely to admit patients to CCHS. We compared the change in all-cause 30-day readmission rates after implementation of Connected Care, using all patients discharged to SNFs within 25 miles to control for temporal changes in local readmission rates. Discharge to specific SNFs was determined solely by patient choice.

Data Collection

For each patient, we collected the following data that has been shown to be associated with readmissions:^16-18 demographics (age, race, sex, ZIP code), lab values on discharge (hemoglobin and sodium); hemodialysis status; medicine or surgical service; elective surgery or nonelective surgery; details of the index admission index (diagnosis-related group [DRG], Medicare severity-diagnosis-related groups [MS-DRG] weight, primary diagnosis code; principal procedure code; admission date; discharge date, length of stay, and post-acute care provider); and common comorbidities, as listed in Table 2. We also calculated each patient’s HOSPITAL^19,20 score. The HOSPITAL score was developed to predict risk of preventable 30-day readmissions,¹⁹ but it has also been validated to predict 30-day all-cause readmission rates for patients discharged to SNF.²¹ The model contains 7 elements (hemoglobin, oncology service, sodium, procedure, index type, admissions within the last year, length of stay) (supplemental Table).Patients with a high score (7 or higher) have a 41% chance of readmission, while those with a low score (4 or lower) have only a 15% chance. ²¹ We assessed all cause 30-day readmission status from CCHS administrative data. Observation patients and outpatient same-day surgeries were not considered to be admissions. For patients with multiple admissions, each admission was counted as a separate index hospitalization. Cleveland Clinic’s Institutional Review Board approved the study.

Statistical Analysis

For the 7 intervention SNFs, patient characteristics were summarized as means and standard deviations or frequencies and percentages for the periods of 2011-2012 and 2013-2014, respectively, and the 2 periods were compared using the Student t test or χ² test as appropriate.

Mixed-effects logistic regression models were used to model 30-day readmission rates. Since the intervention was implemented in the last quarter of 2012, we examined the difference in readmission rates before and after that time point. The model included the following fixed effects: SNF type (intervention or usual care), time points (quarters of 2011-2014), whether the time is pre- or postintervention (binary), and the 3-way interaction between SNF type, pre- or postintervention and time points, and patient characteristics. The model also contained a Gaussian random effect at the SNF level to account for possible correlations among the outcomes of patients from the same SNF. For each quarter, the mean adjusted readmission rates of 2 types of SNFs were calculated from the fitted mixed models and plotted over time. Furthermore, we compared the mean readmission rates of the 2 groups in the pre- and postintervention periods. Subgroup analyses were performed for medical and surgical patients, and for patients in the low, intermediate and high HOSPITAL score groups.

All analyses were performed using RStudio (Boston, Massachusetts). Statistical significance was established with 2-sided P values less than 0.05.

We identified 119 SNFs within a 25-mile radius of the hospital. Of these, 6 did not receive any referrals. Three non-CCHS hospital-based SNFs were excluded, leaving a total of 110 SNFs in the study sample: 7 intervention SNFs and 103 usual-care SNFs. Between January 2011 and December 2014, there were 23,408 SNF discharges from Cleveland Clinic main campus, including 13,544 who were discharged to study SNFs (Supplemental Figure). Of these, 3334 were discharged to 7 intervention SNFs and 10,210 were discharged to usual care SNFs. Characteristics of patients in both periods appear in Table 2. At baseline, patients in the intervention and control SNFs varied in a number of ways. Patients at intervention SNFs were older (75.6 vs. 70.2 years; P < 0.001), more likely to be African American (45.5% vs. 35.9%; P < 0.001), female (61% vs. 55.4%; P < 0.001) and to be insured by Medicare (85.2% vs. 71.4%; P < 0.001). Both groups had similar proportions of patients with high, intermediate, and low readmission risk as measured by HOSPITAL score. Compared to the 2011-2012 pre-intervention period, during the 2013-2014 intervention period, there were more surgeries (34.3% vs. 41.9%; P < 0.001), more elective surgeries (21.8% vs. 25.5%; P = 0.01), fewer medical patients (65.7% vs. 58.1%; P < 0.001), and an increase in comorbidities, including myocardial infarction, peripheral vascular disease, and liver disease (Table 2).

Table 3 shows adjusted 30-day readmissions rates, before and during the intervention period at the intervention and usual care SNFs. Compared to the pre-intervention period, 30-day all-cause adjusted readmission rates declined in the intervention SNFs (28.1% to 21.7%, P < 0.001), while it increased slightly at control sites (27.1% to 28.5%, P < 0.001). The Figure shows the adjusted 30-day readmission rates by quarter throughout the study period.

In this retrospective study of 4 years of discharges to 110 SNFs, we report on the impact of a Connected Care program, in which a physician visited patients on admission to the SNF and 4 to 5 times per week during their stay. Introduction of the program was followed by a 6.8% absolute reduction in all-cause 30-day readmission rates compared to usual care. The absolute reductions ranged from 4.6% for patients at low risk for readmission to 9.1% for patients at high risk, and medical patients benefited more than surgical patients.

Most studies of interventions to reduce hospital readmissions have focused on patients discharged to the community setting.^7-9 Interventions have centered on discharge planning, medication reconciliation, and close follow-up to assess for medication adherence and early signs of deterioration. Because patients in SNFs have their medications administered by staff and are under frequent surveillance, such interventions are unlikely to be helpful in this population. We found no studies that focus on short-stay or skilled patients discharged to SNF. Two studies have demonstrated that interventions can reduce hospitalization from nursing homes.^22,23 Neither study included readmissions. The Evercare model consisted of nurse practitioners providing active primary care services within the nursing home, as well as offering incentive payments to nursing homes for not hospitalizing patients.²² During a 2-year period, long term residents who enrolled in Evercare had an almost 50% reduction in incident hospitalizations compared to those who did not.²² INTERACT II was a quality improvement intervention that provided tools, education, and strategies to help identify and manage acute conditions proactively.²³ In 25 nursing homes employing INTERACT II, there was a 17% reduction in self-reported hospital admissions during the 6-month project, with higher rates of reduction among nursing homes rated as more engaged in the process.²³ Although nursing homes may serve some short-stay or skilled patients, they generally serve long-term populations, and studies have shown that short-stay patients are at higher risk for 30-day readmissions.²⁴

There are a number of reasons that short-term SNF patients are at higher risk for readmission. Although prior to admission, they were considered hospital level of care and received a physician visit daily, on transfer to the SNF, relatively little medical care is available. Current federal regulations regarding physician services at a SNF require the resident to be seen by a physician at least once every 30 days for the first 90 days after admission, and at least once every 60 days thereafter.²⁵

The Connected Care program physicians provided a smooth transition of care from hospital to SNF as well as frequent reassessment. Physicians were alerted prior to hospital discharge and performed an initial comprehensive visit generally on the day of admission to the SNF and always within 48 hours. The initial evaluation is important because miscommunication during the handoff from hospital to SNF may result in incorrect medication regimens or inaccurate assessments. By performing prompt medication reconciliation and periodic reassessments of a patient’s medical condition, the Connected Care providers recreate some of the essential elements of successful outpatient readmissions prevention programs.

They also worked together with each SNF’s interdisciplinary team to deliver quality care. There were monthly meetings at each participating Connected Care SNF. Physicians reviewed monthly 30-day readmissions and performed root-cause analysis. When they discovered challenges to timely medication and treatment delivery during daily rounds, they provided in-services to SNF nurses.

In addition, Connected Care providers discussed goals of care—something that is often overlooked on admission to a SNF. This is particularly important because patients with chronic illnesses who are discharged to SNF often have poor prognoses. For example, Medicare patients with heart failure who are discharged to SNFs have 1-year mortality in excess of 50%.¹³ By implementing a plan of care consistent with patient and family goals, inappropriate readmissions for terminal patients may be avoided.

Reducing readmissions is important for hospitals because under the Hospital Readmissions Reduction Program, hospitals now face substantial penalties for higher than expected readmissions rates. Hospitals involved in bundled payments or other total cost-of-care arrangements have additional incentive to avoid readmissions. Beginning in 2019, SNFs will also receive incentive payments based on their 30-day all-cause hospital readmissions as part of the Skilled Nursing Facility Value-Based Purchasing program.²⁵ The Connected Care model offers 1 means of achieving this goal through partnership between hospitals and SNFs.

Our study has several limitations. First, our study was observational in nature, so the observed reduction in readmissions could have been due to temporal trends unrelated to the intervention. However, no significant reduction was noted during the same time period in other area SNFs. There was also little change in the characteristics of patients admitted to the intervention SNFs. Importantly, the HOSPITAL score, which can predict 30-day readmission rates,²⁰ did not change throughout the study period. Second, the results reflect patients discharged from a single hospital and may not be generalizable to other geographic areas. However, because the program included 7 SNFs, we believe it could be reproduced in other settings. Third, our readmissions measure included only those patients who returned to a CCHS facility. Although we may have missed some readmissions to other hospital systems, such leakage is uncommon—more than 80% of CCHS patients are readmitted to CCHS facilities—and would be unlikely to differ across the short duration of the study. Finally, at the intervention SNFs, most long-stay and some short-stay residents did not receive the Connected Care intervention because they were cared for by their own physicians who did not participate in Connected Care. Had these patients’ readmissions been excluded from our results, the intervention might appear even more effective.

CONCLUSION

A Connected Care intervention reduced 30-day readmission rates among patients discharged to SNFs from a tertiary academic center. While all subgroups had substantial reductions in readmissions following the implementation of the intervention, patients who are at the highest risk of readmission benefited the most. Further study is necessary to know whether Connected Care can be reproduced in other health care systems and whether it reduces overall costs.

Acknowledgments

The authors would like to thank Michael Felver, MD, and teams for their clinical care of patients; Michael Felver, MD, William Zafirau, MD, Dan Blechschmid, MHA, and Kathy Brezine, and Seth Vilensky, MBA, for their administrative support; and Brad Souder, MPT, for assistance with data collection.

Disclosure

Nothing to report.

Approximately 20% of hospitalized Medicare beneficiaries in the U.S. are discharged to skilled nursing facilities (SNFs) for post-acute care,^1,2 and 23.5% of these patients are readmitted within 30 days.³ Because hospital readmissions are costly and associated with worse outcomes,^4,5 30-day readmission rates are considered a quality indicator,⁶ and there are financial penalties for hospitals with higher than expected rates.⁷ As a result, hospitals invest substantial resources in programs to reduce readmissions.^8-10 The SNFs represent an attractive target for readmission reduction efforts, since SNFs contribute a disproportionate share of readmissions.^3,4 Because SNF patients are in a monitored environment with high medication adherence, risk factors for readmission likely differ between patients discharged to SNFs and those sent home. For example, 1 study showed that among heart failure patients with cognitive impairment, those discharged to SNFs had lower readmissions during the first 20 days, likely due to better medication adherence.¹¹ Patients discharged to SNFs generally have more complex illnesses, lower functional status, and higher 1-year mortality than patients discharged to the community.^12,13 Despite this, SNF patients might have infrequent contact with physicians. Federal regulations require only that patients discharged to SNFs need to be seen within 30 days and then at least once every 30 days thereafter.¹⁴ According to the 2014 Office of Inspector General report, one-third of Medicare beneficiaries in SNFs experience adverse events from substandard treatment, inadequate resident monitoring and failure or delay of necessary care, most of which are thought to be preventable.¹⁵

To address this issue, the Cleveland Clinic developed a program called “Connected Care SNF,” in which hospital-employed physicians and advanced practice professionals visit patients in selected SNFs 4 to 5 times per week, for the purpose of reducing preventable readmissions. The aim of this study was to assess whether the program reduced 30-day readmissions, and to identify which patients benefited most from the program.

Setting and Intervention

The Cleveland Clinic main campus is a tertiary academic medical center with 1400 beds and approximately 50,000 admissions per year. In late 2012, the Cleveland Clinic implemented the Connected Care SNF program, wherein Cleveland Clinic physicians regularly visited patients who were discharged from the Cleveland Clinic main campus to 7 regional SNFs. Beginning in December 2012, these 7 high-volume referral SNFs that were not part of the Cleveland Clinic Health System (CCHS) agreed to participate in the program, which focused on reducing avoidable hospital readmissions and delivering quality care (Table 1). The Connected Care team, comprised of 2 geriatricians (1 of whom was also a palliative medicine specialist), 1 internist, 1 family physician, and 5 advanced practice professionals (nurse practitioners and physician assistants), provided medical services at the participating SNFs. These providers aimed to see patients 4 to 5 times per week, were available on site during working hours, and provided telephone coverage at nights and on weekends. All providers had access to hospital electronic medical records and could communicate with the discharging physician and with specialists familiar with the patient as needed. Prior to the admission, providers were informed about patient arrival and, at the time of admission to the SNF, providers reviewed medications and discussed goals of care with patients and their families. In the SNF, providers worked closely with staff members to deliver medications and timely treatment. They also met monthly with multidisciplinary teams for continuous quality improvement and to review outcomes. Patients at Connected Care SNFs who had their own physicians, including most long-stay and some short-stay residents, did not receive the Connected Care intervention. They constituted less than 10% of the patients discharged from Cleveland Clinic main campus.

Study Design and Population

We reviewed administrative and clinical data from a retrospective cohort of patients discharged to SNF from the Cleveland Clinic main campus from January 1, 2011 to December 31, 2014. We included all patients who were discharged to an SNF during the study period. Our main outcome measure was 30-day all-cause readmissions to any hospital in the Cleveland Clinic Health System (CCHS), including the main campus and 8 regional community hospitals. Study patients were followed until January 30, 2015 to capture 30-day readmissions. According to 2012 Medicare data, of CCHS patients who were readmitted within 30 days, 83% of pneumonia, 81% of major joint replacement, 72% of heart failure and 57% of acute myocardial infarction patients were readmitted to a CCHS facility. As the Cleveland Clinic main campus attracts cardiac patients from a 100+-mile radius, they may be more likely to seek care readmission near home and are not reflective of CCHS patients overall. Because we did not have access to readmissions data from non-CCHS hospitals, we excluded patients who were discharged to SNFs beyond a 25-mile radius from the main campus, where they may be more likely to utilize non-CCHS hospitals for acute hospitalization. We also excluded patients discharged to non-CCHS hospital-based SNFs, which may refer readmissions to their own hospital system. Because the Connected Care program began in December 2012, the years 2011-2012 served as the baseline period. The intervention was conducted at 7 SNFs. All other SNFs within the 25-mile radius were included as controls, except for 3 hospital-based SNFs that would be unlikely to admit patients to CCHS. We compared the change in all-cause 30-day readmission rates after implementation of Connected Care, using all patients discharged to SNFs within 25 miles to control for temporal changes in local readmission rates. Discharge to specific SNFs was determined solely by patient choice.

Data Collection

For each patient, we collected the following data that has been shown to be associated with readmissions:^16-18 demographics (age, race, sex, ZIP code), lab values on discharge (hemoglobin and sodium); hemodialysis status; medicine or surgical service; elective surgery or nonelective surgery; details of the index admission index (diagnosis-related group [DRG], Medicare severity-diagnosis-related groups [MS-DRG] weight, primary diagnosis code; principal procedure code; admission date; discharge date, length of stay, and post-acute care provider); and common comorbidities, as listed in Table 2. We also calculated each patient’s HOSPITAL^19,20 score. The HOSPITAL score was developed to predict risk of preventable 30-day readmissions,¹⁹ but it has also been validated to predict 30-day all-cause readmission rates for patients discharged to SNF.²¹ The model contains 7 elements (hemoglobin, oncology service, sodium, procedure, index type, admissions within the last year, length of stay) (supplemental Table).Patients with a high score (7 or higher) have a 41% chance of readmission, while those with a low score (4 or lower) have only a 15% chance. ²¹ We assessed all cause 30-day readmission status from CCHS administrative data. Observation patients and outpatient same-day surgeries were not considered to be admissions. For patients with multiple admissions, each admission was counted as a separate index hospitalization. Cleveland Clinic’s Institutional Review Board approved the study.

Statistical Analysis

For the 7 intervention SNFs, patient characteristics were summarized as means and standard deviations or frequencies and percentages for the periods of 2011-2012 and 2013-2014, respectively, and the 2 periods were compared using the Student t test or χ² test as appropriate.

Mixed-effects logistic regression models were used to model 30-day readmission rates. Since the intervention was implemented in the last quarter of 2012, we examined the difference in readmission rates before and after that time point. The model included the following fixed effects: SNF type (intervention or usual care), time points (quarters of 2011-2014), whether the time is pre- or postintervention (binary), and the 3-way interaction between SNF type, pre- or postintervention and time points, and patient characteristics. The model also contained a Gaussian random effect at the SNF level to account for possible correlations among the outcomes of patients from the same SNF. For each quarter, the mean adjusted readmission rates of 2 types of SNFs were calculated from the fitted mixed models and plotted over time. Furthermore, we compared the mean readmission rates of the 2 groups in the pre- and postintervention periods. Subgroup analyses were performed for medical and surgical patients, and for patients in the low, intermediate and high HOSPITAL score groups.

All analyses were performed using RStudio (Boston, Massachusetts). Statistical significance was established with 2-sided P values less than 0.05.

We identified 119 SNFs within a 25-mile radius of the hospital. Of these, 6 did not receive any referrals. Three non-CCHS hospital-based SNFs were excluded, leaving a total of 110 SNFs in the study sample: 7 intervention SNFs and 103 usual-care SNFs. Between January 2011 and December 2014, there were 23,408 SNF discharges from Cleveland Clinic main campus, including 13,544 who were discharged to study SNFs (Supplemental Figure). Of these, 3334 were discharged to 7 intervention SNFs and 10,210 were discharged to usual care SNFs. Characteristics of patients in both periods appear in Table 2. At baseline, patients in the intervention and control SNFs varied in a number of ways. Patients at intervention SNFs were older (75.6 vs. 70.2 years; P < 0.001), more likely to be African American (45.5% vs. 35.9%; P < 0.001), female (61% vs. 55.4%; P < 0.001) and to be insured by Medicare (85.2% vs. 71.4%; P < 0.001). Both groups had similar proportions of patients with high, intermediate, and low readmission risk as measured by HOSPITAL score. Compared to the 2011-2012 pre-intervention period, during the 2013-2014 intervention period, there were more surgeries (34.3% vs. 41.9%; P < 0.001), more elective surgeries (21.8% vs. 25.5%; P = 0.01), fewer medical patients (65.7% vs. 58.1%; P < 0.001), and an increase in comorbidities, including myocardial infarction, peripheral vascular disease, and liver disease (Table 2).

Table 3 shows adjusted 30-day readmissions rates, before and during the intervention period at the intervention and usual care SNFs. Compared to the pre-intervention period, 30-day all-cause adjusted readmission rates declined in the intervention SNFs (28.1% to 21.7%, P < 0.001), while it increased slightly at control sites (27.1% to 28.5%, P < 0.001). The Figure shows the adjusted 30-day readmission rates by quarter throughout the study period.

In this retrospective study of 4 years of discharges to 110 SNFs, we report on the impact of a Connected Care program, in which a physician visited patients on admission to the SNF and 4 to 5 times per week during their stay. Introduction of the program was followed by a 6.8% absolute reduction in all-cause 30-day readmission rates compared to usual care. The absolute reductions ranged from 4.6% for patients at low risk for readmission to 9.1% for patients at high risk, and medical patients benefited more than surgical patients.

Most studies of interventions to reduce hospital readmissions have focused on patients discharged to the community setting.^7-9 Interventions have centered on discharge planning, medication reconciliation, and close follow-up to assess for medication adherence and early signs of deterioration. Because patients in SNFs have their medications administered by staff and are under frequent surveillance, such interventions are unlikely to be helpful in this population. We found no studies that focus on short-stay or skilled patients discharged to SNF. Two studies have demonstrated that interventions can reduce hospitalization from nursing homes.^22,23 Neither study included readmissions. The Evercare model consisted of nurse practitioners providing active primary care services within the nursing home, as well as offering incentive payments to nursing homes for not hospitalizing patients.²² During a 2-year period, long term residents who enrolled in Evercare had an almost 50% reduction in incident hospitalizations compared to those who did not.²² INTERACT II was a quality improvement intervention that provided tools, education, and strategies to help identify and manage acute conditions proactively.²³ In 25 nursing homes employing INTERACT II, there was a 17% reduction in self-reported hospital admissions during the 6-month project, with higher rates of reduction among nursing homes rated as more engaged in the process.²³ Although nursing homes may serve some short-stay or skilled patients, they generally serve long-term populations, and studies have shown that short-stay patients are at higher risk for 30-day readmissions.²⁴

There are a number of reasons that short-term SNF patients are at higher risk for readmission. Although prior to admission, they were considered hospital level of care and received a physician visit daily, on transfer to the SNF, relatively little medical care is available. Current federal regulations regarding physician services at a SNF require the resident to be seen by a physician at least once every 30 days for the first 90 days after admission, and at least once every 60 days thereafter.²⁵

The Connected Care program physicians provided a smooth transition of care from hospital to SNF as well as frequent reassessment. Physicians were alerted prior to hospital discharge and performed an initial comprehensive visit generally on the day of admission to the SNF and always within 48 hours. The initial evaluation is important because miscommunication during the handoff from hospital to SNF may result in incorrect medication regimens or inaccurate assessments. By performing prompt medication reconciliation and periodic reassessments of a patient’s medical condition, the Connected Care providers recreate some of the essential elements of successful outpatient readmissions prevention programs.

They also worked together with each SNF’s interdisciplinary team to deliver quality care. There were monthly meetings at each participating Connected Care SNF. Physicians reviewed monthly 30-day readmissions and performed root-cause analysis. When they discovered challenges to timely medication and treatment delivery during daily rounds, they provided in-services to SNF nurses.

In addition, Connected Care providers discussed goals of care—something that is often overlooked on admission to a SNF. This is particularly important because patients with chronic illnesses who are discharged to SNF often have poor prognoses. For example, Medicare patients with heart failure who are discharged to SNFs have 1-year mortality in excess of 50%.¹³ By implementing a plan of care consistent with patient and family goals, inappropriate readmissions for terminal patients may be avoided.

Reducing readmissions is important for hospitals because under the Hospital Readmissions Reduction Program, hospitals now face substantial penalties for higher than expected readmissions rates. Hospitals involved in bundled payments or other total cost-of-care arrangements have additional incentive to avoid readmissions. Beginning in 2019, SNFs will also receive incentive payments based on their 30-day all-cause hospital readmissions as part of the Skilled Nursing Facility Value-Based Purchasing program.²⁵ The Connected Care model offers 1 means of achieving this goal through partnership between hospitals and SNFs.

Our study has several limitations. First, our study was observational in nature, so the observed reduction in readmissions could have been due to temporal trends unrelated to the intervention. However, no significant reduction was noted during the same time period in other area SNFs. There was also little change in the characteristics of patients admitted to the intervention SNFs. Importantly, the HOSPITAL score, which can predict 30-day readmission rates,²⁰ did not change throughout the study period. Second, the results reflect patients discharged from a single hospital and may not be generalizable to other geographic areas. However, because the program included 7 SNFs, we believe it could be reproduced in other settings. Third, our readmissions measure included only those patients who returned to a CCHS facility. Although we may have missed some readmissions to other hospital systems, such leakage is uncommon—more than 80% of CCHS patients are readmitted to CCHS facilities—and would be unlikely to differ across the short duration of the study. Finally, at the intervention SNFs, most long-stay and some short-stay residents did not receive the Connected Care intervention because they were cared for by their own physicians who did not participate in Connected Care. Had these patients’ readmissions been excluded from our results, the intervention might appear even more effective.

CONCLUSION

A Connected Care intervention reduced 30-day readmission rates among patients discharged to SNFs from a tertiary academic center. While all subgroups had substantial reductions in readmissions following the implementation of the intervention, patients who are at the highest risk of readmission benefited the most. Further study is necessary to know whether Connected Care can be reproduced in other health care systems and whether it reduces overall costs.

Acknowledgments

The authors would like to thank Michael Felver, MD, and teams for their clinical care of patients; Michael Felver, MD, William Zafirau, MD, Dan Blechschmid, MHA, and Kathy Brezine, and Seth Vilensky, MBA, for their administrative support; and Brad Souder, MPT, for assistance with data collection.

Disclosure

Nothing to report.

Approximately 20% of hospitalized Medicare beneficiaries in the U.S. are discharged to skilled nursing facilities (SNFs) for post-acute care,^1,2 and 23.5% of these patients are readmitted within 30 days.³ Because hospital readmissions are costly and associated with worse outcomes,^4,5 30-day readmission rates are considered a quality indicator,⁶ and there are financial penalties for hospitals with higher than expected rates.⁷ As a result, hospitals invest substantial resources in programs to reduce readmissions.^8-10 The SNFs represent an attractive target for readmission reduction efforts, since SNFs contribute a disproportionate share of readmissions.^3,4 Because SNF patients are in a monitored environment with high medication adherence, risk factors for readmission likely differ between patients discharged to SNFs and those sent home. For example, 1 study showed that among heart failure patients with cognitive impairment, those discharged to SNFs had lower readmissions during the first 20 days, likely due to better medication adherence.¹¹ Patients discharged to SNFs generally have more complex illnesses, lower functional status, and higher 1-year mortality than patients discharged to the community.^12,13 Despite this, SNF patients might have infrequent contact with physicians. Federal regulations require only that patients discharged to SNFs need to be seen within 30 days and then at least once every 30 days thereafter.¹⁴ According to the 2014 Office of Inspector General report, one-third of Medicare beneficiaries in SNFs experience adverse events from substandard treatment, inadequate resident monitoring and failure or delay of necessary care, most of which are thought to be preventable.¹⁵

To address this issue, the Cleveland Clinic developed a program called “Connected Care SNF,” in which hospital-employed physicians and advanced practice professionals visit patients in selected SNFs 4 to 5 times per week, for the purpose of reducing preventable readmissions. The aim of this study was to assess whether the program reduced 30-day readmissions, and to identify which patients benefited most from the program.

Setting and Intervention

The Cleveland Clinic main campus is a tertiary academic medical center with 1400 beds and approximately 50,000 admissions per year. In late 2012, the Cleveland Clinic implemented the Connected Care SNF program, wherein Cleveland Clinic physicians regularly visited patients who were discharged from the Cleveland Clinic main campus to 7 regional SNFs. Beginning in December 2012, these 7 high-volume referral SNFs that were not part of the Cleveland Clinic Health System (CCHS) agreed to participate in the program, which focused on reducing avoidable hospital readmissions and delivering quality care (Table 1). The Connected Care team, comprised of 2 geriatricians (1 of whom was also a palliative medicine specialist), 1 internist, 1 family physician, and 5 advanced practice professionals (nurse practitioners and physician assistants), provided medical services at the participating SNFs. These providers aimed to see patients 4 to 5 times per week, were available on site during working hours, and provided telephone coverage at nights and on weekends. All providers had access to hospital electronic medical records and could communicate with the discharging physician and with specialists familiar with the patient as needed. Prior to the admission, providers were informed about patient arrival and, at the time of admission to the SNF, providers reviewed medications and discussed goals of care with patients and their families. In the SNF, providers worked closely with staff members to deliver medications and timely treatment. They also met monthly with multidisciplinary teams for continuous quality improvement and to review outcomes. Patients at Connected Care SNFs who had their own physicians, including most long-stay and some short-stay residents, did not receive the Connected Care intervention. They constituted less than 10% of the patients discharged from Cleveland Clinic main campus.

Study Design and Population

We reviewed administrative and clinical data from a retrospective cohort of patients discharged to SNF from the Cleveland Clinic main campus from January 1, 2011 to December 31, 2014. We included all patients who were discharged to an SNF during the study period. Our main outcome measure was 30-day all-cause readmissions to any hospital in the Cleveland Clinic Health System (CCHS), including the main campus and 8 regional community hospitals. Study patients were followed until January 30, 2015 to capture 30-day readmissions. According to 2012 Medicare data, of CCHS patients who were readmitted within 30 days, 83% of pneumonia, 81% of major joint replacement, 72% of heart failure and 57% of acute myocardial infarction patients were readmitted to a CCHS facility. As the Cleveland Clinic main campus attracts cardiac patients from a 100+-mile radius, they may be more likely to seek care readmission near home and are not reflective of CCHS patients overall. Because we did not have access to readmissions data from non-CCHS hospitals, we excluded patients who were discharged to SNFs beyond a 25-mile radius from the main campus, where they may be more likely to utilize non-CCHS hospitals for acute hospitalization. We also excluded patients discharged to non-CCHS hospital-based SNFs, which may refer readmissions to their own hospital system. Because the Connected Care program began in December 2012, the years 2011-2012 served as the baseline period. The intervention was conducted at 7 SNFs. All other SNFs within the 25-mile radius were included as controls, except for 3 hospital-based SNFs that would be unlikely to admit patients to CCHS. We compared the change in all-cause 30-day readmission rates after implementation of Connected Care, using all patients discharged to SNFs within 25 miles to control for temporal changes in local readmission rates. Discharge to specific SNFs was determined solely by patient choice.

Data Collection

For each patient, we collected the following data that has been shown to be associated with readmissions:^16-18 demographics (age, race, sex, ZIP code), lab values on discharge (hemoglobin and sodium); hemodialysis status; medicine or surgical service; elective surgery or nonelective surgery; details of the index admission index (diagnosis-related group [DRG], Medicare severity-diagnosis-related groups [MS-DRG] weight, primary diagnosis code; principal procedure code; admission date; discharge date, length of stay, and post-acute care provider); and common comorbidities, as listed in Table 2. We also calculated each patient’s HOSPITAL^19,20 score. The HOSPITAL score was developed to predict risk of preventable 30-day readmissions,¹⁹ but it has also been validated to predict 30-day all-cause readmission rates for patients discharged to SNF.²¹ The model contains 7 elements (hemoglobin, oncology service, sodium, procedure, index type, admissions within the last year, length of stay) (supplemental Table).Patients with a high score (7 or higher) have a 41% chance of readmission, while those with a low score (4 or lower) have only a 15% chance. ²¹ We assessed all cause 30-day readmission status from CCHS administrative data. Observation patients and outpatient same-day surgeries were not considered to be admissions. For patients with multiple admissions, each admission was counted as a separate index hospitalization. Cleveland Clinic’s Institutional Review Board approved the study.

Statistical Analysis

For the 7 intervention SNFs, patient characteristics were summarized as means and standard deviations or frequencies and percentages for the periods of 2011-2012 and 2013-2014, respectively, and the 2 periods were compared using the Student t test or χ² test as appropriate.

Mixed-effects logistic regression models were used to model 30-day readmission rates. Since the intervention was implemented in the last quarter of 2012, we examined the difference in readmission rates before and after that time point. The model included the following fixed effects: SNF type (intervention or usual care), time points (quarters of 2011-2014), whether the time is pre- or postintervention (binary), and the 3-way interaction between SNF type, pre- or postintervention and time points, and patient characteristics. The model also contained a Gaussian random effect at the SNF level to account for possible correlations among the outcomes of patients from the same SNF. For each quarter, the mean adjusted readmission rates of 2 types of SNFs were calculated from the fitted mixed models and plotted over time. Furthermore, we compared the mean readmission rates of the 2 groups in the pre- and postintervention periods. Subgroup analyses were performed for medical and surgical patients, and for patients in the low, intermediate and high HOSPITAL score groups.

All analyses were performed using RStudio (Boston, Massachusetts). Statistical significance was established with 2-sided P values less than 0.05.

We identified 119 SNFs within a 25-mile radius of the hospital. Of these, 6 did not receive any referrals. Three non-CCHS hospital-based SNFs were excluded, leaving a total of 110 SNFs in the study sample: 7 intervention SNFs and 103 usual-care SNFs. Between January 2011 and December 2014, there were 23,408 SNF discharges from Cleveland Clinic main campus, including 13,544 who were discharged to study SNFs (Supplemental Figure). Of these, 3334 were discharged to 7 intervention SNFs and 10,210 were discharged to usual care SNFs. Characteristics of patients in both periods appear in Table 2. At baseline, patients in the intervention and control SNFs varied in a number of ways. Patients at intervention SNFs were older (75.6 vs. 70.2 years; P < 0.001), more likely to be African American (45.5% vs. 35.9%; P < 0.001), female (61% vs. 55.4%; P < 0.001) and to be insured by Medicare (85.2% vs. 71.4%; P < 0.001). Both groups had similar proportions of patients with high, intermediate, and low readmission risk as measured by HOSPITAL score. Compared to the 2011-2012 pre-intervention period, during the 2013-2014 intervention period, there were more surgeries (34.3% vs. 41.9%; P < 0.001), more elective surgeries (21.8% vs. 25.5%; P = 0.01), fewer medical patients (65.7% vs. 58.1%; P < 0.001), and an increase in comorbidities, including myocardial infarction, peripheral vascular disease, and liver disease (Table 2).

Table 3 shows adjusted 30-day readmissions rates, before and during the intervention period at the intervention and usual care SNFs. Compared to the pre-intervention period, 30-day all-cause adjusted readmission rates declined in the intervention SNFs (28.1% to 21.7%, P < 0.001), while it increased slightly at control sites (27.1% to 28.5%, P < 0.001). The Figure shows the adjusted 30-day readmission rates by quarter throughout the study period.

In this retrospective study of 4 years of discharges to 110 SNFs, we report on the impact of a Connected Care program, in which a physician visited patients on admission to the SNF and 4 to 5 times per week during their stay. Introduction of the program was followed by a 6.8% absolute reduction in all-cause 30-day readmission rates compared to usual care. The absolute reductions ranged from 4.6% for patients at low risk for readmission to 9.1% for patients at high risk, and medical patients benefited more than surgical patients.

Most studies of interventions to reduce hospital readmissions have focused on patients discharged to the community setting.^7-9 Interventions have centered on discharge planning, medication reconciliation, and close follow-up to assess for medication adherence and early signs of deterioration. Because patients in SNFs have their medications administered by staff and are under frequent surveillance, such interventions are unlikely to be helpful in this population. We found no studies that focus on short-stay or skilled patients discharged to SNF. Two studies have demonstrated that interventions can reduce hospitalization from nursing homes.^22,23 Neither study included readmissions. The Evercare model consisted of nurse practitioners providing active primary care services within the nursing home, as well as offering incentive payments to nursing homes for not hospitalizing patients.²² During a 2-year period, long term residents who enrolled in Evercare had an almost 50% reduction in incident hospitalizations compared to those who did not.²² INTERACT II was a quality improvement intervention that provided tools, education, and strategies to help identify and manage acute conditions proactively.²³ In 25 nursing homes employing INTERACT II, there was a 17% reduction in self-reported hospital admissions during the 6-month project, with higher rates of reduction among nursing homes rated as more engaged in the process.²³ Although nursing homes may serve some short-stay or skilled patients, they generally serve long-term populations, and studies have shown that short-stay patients are at higher risk for 30-day readmissions.²⁴

There are a number of reasons that short-term SNF patients are at higher risk for readmission. Although prior to admission, they were considered hospital level of care and received a physician visit daily, on transfer to the SNF, relatively little medical care is available. Current federal regulations regarding physician services at a SNF require the resident to be seen by a physician at least once every 30 days for the first 90 days after admission, and at least once every 60 days thereafter.²⁵

The Connected Care program physicians provided a smooth transition of care from hospital to SNF as well as frequent reassessment. Physicians were alerted prior to hospital discharge and performed an initial comprehensive visit generally on the day of admission to the SNF and always within 48 hours. The initial evaluation is important because miscommunication during the handoff from hospital to SNF may result in incorrect medication regimens or inaccurate assessments. By performing prompt medication reconciliation and periodic reassessments of a patient’s medical condition, the Connected Care providers recreate some of the essential elements of successful outpatient readmissions prevention programs.

They also worked together with each SNF’s interdisciplinary team to deliver quality care. There were monthly meetings at each participating Connected Care SNF. Physicians reviewed monthly 30-day readmissions and performed root-cause analysis. When they discovered challenges to timely medication and treatment delivery during daily rounds, they provided in-services to SNF nurses.

In addition, Connected Care providers discussed goals of care—something that is often overlooked on admission to a SNF. This is particularly important because patients with chronic illnesses who are discharged to SNF often have poor prognoses. For example, Medicare patients with heart failure who are discharged to SNFs have 1-year mortality in excess of 50%.¹³ By implementing a plan of care consistent with patient and family goals, inappropriate readmissions for terminal patients may be avoided.

Reducing readmissions is important for hospitals because under the Hospital Readmissions Reduction Program, hospitals now face substantial penalties for higher than expected readmissions rates. Hospitals involved in bundled payments or other total cost-of-care arrangements have additional incentive to avoid readmissions. Beginning in 2019, SNFs will also receive incentive payments based on their 30-day all-cause hospital readmissions as part of the Skilled Nursing Facility Value-Based Purchasing program.²⁵ The Connected Care model offers 1 means of achieving this goal through partnership between hospitals and SNFs.

Our study has several limitations. First, our study was observational in nature, so the observed reduction in readmissions could have been due to temporal trends unrelated to the intervention. However, no significant reduction was noted during the same time period in other area SNFs. There was also little change in the characteristics of patients admitted to the intervention SNFs. Importantly, the HOSPITAL score, which can predict 30-day readmission rates,²⁰ did not change throughout the study period. Second, the results reflect patients discharged from a single hospital and may not be generalizable to other geographic areas. However, because the program included 7 SNFs, we believe it could be reproduced in other settings. Third, our readmissions measure included only those patients who returned to a CCHS facility. Although we may have missed some readmissions to other hospital systems, such leakage is uncommon—more than 80% of CCHS patients are readmitted to CCHS facilities—and would be unlikely to differ across the short duration of the study. Finally, at the intervention SNFs, most long-stay and some short-stay residents did not receive the Connected Care intervention because they were cared for by their own physicians who did not participate in Connected Care. Had these patients’ readmissions been excluded from our results, the intervention might appear even more effective.

CONCLUSION

A Connected Care intervention reduced 30-day readmission rates among patients discharged to SNFs from a tertiary academic center. While all subgroups had substantial reductions in readmissions following the implementation of the intervention, patients who are at the highest risk of readmission benefited the most. Further study is necessary to know whether Connected Care can be reproduced in other health care systems and whether it reduces overall costs.

Acknowledgments

The authors would like to thank Michael Felver, MD, and teams for their clinical care of patients; Michael Felver, MD, William Zafirau, MD, Dan Blechschmid, MHA, and Kathy Brezine, and Seth Vilensky, MBA, for their administrative support; and Brad Souder, MPT, for assistance with data collection.

Disclosure

Nothing to report.

A patient’s prognosis can strongly influence their medical care. Decisions about diagnostic modalities, treatment options, and the use of preventive therapies can all be affected by the likelihood of a patient’s death in the near future. For example, patients with severely limited survival might forego prophylactic therapy, avoid interventions for asymptomatic issues, and cease screening interventions. Knowing survival probability would also be very helpful as a controlling variable in research analyses whenever death risk might be a possible confounder.

Sixteen indices that aim to predict patient death risk have been described by Yourman et al.¹ They were all created from secondary analyses of clinical and administrative datasets, were applicable to patients in a variety of settings (including the community, nursing home, or hospital), and predicted survival probabilities in time horizons ranging from 6 months to 5 years. Prognostic factors that were most commonly included in these indices were comorbidity and functional status. In validation populations, the discrimination of these indices for 1-year survival in hospitalized patients was moderate (with C statistics that ranged from 0.64 to 0.79) with good calibration for broad prognostic ranges.

In 2014, we published the Hospitalized-patient One-year Mortality Risk (HOMR) score.² This study used health administrative data for all adult Ontarians admitted in 2011 to hospital under nonpsychiatric services (n = 640,022) to estimate the probability of dying within 1 year of admission to hospital (which happened in 11.7% of people). The HOMR score included 12 patient and hospitalization factors (Table 1). It was highly discriminative (C statistic, 0.923; [0.922-0.924]) and well calibrated (the mean relative difference between observed and expected death risk was 2.0% [range, 0.0% to 7.0%]). It was externally validated in more than 3 million adults from Ontario, Alberta, and Boston in whom the C statistic ranged from 0.89 to 0.92 and calibration was excellent.³ We concluded from these studies that the HOMR score is excellent for prognosticating a diverse group of patients using health administrative data.

Study Cohort

The study, which was approved by our local research ethics board, took place at the Ottawa Hospital, a 1000-bed teaching hospital that is the primary referral center in our region. We used the hospital admission registry to identify all people 18 years or older who were admitted to a nonpsychiatric service at our hospital between January 1, 2011 and December 31, 2011 (this time frame corresponds with the year used to derive the HOMR score). We excluded overnight patients in the same-day surgery or the bone-marrow transplant units (since they would not have been included in the original study) and those without a valid health card number (which was required to link to provincial data to identify outcomes). From this list, we randomly selected 5000 patients.

Primary Data Collection

For each patient, we retrieved all data required to calculate the HOMR score from the medical record (Table 1). Patient registration information in our electronic medical record was used to identify patient age, sex, admitting service, number of emergency department (ED) visits in the previous year, number of admissions in the previous year (the nursing triage note was reviewed for each admission to determine if it was by ambulance), and whether or not the patient had been discharged from hospital in the previous 30 days. The admitting service consult note was used to determine the admitting diagnosis and whether or not the patient was admitted directly to the intensive care unit. If they were present, the emergency nursing triage note, the ED record of treatment, the admission consult note, the pre-operative consult note, and consult notes were all used to determine the patient’s comorbidities, living status, and home oxygen status. Admission urgency was determined using information from the patient registration information and the ED nursing triage note. All data were abstracted from information that had been registered prior to when the patient was physically transferred to their hospital bed. This ensured that we used only data available at the start of the admission.

Patient functional status has been shown to be strongly associated with survival⁴ but HOMR only indirectly captures functional information (through the patient’s living status). We, therefore, collected more detailed functional information from the medical record by determining if the patient was dependent for any activities of daily living (ADL) from the emergency nursing triage note, the ED record of treatment, the admission consult note, and the pre-operative consultation. We also collected information that might indicate frailty, which we defined per Clegg et al.⁵ as “a state of increased vulnerability to poor resolution of homeostasis following a stress.” This information included: delirium or more than 1 fall recorded on the emergency nursing triage note, the ED record of treatment, or the admission consultation note; or whether a geriatric nursing specialist assessment occurred in the ED in the previous 6 months. Finally, we recorded possible indicators of limited social support (no fixed address [from patient registration and nursing triage note], primary contact is not a family member [from the emergency notes, consult, and patient registration], and no religion noted in system [from patient registration]). Patients for whom religion status was missing were classified as having “no religion.”

Analysis

These data were encrypted and linked anonymously to population-based databases to determine whether patients died within 1 year of admission to hospital. We calculated the chart-HOMR score using information from the chart review and determined its association with the outcome using bivariate logistic regression. We compared observed and expected risk of death within 1 year of admission to hospital for each chart-HOMR score value, with expected risks determined from the external validation study.³ We regressed observed death risks on expected death risks for chart-HOMR scores (clustered into 22 groups to ensure adequate numbers in each group); and we gauged overall deviations from expected risk and the relationship between the observed and expected death risk (based on the chart-HOMR score) using the line’s intercept and slope, respectively.⁶ Next, we replicated methods from our studies^2,3 to calculate the administrative-HOMR score in our study cohort using administrative databases. We compared these chart-HOMR and administrative-HOMR scores (and scores for each of its components). Finally, we determined which of the socio-functional factors were associated with 1-year death risk independent of the chart-HOMR score. We used the likelihood ratio test to determine whether these additional socio-functional factors significantly improved the model beyond the chart-HOMR score.⁷ This test subtracted the -2 logL value of the full model from that containing the chart-HOMR score alone, comparing its value to the χ² distribution (with degrees of freedom equivalent to the number of additional parameters in the nested model) to determine statistical significance. All analyses were completed using SAS v9.4 (SAS Institute Inc., Cary, North Carolina).

There were 43,883 overnight hospitalizations at our hospital in 2011, and 38,886 hospitalizations were excluded: 1883 hospitalizations were in the same-day surgery or the bone-marrow transplant unit; 2485 did not have a valid health card number; 34,515 were not randomly selected; the records of 3 randomly selected patients had been blocked by our hospital’s privacy department; and 1 patient could not be linked with the population-based administrative datasets.

The 4996 study patients were middle-aged and predominantly female (Table 2). The extensive majority of patients was admitted from the community, was independent for ADL, had a family member as the principal contact, and had no admissions by ambulance in the previous year. Most people had no significant comorbidities or ED visits in the year prior to their admission. The mean chart-HOMR score was 22 (standard deviation [SD], 12), which is associated with a 1.2% expected risk of death within 1 year of hospital admission (Appendix 1).³

In a large random sample of patients from our hospital, we found that the HOMR score using data abstracted from the medical record was significantly associated with 1-year death risk. The expected death risk based on the chart-HOMR score underestimated observed death risk but the relationship between the chart-HOMR score and death risk was similar to that in studies using administrative data. The HOMR score calculated using data from the chart was lower than that calculated using data from population-based administrative datasets; additional variables indicating patient frailty were significantly associated with 1-year death risk independent of the chart-HOMR score. Since the HOMR score was derived and initially validated using health administrative data, this study using data abstracted from the health record shows that the HOMR score has methodological generalizability.⁸

We think that our study has several notable findings. First, we found that data abstracted from the medical record can be used to calculate the HOMR score to accurately predict individual death risk. The chart-HOMR score discriminated very well between patients who did and did not die (C statistic, 0.88), which extensively exceeds the discrimination of published death risk indices (whose C statistics range between 0.69 and 0.82). It is also possible that chart abstraction for the HOMR score—without functional status—is simpler than other indices since its components are primarily very objective. (Other indices for hospital-based patients required factors that could be difficult to abstract reliably from the medical record including meeting more than 1 guideline for noncancer hospice care⁹; ambulation difficulties¹⁰; scales such as the Exton-Smith Scale or the Short Portable Mental Status Questionnaire¹¹; weight loss¹²; functional status⁴; and pressure sore risk.¹³) Although expected risks for the chart-HOMR consistently underestimated observed risks (Figure 1), the mean deviation was small (with an absolute difference of 3.5% that can be used as a correction factor when determining expected risks with HOMR scores calculated from chart review), but it was an association between the chart-HOMR score and death risk that remained consistent through the cohort. Second, we found a small but significant decrease in the chart-HOMR score vs. the administrative-HOMR score (Figure 2). Some of these underestimates such as those for the number of ED visits or admissions by ambulance were expected since population-based health administrative databases would best capture such data. However, we were surprised that the comorbidity score was less when calculated using chart vs. database data (Figure 2). This finding is distinct from studies finding that particular comorbidities are documented in the chart are sometimes not coded.^14,15 However, we identified comorbidities in the administrative databases using a 1-year ‘look-back’ period so that diagnostic codes from multiple hospitalizations (and from multiple hospitals) could be used to calculate the Charlson Comorbidity Index for a particular patient; this has been shown to increase the capture of comorbidities.¹⁶ Third, we found that variables from the chart review indicating frailty were predictive of 1-year death risk independent of the chart-HOMR score (Table 2). This illustrates that mortality risk prediction can be improved for particular patient groups by adding new covariates to the HOMR. Further work is required to determine how to incorporate these (and possibly other) covariates into the HOMR to create a unique chart-HOMR score. Finally, we found that a geriatrics assessment in the ED was associated with a significant (and notable) decrease in death risk. With these data, we are unable to indicate whether this association is causative. However, these findings indicate that the influence of emergency geriatric assessments on patient survival needs to be explored in more detail.

Several issues about our study should be considered when interpreting its results. First, this was a single-center study and the generalizability of our results to other centers is unknown. However, our study had the largest sample size of all primary data prognostic index validation studies¹ ensuring that our results are, at the very least, internally reliable. In addition, our simple random sample ensured that we studied a broad assortment of patients to be certain that our results are representative of our institution. Second, we used a single abstractor for the study, which could limit the generalizability of our results. However, almost all the data points that were abstracted for our study were very objective.

In summary, our study shows that the HOMR score can be used to accurately predict 1-year death risk using data abstracted from the patient record. These findings will aid in individual patient prognostication for clinicians and researchers.

Disclosure

The authors report no financial conflicts of interest.

A patient’s prognosis can strongly influence their medical care. Decisions about diagnostic modalities, treatment options, and the use of preventive therapies can all be affected by the likelihood of a patient’s death in the near future. For example, patients with severely limited survival might forego prophylactic therapy, avoid interventions for asymptomatic issues, and cease screening interventions. Knowing survival probability would also be very helpful as a controlling variable in research analyses whenever death risk might be a possible confounder.

Sixteen indices that aim to predict patient death risk have been described by Yourman et al.¹ They were all created from secondary analyses of clinical and administrative datasets, were applicable to patients in a variety of settings (including the community, nursing home, or hospital), and predicted survival probabilities in time horizons ranging from 6 months to 5 years. Prognostic factors that were most commonly included in these indices were comorbidity and functional status. In validation populations, the discrimination of these indices for 1-year survival in hospitalized patients was moderate (with C statistics that ranged from 0.64 to 0.79) with good calibration for broad prognostic ranges.

In 2014, we published the Hospitalized-patient One-year Mortality Risk (HOMR) score.² This study used health administrative data for all adult Ontarians admitted in 2011 to hospital under nonpsychiatric services (n = 640,022) to estimate the probability of dying within 1 year of admission to hospital (which happened in 11.7% of people). The HOMR score included 12 patient and hospitalization factors (Table 1). It was highly discriminative (C statistic, 0.923; [0.922-0.924]) and well calibrated (the mean relative difference between observed and expected death risk was 2.0% [range, 0.0% to 7.0%]). It was externally validated in more than 3 million adults from Ontario, Alberta, and Boston in whom the C statistic ranged from 0.89 to 0.92 and calibration was excellent.³ We concluded from these studies that the HOMR score is excellent for prognosticating a diverse group of patients using health administrative data.

Study Cohort

The study, which was approved by our local research ethics board, took place at the Ottawa Hospital, a 1000-bed teaching hospital that is the primary referral center in our region. We used the hospital admission registry to identify all people 18 years or older who were admitted to a nonpsychiatric service at our hospital between January 1, 2011 and December 31, 2011 (this time frame corresponds with the year used to derive the HOMR score). We excluded overnight patients in the same-day surgery or the bone-marrow transplant units (since they would not have been included in the original study) and those without a valid health card number (which was required to link to provincial data to identify outcomes). From this list, we randomly selected 5000 patients.

Primary Data Collection

For each patient, we retrieved all data required to calculate the HOMR score from the medical record (Table 1). Patient registration information in our electronic medical record was used to identify patient age, sex, admitting service, number of emergency department (ED) visits in the previous year, number of admissions in the previous year (the nursing triage note was reviewed for each admission to determine if it was by ambulance), and whether or not the patient had been discharged from hospital in the previous 30 days. The admitting service consult note was used to determine the admitting diagnosis and whether or not the patient was admitted directly to the intensive care unit. If they were present, the emergency nursing triage note, the ED record of treatment, the admission consult note, the pre-operative consult note, and consult notes were all used to determine the patient’s comorbidities, living status, and home oxygen status. Admission urgency was determined using information from the patient registration information and the ED nursing triage note. All data were abstracted from information that had been registered prior to when the patient was physically transferred to their hospital bed. This ensured that we used only data available at the start of the admission.

Patient functional status has been shown to be strongly associated with survival⁴ but HOMR only indirectly captures functional information (through the patient’s living status). We, therefore, collected more detailed functional information from the medical record by determining if the patient was dependent for any activities of daily living (ADL) from the emergency nursing triage note, the ED record of treatment, the admission consult note, and the pre-operative consultation. We also collected information that might indicate frailty, which we defined per Clegg et al.⁵ as “a state of increased vulnerability to poor resolution of homeostasis following a stress.” This information included: delirium or more than 1 fall recorded on the emergency nursing triage note, the ED record of treatment, or the admission consultation note; or whether a geriatric nursing specialist assessment occurred in the ED in the previous 6 months. Finally, we recorded possible indicators of limited social support (no fixed address [from patient registration and nursing triage note], primary contact is not a family member [from the emergency notes, consult, and patient registration], and no religion noted in system [from patient registration]). Patients for whom religion status was missing were classified as having “no religion.”

Analysis

These data were encrypted and linked anonymously to population-based databases to determine whether patients died within 1 year of admission to hospital. We calculated the chart-HOMR score using information from the chart review and determined its association with the outcome using bivariate logistic regression. We compared observed and expected risk of death within 1 year of admission to hospital for each chart-HOMR score value, with expected risks determined from the external validation study.³ We regressed observed death risks on expected death risks for chart-HOMR scores (clustered into 22 groups to ensure adequate numbers in each group); and we gauged overall deviations from expected risk and the relationship between the observed and expected death risk (based on the chart-HOMR score) using the line’s intercept and slope, respectively.⁶ Next, we replicated methods from our studies^2,3 to calculate the administrative-HOMR score in our study cohort using administrative databases. We compared these chart-HOMR and administrative-HOMR scores (and scores for each of its components). Finally, we determined which of the socio-functional factors were associated with 1-year death risk independent of the chart-HOMR score. We used the likelihood ratio test to determine whether these additional socio-functional factors significantly improved the model beyond the chart-HOMR score.⁷ This test subtracted the -2 logL value of the full model from that containing the chart-HOMR score alone, comparing its value to the χ² distribution (with degrees of freedom equivalent to the number of additional parameters in the nested model) to determine statistical significance. All analyses were completed using SAS v9.4 (SAS Institute Inc., Cary, North Carolina).

There were 43,883 overnight hospitalizations at our hospital in 2011, and 38,886 hospitalizations were excluded: 1883 hospitalizations were in the same-day surgery or the bone-marrow transplant unit; 2485 did not have a valid health card number; 34,515 were not randomly selected; the records of 3 randomly selected patients had been blocked by our hospital’s privacy department; and 1 patient could not be linked with the population-based administrative datasets.

The 4996 study patients were middle-aged and predominantly female (Table 2). The extensive majority of patients was admitted from the community, was independent for ADL, had a family member as the principal contact, and had no admissions by ambulance in the previous year. Most people had no significant comorbidities or ED visits in the year prior to their admission. The mean chart-HOMR score was 22 (standard deviation [SD], 12), which is associated with a 1.2% expected risk of death within 1 year of hospital admission (Appendix 1).³

In a large random sample of patients from our hospital, we found that the HOMR score using data abstracted from the medical record was significantly associated with 1-year death risk. The expected death risk based on the chart-HOMR score underestimated observed death risk but the relationship between the chart-HOMR score and death risk was similar to that in studies using administrative data. The HOMR score calculated using data from the chart was lower than that calculated using data from population-based administrative datasets; additional variables indicating patient frailty were significantly associated with 1-year death risk independent of the chart-HOMR score. Since the HOMR score was derived and initially validated using health administrative data, this study using data abstracted from the health record shows that the HOMR score has methodological generalizability.⁸

We think that our study has several notable findings. First, we found that data abstracted from the medical record can be used to calculate the HOMR score to accurately predict individual death risk. The chart-HOMR score discriminated very well between patients who did and did not die (C statistic, 0.88), which extensively exceeds the discrimination of published death risk indices (whose C statistics range between 0.69 and 0.82). It is also possible that chart abstraction for the HOMR score—without functional status—is simpler than other indices since its components are primarily very objective. (Other indices for hospital-based patients required factors that could be difficult to abstract reliably from the medical record including meeting more than 1 guideline for noncancer hospice care⁹; ambulation difficulties¹⁰; scales such as the Exton-Smith Scale or the Short Portable Mental Status Questionnaire¹¹; weight loss¹²; functional status⁴; and pressure sore risk.¹³) Although expected risks for the chart-HOMR consistently underestimated observed risks (Figure 1), the mean deviation was small (with an absolute difference of 3.5% that can be used as a correction factor when determining expected risks with HOMR scores calculated from chart review), but it was an association between the chart-HOMR score and death risk that remained consistent through the cohort. Second, we found a small but significant decrease in the chart-HOMR score vs. the administrative-HOMR score (Figure 2). Some of these underestimates such as those for the number of ED visits or admissions by ambulance were expected since population-based health administrative databases would best capture such data. However, we were surprised that the comorbidity score was less when calculated using chart vs. database data (Figure 2). This finding is distinct from studies finding that particular comorbidities are documented in the chart are sometimes not coded.^14,15 However, we identified comorbidities in the administrative databases using a 1-year ‘look-back’ period so that diagnostic codes from multiple hospitalizations (and from multiple hospitals) could be used to calculate the Charlson Comorbidity Index for a particular patient; this has been shown to increase the capture of comorbidities.¹⁶ Third, we found that variables from the chart review indicating frailty were predictive of 1-year death risk independent of the chart-HOMR score (Table 2). This illustrates that mortality risk prediction can be improved for particular patient groups by adding new covariates to the HOMR. Further work is required to determine how to incorporate these (and possibly other) covariates into the HOMR to create a unique chart-HOMR score. Finally, we found that a geriatrics assessment in the ED was associated with a significant (and notable) decrease in death risk. With these data, we are unable to indicate whether this association is causative. However, these findings indicate that the influence of emergency geriatric assessments on patient survival needs to be explored in more detail.

Several issues about our study should be considered when interpreting its results. First, this was a single-center study and the generalizability of our results to other centers is unknown. However, our study had the largest sample size of all primary data prognostic index validation studies¹ ensuring that our results are, at the very least, internally reliable. In addition, our simple random sample ensured that we studied a broad assortment of patients to be certain that our results are representative of our institution. Second, we used a single abstractor for the study, which could limit the generalizability of our results. However, almost all the data points that were abstracted for our study were very objective.

In summary, our study shows that the HOMR score can be used to accurately predict 1-year death risk using data abstracted from the patient record. These findings will aid in individual patient prognostication for clinicians and researchers.

Disclosure

The authors report no financial conflicts of interest.

A patient’s prognosis can strongly influence their medical care. Decisions about diagnostic modalities, treatment options, and the use of preventive therapies can all be affected by the likelihood of a patient’s death in the near future. For example, patients with severely limited survival might forego prophylactic therapy, avoid interventions for asymptomatic issues, and cease screening interventions. Knowing survival probability would also be very helpful as a controlling variable in research analyses whenever death risk might be a possible confounder.

Sixteen indices that aim to predict patient death risk have been described by Yourman et al.¹ They were all created from secondary analyses of clinical and administrative datasets, were applicable to patients in a variety of settings (including the community, nursing home, or hospital), and predicted survival probabilities in time horizons ranging from 6 months to 5 years. Prognostic factors that were most commonly included in these indices were comorbidity and functional status. In validation populations, the discrimination of these indices for 1-year survival in hospitalized patients was moderate (with C statistics that ranged from 0.64 to 0.79) with good calibration for broad prognostic ranges.

In 2014, we published the Hospitalized-patient One-year Mortality Risk (HOMR) score.² This study used health administrative data for all adult Ontarians admitted in 2011 to hospital under nonpsychiatric services (n = 640,022) to estimate the probability of dying within 1 year of admission to hospital (which happened in 11.7% of people). The HOMR score included 12 patient and hospitalization factors (Table 1). It was highly discriminative (C statistic, 0.923; [0.922-0.924]) and well calibrated (the mean relative difference between observed and expected death risk was 2.0% [range, 0.0% to 7.0%]). It was externally validated in more than 3 million adults from Ontario, Alberta, and Boston in whom the C statistic ranged from 0.89 to 0.92 and calibration was excellent.³ We concluded from these studies that the HOMR score is excellent for prognosticating a diverse group of patients using health administrative data.

Study Cohort

The study, which was approved by our local research ethics board, took place at the Ottawa Hospital, a 1000-bed teaching hospital that is the primary referral center in our region. We used the hospital admission registry to identify all people 18 years or older who were admitted to a nonpsychiatric service at our hospital between January 1, 2011 and December 31, 2011 (this time frame corresponds with the year used to derive the HOMR score). We excluded overnight patients in the same-day surgery or the bone-marrow transplant units (since they would not have been included in the original study) and those without a valid health card number (which was required to link to provincial data to identify outcomes). From this list, we randomly selected 5000 patients.

Primary Data Collection

For each patient, we retrieved all data required to calculate the HOMR score from the medical record (Table 1). Patient registration information in our electronic medical record was used to identify patient age, sex, admitting service, number of emergency department (ED) visits in the previous year, number of admissions in the previous year (the nursing triage note was reviewed for each admission to determine if it was by ambulance), and whether or not the patient had been discharged from hospital in the previous 30 days. The admitting service consult note was used to determine the admitting diagnosis and whether or not the patient was admitted directly to the intensive care unit. If they were present, the emergency nursing triage note, the ED record of treatment, the admission consult note, the pre-operative consult note, and consult notes were all used to determine the patient’s comorbidities, living status, and home oxygen status. Admission urgency was determined using information from the patient registration information and the ED nursing triage note. All data were abstracted from information that had been registered prior to when the patient was physically transferred to their hospital bed. This ensured that we used only data available at the start of the admission.

Patient functional status has been shown to be strongly associated with survival⁴ but HOMR only indirectly captures functional information (through the patient’s living status). We, therefore, collected more detailed functional information from the medical record by determining if the patient was dependent for any activities of daily living (ADL) from the emergency nursing triage note, the ED record of treatment, the admission consult note, and the pre-operative consultation. We also collected information that might indicate frailty, which we defined per Clegg et al.⁵ as “a state of increased vulnerability to poor resolution of homeostasis following a stress.” This information included: delirium or more than 1 fall recorded on the emergency nursing triage note, the ED record of treatment, or the admission consultation note; or whether a geriatric nursing specialist assessment occurred in the ED in the previous 6 months. Finally, we recorded possible indicators of limited social support (no fixed address [from patient registration and nursing triage note], primary contact is not a family member [from the emergency notes, consult, and patient registration], and no religion noted in system [from patient registration]). Patients for whom religion status was missing were classified as having “no religion.”

Analysis

These data were encrypted and linked anonymously to population-based databases to determine whether patients died within 1 year of admission to hospital. We calculated the chart-HOMR score using information from the chart review and determined its association with the outcome using bivariate logistic regression. We compared observed and expected risk of death within 1 year of admission to hospital for each chart-HOMR score value, with expected risks determined from the external validation study.³ We regressed observed death risks on expected death risks for chart-HOMR scores (clustered into 22 groups to ensure adequate numbers in each group); and we gauged overall deviations from expected risk and the relationship between the observed and expected death risk (based on the chart-HOMR score) using the line’s intercept and slope, respectively.⁶ Next, we replicated methods from our studies^2,3 to calculate the administrative-HOMR score in our study cohort using administrative databases. We compared these chart-HOMR and administrative-HOMR scores (and scores for each of its components). Finally, we determined which of the socio-functional factors were associated with 1-year death risk independent of the chart-HOMR score. We used the likelihood ratio test to determine whether these additional socio-functional factors significantly improved the model beyond the chart-HOMR score.⁷ This test subtracted the -2 logL value of the full model from that containing the chart-HOMR score alone, comparing its value to the χ² distribution (with degrees of freedom equivalent to the number of additional parameters in the nested model) to determine statistical significance. All analyses were completed using SAS v9.4 (SAS Institute Inc., Cary, North Carolina).

There were 43,883 overnight hospitalizations at our hospital in 2011, and 38,886 hospitalizations were excluded: 1883 hospitalizations were in the same-day surgery or the bone-marrow transplant unit; 2485 did not have a valid health card number; 34,515 were not randomly selected; the records of 3 randomly selected patients had been blocked by our hospital’s privacy department; and 1 patient could not be linked with the population-based administrative datasets.

The 4996 study patients were middle-aged and predominantly female (Table 2). The extensive majority of patients was admitted from the community, was independent for ADL, had a family member as the principal contact, and had no admissions by ambulance in the previous year. Most people had no significant comorbidities or ED visits in the year prior to their admission. The mean chart-HOMR score was 22 (standard deviation [SD], 12), which is associated with a 1.2% expected risk of death within 1 year of hospital admission (Appendix 1).³

In a large random sample of patients from our hospital, we found that the HOMR score using data abstracted from the medical record was significantly associated with 1-year death risk. The expected death risk based on the chart-HOMR score underestimated observed death risk but the relationship between the chart-HOMR score and death risk was similar to that in studies using administrative data. The HOMR score calculated using data from the chart was lower than that calculated using data from population-based administrative datasets; additional variables indicating patient frailty were significantly associated with 1-year death risk independent of the chart-HOMR score. Since the HOMR score was derived and initially validated using health administrative data, this study using data abstracted from the health record shows that the HOMR score has methodological generalizability.⁸

We think that our study has several notable findings. First, we found that data abstracted from the medical record can be used to calculate the HOMR score to accurately predict individual death risk. The chart-HOMR score discriminated very well between patients who did and did not die (C statistic, 0.88), which extensively exceeds the discrimination of published death risk indices (whose C statistics range between 0.69 and 0.82). It is also possible that chart abstraction for the HOMR score—without functional status—is simpler than other indices since its components are primarily very objective. (Other indices for hospital-based patients required factors that could be difficult to abstract reliably from the medical record including meeting more than 1 guideline for noncancer hospice care⁹; ambulation difficulties¹⁰; scales such as the Exton-Smith Scale or the Short Portable Mental Status Questionnaire¹¹; weight loss¹²; functional status⁴; and pressure sore risk.¹³) Although expected risks for the chart-HOMR consistently underestimated observed risks (Figure 1), the mean deviation was small (with an absolute difference of 3.5% that can be used as a correction factor when determining expected risks with HOMR scores calculated from chart review), but it was an association between the chart-HOMR score and death risk that remained consistent through the cohort. Second, we found a small but significant decrease in the chart-HOMR score vs. the administrative-HOMR score (Figure 2). Some of these underestimates such as those for the number of ED visits or admissions by ambulance were expected since population-based health administrative databases would best capture such data. However, we were surprised that the comorbidity score was less when calculated using chart vs. database data (Figure 2). This finding is distinct from studies finding that particular comorbidities are documented in the chart are sometimes not coded.^14,15 However, we identified comorbidities in the administrative databases using a 1-year ‘look-back’ period so that diagnostic codes from multiple hospitalizations (and from multiple hospitals) could be used to calculate the Charlson Comorbidity Index for a particular patient; this has been shown to increase the capture of comorbidities.¹⁶ Third, we found that variables from the chart review indicating frailty were predictive of 1-year death risk independent of the chart-HOMR score (Table 2). This illustrates that mortality risk prediction can be improved for particular patient groups by adding new covariates to the HOMR. Further work is required to determine how to incorporate these (and possibly other) covariates into the HOMR to create a unique chart-HOMR score. Finally, we found that a geriatrics assessment in the ED was associated with a significant (and notable) decrease in death risk. With these data, we are unable to indicate whether this association is causative. However, these findings indicate that the influence of emergency geriatric assessments on patient survival needs to be explored in more detail.

Several issues about our study should be considered when interpreting its results. First, this was a single-center study and the generalizability of our results to other centers is unknown. However, our study had the largest sample size of all primary data prognostic index validation studies¹ ensuring that our results are, at the very least, internally reliable. In addition, our simple random sample ensured that we studied a broad assortment of patients to be certain that our results are representative of our institution. Second, we used a single abstractor for the study, which could limit the generalizability of our results. However, almost all the data points that were abstracted for our study were very objective.

In summary, our study shows that the HOMR score can be used to accurately predict 1-year death risk using data abstracted from the patient record. These findings will aid in individual patient prognostication for clinicians and researchers.

Disclosure

The authors report no financial conflicts of interest.

Pneumonia is a leading cause of hospitalizations in the U.S., accounting for more than 1.1 million discharges annually.¹ Pneumonia is frequently complicated by hospital readmission, which is costly and potentially avoidable.^2,3 Due to financial penalties imposed on hospitals for higher than expected 30-day readmission rates, there is increasing attention to implementing interventions to reduce readmissions in this population.^4,5 However, because these programs are resource-intensive, interventions are thought to be most cost-effective if they are targeted to high-risk individuals who are most likely to benefit.^6-8

Current pneumonia-specific readmission risk-prediction models that could enable identification of high-risk patients suffer from poor predictive ability, greatly limiting their use, and most were validated among older adults or by using data from single academic medical centers, limiting their generalizability.^9-14 A potential reason for poor predictive accuracy is the omission of known robust clinical predictors of pneumonia-related outcomes, including pneumonia severity of illness and stability on discharge.^15-17 Approaches using electronic health record (EHR) data, which include this clinically granular data, could enable hospitals to more accurately and pragmatically identify high-risk patients during the index hospitalization and enable interventions to be initiated prior to discharge.

An alternative strategy to identifying high-risk patients for readmission is to use a multi-condition risk-prediction model. Developing and implementing models for every condition may be time-consuming and costly. We have derived and validated 2 multi-condition risk-prediction models using EHR data—1 using data from the first day of hospital admission (‘first-day’ model), and the second incorporating data from the entire hospitalization (‘full-stay’ model) to reflect in-hospital complications and clinical stability at discharge.^18,19 However, it is unknown if a multi-condition model for pneumonia would perform as well as a disease-specific model.

This study aimed to develop 2 EHR-based pneumonia-specific readmission risk-prediction models using data routinely collected in clinical practice—a ‘first-day’ and a ‘full-stay’ model—and compare the performance of each model to: 1) one another; 2) the corresponding multi-condition EHR model; and 3) to other potentially useful models in predicting pneumonia readmissions (the Centers for Medicare and Medicaid Services [CMS] pneumonia model, and 2 commonly used pneumonia severity of illness scores validated for predicting mortality). We hypothesized that the pneumonia-specific EHR models would outperform other models; and the full-stay pneumonia-specific model would outperform the first-day pneumonia-specific model.

Study Design, Population, and Data Sources

We conducted an observational study using EHR data collected from 6 hospitals (including safety net, community, teaching, and nonteaching hospitals) in north Texas between November 2009 and October 2010, All hospitals used the Epic EHR (Epic Systems Corporation, Verona, WI). Details of this cohort have been published.^18,19

We included consecutive hospitalizations among adults 18 years and older discharged from any medicine service with principal discharge diagnoses of pneumonia (ICD-9-CM codes 480-483, 485, 486-487), sepsis (ICD-9-CM codes 038, 995.91, 995.92, 785.52), or respiratory failure (ICD-9-CM codes 518.81, 518.82, 518.84, 799.1) when the latter 2 were also accompanied by a secondary diagnosis of pneumonia.²⁰ For individuals with multiple hospitalizations during the study period, we included only the first hospitalization. We excluded individuals who died during the index hospitalization or within 30 days of discharge, were transferred to another acute care facility, or left against medical advice.

Outcomes

The primary outcome was all-cause 30-day readmission, defined as a nonelective hospitalization within 30 days of discharge to any of 75 acute care hospitals within a 100-mile radius of Dallas, ascertained from an all-payer regional hospitalization database.

Predictor Variables for the Pneumonia-Specific Readmission Models

The selection of candidate predictors was informed by our validated multi-condition risk-prediction models using EHR data available within 24 hours of admission (‘first-day’ multi-condition EHR model) or during the entire hospitalization (‘full-stay’ multi-condition EHR model).^18,19 For the pneumonia-specific models, we included all variables in our published multi-condition models as candidate predictors, including sociodemographics, prior utilization, Charlson Comorbidity Index, select laboratory and vital sign abnormalities, length of stay, hospital complications (eg, venous thromboembolism), vital sign instabilities, and disposition status (see Supplemental Table 1 for complete list of variables). We also assessed additional variables specific to pneumonia for inclusion that were: (1) available in the EHR of all participating hospitals; (2) routinely collected or available at the time of admission or discharge; and (3) plausible predictors of adverse outcomes based on literature and clinical expertise. These included select comorbidities (eg, psychiatric conditions, chronic lung disease, history of pneumonia),^10,11,21,22 the pneumonia severity index (PSI),^16,23,24 intensive care unit stay, and receipt of invasive or noninvasive ventilation. We used a modified PSI score because certain data elements were missing. The modified PSI (henceforth referred to as PSI) did not include nursing home residence and included diagnostic codes as proxies for the presence of pleural effusion (ICD-9-CM codes 510, 511.1, and 511.9) and altered mental status (ICD-9-CM codes 780.0X, 780.97, 293.0, 293.1, and 348.3X).

Statistical Analysis

Model Derivation. Candidate predictor variables were classified as available in the EHR within 24 hours of admission and/or at the time of discharge. For example, socioeconomic factors could be ascertained within the first day of hospitalization, whereas length of stay would not be available until the day of discharge. Predictors with missing values were assumed to be normal (less than 1% missing for each variable). Univariate relationships between readmission and each candidate predictor were assessed in the overall cohort using a pre-specified significance threshold of P ≤ 0.10. Significant variables were entered in the respective first-day and full-stay pneumonia-specific multivariable logistic regression models using stepwise-backward selection with a pre-specified significance threshold of P ≤ 0.05. In sensitivity analyses, we alternately derived our models using stepwise-forward selection, as well as stepwise-backward selection minimizing the Bayesian information criterion and Akaike information criterion separately. These alternate modeling strategies yielded identical predictors to our final models.

Model Validation. Model validation was performed using 5-fold cross-validation, with the overall cohort randomly divided into 5 equal-size subsets.²⁵ For each cycle, 4 subsets were used for training to estimate model coefficients, and the fifth subset was used for validation. This cycle was repeated 5 times with each randomly-divided subset used once as the validation set. We repeated this entire process 50 times and averaged the C statistic estimates to derive an optimism-corrected C statistic. Model calibration was assessed qualitatively by comparing predicted to observed probabilities of readmission by quintiles of predicted risk, and with the Hosmer-Lemeshow goodness-of-fit test.

Comparison to Other Models. The main comparisons of the first-day and full-stay pneumonia-specific EHR model performance were to each other and the corresponding multi-condition EHR model.^18,19 The multi-condition EHR models were separately derived and validated within the larger parent cohort from which this study cohort was derived, and outperformed the CMS all-cause model, the HOSPITAL model, and the LACE index.¹⁹ To further triangulate our findings, given the lack of other rigorously validated pneumonia-specific risk-prediction models for readmission,¹⁴ we compared the pneumonia-specific EHR models to the CMS pneumonia model derived from administrative claims data,¹⁰ and 2 commonly used risk-prediction scores for short-term mortality among patients with community-acquired pneumonia, the PSI and CURB-65 scores.¹⁶ Although derived and validated using patient-level data, the CMS model was developed to benchmark hospitals according to hospital-level readmission rates.¹⁰ The CURB-65 score in this study was also modified to include the same altered mental status diagnostic codes according to the modified PSI as a proxy for “confusion.” Both the PSI and CURB-65 scores were calculated using the most abnormal values within the first 24 hours of admission. The ‘updated’ PSI and the ‘updated’ CURB-65 were calculated using the most abnormal values within 24 hours prior to discharge, or the last known observation prior to discharge if no results were recorded within this time period. A complete list of variables for each of the comparison models are shown in Supplemental Table 1.

We assessed model performance by calculating the C statistic, integrated discrimination index, and net reclassification index (NRI) compared to our pneumonia-specific models. The integrated discrimination index is the difference in the mean predicted probability of readmission between patients who were and were not actually readmitted between 2 models, where more positive values suggest improvement in model performance compared to a reference model.²⁶ The NRI is defined as the sum of the net proportions of correctly reclassified persons with and without the event of interest.²⁷ Here, we calculated a category-based NRI to evaluate the performance of pneumonia-specific models in correctly classifying individuals with and without readmissions into the 2 highest readmission risk quintiles vs the lowest 3 risk quintiles compared to other models.²⁷ This pre-specified cutoff is relevant for hospitals interested in identifying the highest risk individuals for targeted intervention.⁷ Finally, we assessed calibration of comparator models in our cohort by comparing predicted probability to observed probability of readmission by quintiles of risk for each model. We conducted all analyses using Stata 12.1 (StataCorp, College Station, Texas). This study was approved by the University of Texas Southwestern Medical Center Institutional Review Board.

Of 1463 index hospitalizations (Supplemental Figure 1), the 30-day all-cause readmission rate was 13.6%. Individuals with a 30-day readmission had markedly different sociodemographic and clinical characteristics compared to those not readmitted (Table 1; see Supplemental Table 2 for additional clinical characteristics).

Derivation, Validation, and Performance of the Pneumonia-Specific Readmission Risk-Prediction Models

The final first-day pneumonia-specific EHR model included 7 variables, including sociodemographic characteristics; prior hospitalizations; thrombocytosis, and PSI (Table 2). The first-day pneumonia-specific model had adequate discrimination (C statistic, 0.695; optimism-corrected C statistic 0.675, 95% confidence interval [CI], 0.667-0.685; Table 3). It also effectively stratified individuals across a broad range of risk (average predicted decile of risk ranged from 4% to 33%; Table 3) and was well calibrated (Supplemental Table 3).

The final full-stay pneumonia-specific EHR readmission model included 8 predictors, including 3 variables from the first-day model (median income, thrombocytosis, and prior hospitalizations; Table 2). The full-stay pneumonia-specific EHR model also included vital sign instabilities on discharge, updated PSI, and disposition status (ie, being discharged with home health or to a post-acute care facility was associated with greater odds of readmission, and hospice with lower odds). The full-stay pneumonia-specific EHR model had good discrimination (C statistic, 0.731; optimism-corrected C statistic, 0.714; 95% CI, 0.706-0.720), and stratified individuals across a broad range of risk (average predicted decile of risk ranged from 3% to 37%; Table 3), and was also well calibrated (Supplemental Table 3).

First-Day Pneumonia-Specific EHR Model vs First-Day Multi-Condition EHR Model

The first-day pneumonia-specific EHR model outperformed the first-day multi-condition EHR model with better discrimination (P = 0.029) and more correctly classified individuals in the top 2 highest risk quintiles vs the bottom 3 risk quintiles (Table 3, Supplemental Table 4, and Supplemental Figure 2A). With respect to calibration, the first-day multi-condition EHR model overestimated risk among the highest quintile risk group compared to the first-day pneumonia-specific EHR model (Figure 1A, 1B).

Full-Stay Pneumonia-Specific EHR Model vs Other Models

The full-stay pneumonia-specific EHR model comparatively outperformed the corresponding full-stay multi-condition EHR model, as well as the first-day pneumonia-specific EHR model, the CMS pneumonia model, the updated PSI, and the updated CURB-65 (Table 3, Supplemental Table 5, Supplemental Table 6, and Supplemental Figures 2B and 2C). Compared to the full-stay multi-condition and first-day pneumonia-specific EHR models, the full-stay pneumonia-specific EHR model had better discrimination, better reclassification (NRI, 0.09 and 0.08, respectively), and was able to stratify individuals across a broader range of readmission risk (Table 3). It also had better calibration in the highest quintile risk group compared to the full-stay multi-condition EHR model (Figure 1C and 1D).

Updated vs First-Day Modified PSI and CURB-65 Scores

The updated PSI was more strongly predictive of readmission than the PSI calculated on the day of admission (Wald test, 9.83; P = 0.002). Each 10-point increase in the updated PSI was associated with a 22% increased odds of readmission vs an 11% increase for the PSI calculated upon admission (Table 2). The improved predictive ability of the updated PSI and CURB-65 scores was also reflected in the superior discrimination and calibration vs the respective first-day pneumonia severity of illness scores (Table 3).

Using routinely available EHR data from 6 diverse hospitals, we developed 2 pneumonia-specific readmission risk-prediction models that aimed to allow hospitals to identify patients hospitalized with pneumonia at high risk for readmission. Overall, we found that a pneumonia-specific model using EHR data from the entire hospitalization outperformed all other models—including the first-day pneumonia-specific model using data present only on admission, our own multi-condition EHR models, and the CMS pneumonia model based on administrative claims data—in all aspects of model performance (discrimination, calibration, and reclassification). We found that socioeconomic status, prior hospitalizations, thrombocytosis, and measures of clinical severity and stability were important predictors of 30-day all-cause readmissions among patients hospitalized with pneumonia. Additionally, an updated discharge PSI score was a stronger independent predictor of readmissions compared to the PSI score calculated upon admission; and inclusion of the updated PSI in our full-stay pneumonia model led to improved prediction of 30-day readmissions.

The marked improvement in performance of the full-stay pneumonia-specific EHR model compared to the first-day pneumonia-specific model suggests that clinical stability and trajectory during hospitalization (as modeled through disposition status, updated PSI, and vital sign instabilities at discharge) are important predictors of 30-day readmission among patients hospitalized for pneumonia, which was not the case for our EHR-based multi-condition models.¹⁹ With the inclusion of these measures, the full-stay pneumonia-specific model correctly reclassified an additional 8% of patients according to their true risk compared to the first-day pneumonia-specific model. One implication of these findings is that hospitals interested in targeting their highest risk individuals with pneumonia for transitional care interventions could do so using the first-day pneumonia-specific EHR model and could refine their targeted strategy at the time of discharge by using the full-stay pneumonia model. This staged risk-prediction strategy would enable hospitals to initiate transitional care interventions for high-risk individuals in the inpatient setting (ie, patient education).⁷ Then, hospitals could enroll both persistent and newly identified high-risk individuals for outpatient interventions (ie, follow-up telephone call) in the immediate post-discharge period, an interval characterized by heightened vulnerability for adverse events,²⁸ based on patients’ illness severity and stability at discharge. This approach can be implemented by hospitals by building these risk-prediction models directly into the EHR, or by extracting EHR data in near real time as our group has done successfully for heart failure.⁷

Another key implication of our study is that, for pneumonia, a disease-specific modeling approach has better predictive ability than using a multi-condition model. Compared to multi-condition models, the first-day and full-stay pneumonia-specific EHR models correctly reclassified an additional 6% and 9% of patients, respectively. Thus, hospitals interested in identifying the highest risk patients with pneumonia for targeted interventions should do so using the disease-specific models, if the costs and resources of doing so are within reach of the healthcare system.

An additional novel finding of our study is the added value of an updated PSI for predicting adverse events. Studies of pneumonia severity of illness scores have calculated the PSI and CURB-65 scores using data present only on admission.^16,24 While our study also confirms that the PSI calculated upon admission is a significant predictor of readmission,^23,29 this study extends this work by showing that an updated PSI score calculated at the time of discharge is an even stronger predictor for readmission, and its inclusion in the model significantly improves risk stratification and prognostication.

Our study was noteworthy for several strengths. First, we used data from a common EHR system, thus potentially allowing for the implementation of the pneumonia-specific models in real time across a number of hospitals. The use of routinely collected data for risk-prediction modeling makes this approach scalable and sustainable, because it obviates the need for burdensome data collection and entry. Second, to our knowledge, this is the first study to measure the additive influence of illness severity and stability at discharge on the readmission risk among patients hospitalized with pneumonia. Third, our study population was derived from 6 hospitals diverse in payer status, age, race/ethnicity, and socioeconomic status. Fourth, our models are less likely to be overfit to the idiosyncrasies of our data given that several predictors included in our final pneumonia-specific models have been associated with readmission in this population, including marital status,^13,30 income,^11,31 prior hospitalizations,^11,13 thrombocytosis,^32-34 and vital sign instabilities on discharge.¹⁷ Lastly, the discrimination of the CMS pneumonia model in our cohort (C statistic, 0.64) closely matched the discrimination observed in 4 independent cohorts (C statistic, 0.63), suggesting adequate generalizability of our study setting and population.^10,12

Our results should be interpreted in the context of several limitations. First, generalizability to other regions beyond north Texas is unknown. Second, although we included a diverse cohort of safety net, community, teaching, and nonteaching hospitals, the pneumonia-specific models were not externally validated in a separate cohort, which may lead to more optimistic estimates of model performance. Third, PSI and CURB-65 scores were modified to use diagnostic codes for altered mental status and pleural effusion, and omitted nursing home residence. Thus, the independent associations for the PSI and CURB-65 scores and their predictive ability are likely attenuated. Fourth, we were unable to include data on medications (antibiotics and steroid use) and outpatient visits, which may influence readmission risk.^2,9,13,35-40 Fifth, we included only the first pneumonia hospitalization per patient in this study. Had we included multiple hospitalizations per patient, we anticipate better model performance for the 2 pneumonia-specific EHR models since prior hospitalization was a robust predictor of readmission.

In conclusion, the full-stay pneumonia-specific EHR readmission risk-prediction model outperformed the first-day pneumonia-specific model, multi-condition EHR models, and the CMS pneumonia model. This suggests that: measures of clinical severity and stability at the time of discharge are important predictors for identifying patients at highest risk for readmission; and that EHR data routinely collected for clinical practice can be used to accurately predict risk of readmission among patients hospitalized for pneumonia.

Acknowledgments

The authors would like to acknowledge Ruben Amarasingham, MD, MBA, president and chief executive officer of Parkland Center for Clinical Innovation, and Ferdinand Velasco, MD, chief health information officer at Texas Health Resources for their assistance in assembling the 6-hospital cohort used in this study.

Disclosures

This work was supported by the Agency for Healthcare Research and Quality-funded UT Southwestern Center for Patient-Centered Outcomes Research (R24 HS022418-01); the Commonwealth Foundation (#20100323); the UT Southwestern KL2 Scholars Program supported by the National Institutes of Health (KL2 TR001103 to ANM and OKN); and the National Center for Advancing Translational Sciences at the National Institute of Health (U54 RFA-TR-12-006 to E.A.H.). The study sponsors had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no financial conflicts of interest to disclose

Pneumonia is a leading cause of hospitalizations in the U.S., accounting for more than 1.1 million discharges annually.¹ Pneumonia is frequently complicated by hospital readmission, which is costly and potentially avoidable.^2,3 Due to financial penalties imposed on hospitals for higher than expected 30-day readmission rates, there is increasing attention to implementing interventions to reduce readmissions in this population.^4,5 However, because these programs are resource-intensive, interventions are thought to be most cost-effective if they are targeted to high-risk individuals who are most likely to benefit.^6-8

Current pneumonia-specific readmission risk-prediction models that could enable identification of high-risk patients suffer from poor predictive ability, greatly limiting their use, and most were validated among older adults or by using data from single academic medical centers, limiting their generalizability.^9-14 A potential reason for poor predictive accuracy is the omission of known robust clinical predictors of pneumonia-related outcomes, including pneumonia severity of illness and stability on discharge.^15-17 Approaches using electronic health record (EHR) data, which include this clinically granular data, could enable hospitals to more accurately and pragmatically identify high-risk patients during the index hospitalization and enable interventions to be initiated prior to discharge.

An alternative strategy to identifying high-risk patients for readmission is to use a multi-condition risk-prediction model. Developing and implementing models for every condition may be time-consuming and costly. We have derived and validated 2 multi-condition risk-prediction models using EHR data—1 using data from the first day of hospital admission (‘first-day’ model), and the second incorporating data from the entire hospitalization (‘full-stay’ model) to reflect in-hospital complications and clinical stability at discharge.^18,19 However, it is unknown if a multi-condition model for pneumonia would perform as well as a disease-specific model.

This study aimed to develop 2 EHR-based pneumonia-specific readmission risk-prediction models using data routinely collected in clinical practice—a ‘first-day’ and a ‘full-stay’ model—and compare the performance of each model to: 1) one another; 2) the corresponding multi-condition EHR model; and 3) to other potentially useful models in predicting pneumonia readmissions (the Centers for Medicare and Medicaid Services [CMS] pneumonia model, and 2 commonly used pneumonia severity of illness scores validated for predicting mortality). We hypothesized that the pneumonia-specific EHR models would outperform other models; and the full-stay pneumonia-specific model would outperform the first-day pneumonia-specific model.

Study Design, Population, and Data Sources

We conducted an observational study using EHR data collected from 6 hospitals (including safety net, community, teaching, and nonteaching hospitals) in north Texas between November 2009 and October 2010, All hospitals used the Epic EHR (Epic Systems Corporation, Verona, WI). Details of this cohort have been published.^18,19

We included consecutive hospitalizations among adults 18 years and older discharged from any medicine service with principal discharge diagnoses of pneumonia (ICD-9-CM codes 480-483, 485, 486-487), sepsis (ICD-9-CM codes 038, 995.91, 995.92, 785.52), or respiratory failure (ICD-9-CM codes 518.81, 518.82, 518.84, 799.1) when the latter 2 were also accompanied by a secondary diagnosis of pneumonia.²⁰ For individuals with multiple hospitalizations during the study period, we included only the first hospitalization. We excluded individuals who died during the index hospitalization or within 30 days of discharge, were transferred to another acute care facility, or left against medical advice.

Outcomes

The primary outcome was all-cause 30-day readmission, defined as a nonelective hospitalization within 30 days of discharge to any of 75 acute care hospitals within a 100-mile radius of Dallas, ascertained from an all-payer regional hospitalization database.

Predictor Variables for the Pneumonia-Specific Readmission Models

The selection of candidate predictors was informed by our validated multi-condition risk-prediction models using EHR data available within 24 hours of admission (‘first-day’ multi-condition EHR model) or during the entire hospitalization (‘full-stay’ multi-condition EHR model).^18,19 For the pneumonia-specific models, we included all variables in our published multi-condition models as candidate predictors, including sociodemographics, prior utilization, Charlson Comorbidity Index, select laboratory and vital sign abnormalities, length of stay, hospital complications (eg, venous thromboembolism), vital sign instabilities, and disposition status (see Supplemental Table 1 for complete list of variables). We also assessed additional variables specific to pneumonia for inclusion that were: (1) available in the EHR of all participating hospitals; (2) routinely collected or available at the time of admission or discharge; and (3) plausible predictors of adverse outcomes based on literature and clinical expertise. These included select comorbidities (eg, psychiatric conditions, chronic lung disease, history of pneumonia),^10,11,21,22 the pneumonia severity index (PSI),^16,23,24 intensive care unit stay, and receipt of invasive or noninvasive ventilation. We used a modified PSI score because certain data elements were missing. The modified PSI (henceforth referred to as PSI) did not include nursing home residence and included diagnostic codes as proxies for the presence of pleural effusion (ICD-9-CM codes 510, 511.1, and 511.9) and altered mental status (ICD-9-CM codes 780.0X, 780.97, 293.0, 293.1, and 348.3X).

Statistical Analysis

Model Derivation. Candidate predictor variables were classified as available in the EHR within 24 hours of admission and/or at the time of discharge. For example, socioeconomic factors could be ascertained within the first day of hospitalization, whereas length of stay would not be available until the day of discharge. Predictors with missing values were assumed to be normal (less than 1% missing for each variable). Univariate relationships between readmission and each candidate predictor were assessed in the overall cohort using a pre-specified significance threshold of P ≤ 0.10. Significant variables were entered in the respective first-day and full-stay pneumonia-specific multivariable logistic regression models using stepwise-backward selection with a pre-specified significance threshold of P ≤ 0.05. In sensitivity analyses, we alternately derived our models using stepwise-forward selection, as well as stepwise-backward selection minimizing the Bayesian information criterion and Akaike information criterion separately. These alternate modeling strategies yielded identical predictors to our final models.

Model Validation. Model validation was performed using 5-fold cross-validation, with the overall cohort randomly divided into 5 equal-size subsets.²⁵ For each cycle, 4 subsets were used for training to estimate model coefficients, and the fifth subset was used for validation. This cycle was repeated 5 times with each randomly-divided subset used once as the validation set. We repeated this entire process 50 times and averaged the C statistic estimates to derive an optimism-corrected C statistic. Model calibration was assessed qualitatively by comparing predicted to observed probabilities of readmission by quintiles of predicted risk, and with the Hosmer-Lemeshow goodness-of-fit test.

Comparison to Other Models. The main comparisons of the first-day and full-stay pneumonia-specific EHR model performance were to each other and the corresponding multi-condition EHR model.^18,19 The multi-condition EHR models were separately derived and validated within the larger parent cohort from which this study cohort was derived, and outperformed the CMS all-cause model, the HOSPITAL model, and the LACE index.¹⁹ To further triangulate our findings, given the lack of other rigorously validated pneumonia-specific risk-prediction models for readmission,¹⁴ we compared the pneumonia-specific EHR models to the CMS pneumonia model derived from administrative claims data,¹⁰ and 2 commonly used risk-prediction scores for short-term mortality among patients with community-acquired pneumonia, the PSI and CURB-65 scores.¹⁶ Although derived and validated using patient-level data, the CMS model was developed to benchmark hospitals according to hospital-level readmission rates.¹⁰ The CURB-65 score in this study was also modified to include the same altered mental status diagnostic codes according to the modified PSI as a proxy for “confusion.” Both the PSI and CURB-65 scores were calculated using the most abnormal values within the first 24 hours of admission. The ‘updated’ PSI and the ‘updated’ CURB-65 were calculated using the most abnormal values within 24 hours prior to discharge, or the last known observation prior to discharge if no results were recorded within this time period. A complete list of variables for each of the comparison models are shown in Supplemental Table 1.

We assessed model performance by calculating the C statistic, integrated discrimination index, and net reclassification index (NRI) compared to our pneumonia-specific models. The integrated discrimination index is the difference in the mean predicted probability of readmission between patients who were and were not actually readmitted between 2 models, where more positive values suggest improvement in model performance compared to a reference model.²⁶ The NRI is defined as the sum of the net proportions of correctly reclassified persons with and without the event of interest.²⁷ Here, we calculated a category-based NRI to evaluate the performance of pneumonia-specific models in correctly classifying individuals with and without readmissions into the 2 highest readmission risk quintiles vs the lowest 3 risk quintiles compared to other models.²⁷ This pre-specified cutoff is relevant for hospitals interested in identifying the highest risk individuals for targeted intervention.⁷ Finally, we assessed calibration of comparator models in our cohort by comparing predicted probability to observed probability of readmission by quintiles of risk for each model. We conducted all analyses using Stata 12.1 (StataCorp, College Station, Texas). This study was approved by the University of Texas Southwestern Medical Center Institutional Review Board.

Of 1463 index hospitalizations (Supplemental Figure 1), the 30-day all-cause readmission rate was 13.6%. Individuals with a 30-day readmission had markedly different sociodemographic and clinical characteristics compared to those not readmitted (Table 1; see Supplemental Table 2 for additional clinical characteristics).

Derivation, Validation, and Performance of the Pneumonia-Specific Readmission Risk-Prediction Models

The final first-day pneumonia-specific EHR model included 7 variables, including sociodemographic characteristics; prior hospitalizations; thrombocytosis, and PSI (Table 2). The first-day pneumonia-specific model had adequate discrimination (C statistic, 0.695; optimism-corrected C statistic 0.675, 95% confidence interval [CI], 0.667-0.685; Table 3). It also effectively stratified individuals across a broad range of risk (average predicted decile of risk ranged from 4% to 33%; Table 3) and was well calibrated (Supplemental Table 3).

The final full-stay pneumonia-specific EHR readmission model included 8 predictors, including 3 variables from the first-day model (median income, thrombocytosis, and prior hospitalizations; Table 2). The full-stay pneumonia-specific EHR model also included vital sign instabilities on discharge, updated PSI, and disposition status (ie, being discharged with home health or to a post-acute care facility was associated with greater odds of readmission, and hospice with lower odds). The full-stay pneumonia-specific EHR model had good discrimination (C statistic, 0.731; optimism-corrected C statistic, 0.714; 95% CI, 0.706-0.720), and stratified individuals across a broad range of risk (average predicted decile of risk ranged from 3% to 37%; Table 3), and was also well calibrated (Supplemental Table 3).

First-Day Pneumonia-Specific EHR Model vs First-Day Multi-Condition EHR Model

The first-day pneumonia-specific EHR model outperformed the first-day multi-condition EHR model with better discrimination (P = 0.029) and more correctly classified individuals in the top 2 highest risk quintiles vs the bottom 3 risk quintiles (Table 3, Supplemental Table 4, and Supplemental Figure 2A). With respect to calibration, the first-day multi-condition EHR model overestimated risk among the highest quintile risk group compared to the first-day pneumonia-specific EHR model (Figure 1A, 1B).

Full-Stay Pneumonia-Specific EHR Model vs Other Models

The full-stay pneumonia-specific EHR model comparatively outperformed the corresponding full-stay multi-condition EHR model, as well as the first-day pneumonia-specific EHR model, the CMS pneumonia model, the updated PSI, and the updated CURB-65 (Table 3, Supplemental Table 5, Supplemental Table 6, and Supplemental Figures 2B and 2C). Compared to the full-stay multi-condition and first-day pneumonia-specific EHR models, the full-stay pneumonia-specific EHR model had better discrimination, better reclassification (NRI, 0.09 and 0.08, respectively), and was able to stratify individuals across a broader range of readmission risk (Table 3). It also had better calibration in the highest quintile risk group compared to the full-stay multi-condition EHR model (Figure 1C and 1D).

Updated vs First-Day Modified PSI and CURB-65 Scores

The updated PSI was more strongly predictive of readmission than the PSI calculated on the day of admission (Wald test, 9.83; P = 0.002). Each 10-point increase in the updated PSI was associated with a 22% increased odds of readmission vs an 11% increase for the PSI calculated upon admission (Table 2). The improved predictive ability of the updated PSI and CURB-65 scores was also reflected in the superior discrimination and calibration vs the respective first-day pneumonia severity of illness scores (Table 3).

Using routinely available EHR data from 6 diverse hospitals, we developed 2 pneumonia-specific readmission risk-prediction models that aimed to allow hospitals to identify patients hospitalized with pneumonia at high risk for readmission. Overall, we found that a pneumonia-specific model using EHR data from the entire hospitalization outperformed all other models—including the first-day pneumonia-specific model using data present only on admission, our own multi-condition EHR models, and the CMS pneumonia model based on administrative claims data—in all aspects of model performance (discrimination, calibration, and reclassification). We found that socioeconomic status, prior hospitalizations, thrombocytosis, and measures of clinical severity and stability were important predictors of 30-day all-cause readmissions among patients hospitalized with pneumonia. Additionally, an updated discharge PSI score was a stronger independent predictor of readmissions compared to the PSI score calculated upon admission; and inclusion of the updated PSI in our full-stay pneumonia model led to improved prediction of 30-day readmissions.

The marked improvement in performance of the full-stay pneumonia-specific EHR model compared to the first-day pneumonia-specific model suggests that clinical stability and trajectory during hospitalization (as modeled through disposition status, updated PSI, and vital sign instabilities at discharge) are important predictors of 30-day readmission among patients hospitalized for pneumonia, which was not the case for our EHR-based multi-condition models.¹⁹ With the inclusion of these measures, the full-stay pneumonia-specific model correctly reclassified an additional 8% of patients according to their true risk compared to the first-day pneumonia-specific model. One implication of these findings is that hospitals interested in targeting their highest risk individuals with pneumonia for transitional care interventions could do so using the first-day pneumonia-specific EHR model and could refine their targeted strategy at the time of discharge by using the full-stay pneumonia model. This staged risk-prediction strategy would enable hospitals to initiate transitional care interventions for high-risk individuals in the inpatient setting (ie, patient education).⁷ Then, hospitals could enroll both persistent and newly identified high-risk individuals for outpatient interventions (ie, follow-up telephone call) in the immediate post-discharge period, an interval characterized by heightened vulnerability for adverse events,²⁸ based on patients’ illness severity and stability at discharge. This approach can be implemented by hospitals by building these risk-prediction models directly into the EHR, or by extracting EHR data in near real time as our group has done successfully for heart failure.⁷

Another key implication of our study is that, for pneumonia, a disease-specific modeling approach has better predictive ability than using a multi-condition model. Compared to multi-condition models, the first-day and full-stay pneumonia-specific EHR models correctly reclassified an additional 6% and 9% of patients, respectively. Thus, hospitals interested in identifying the highest risk patients with pneumonia for targeted interventions should do so using the disease-specific models, if the costs and resources of doing so are within reach of the healthcare system.

An additional novel finding of our study is the added value of an updated PSI for predicting adverse events. Studies of pneumonia severity of illness scores have calculated the PSI and CURB-65 scores using data present only on admission.^16,24 While our study also confirms that the PSI calculated upon admission is a significant predictor of readmission,^23,29 this study extends this work by showing that an updated PSI score calculated at the time of discharge is an even stronger predictor for readmission, and its inclusion in the model significantly improves risk stratification and prognostication.

Our study was noteworthy for several strengths. First, we used data from a common EHR system, thus potentially allowing for the implementation of the pneumonia-specific models in real time across a number of hospitals. The use of routinely collected data for risk-prediction modeling makes this approach scalable and sustainable, because it obviates the need for burdensome data collection and entry. Second, to our knowledge, this is the first study to measure the additive influence of illness severity and stability at discharge on the readmission risk among patients hospitalized with pneumonia. Third, our study population was derived from 6 hospitals diverse in payer status, age, race/ethnicity, and socioeconomic status. Fourth, our models are less likely to be overfit to the idiosyncrasies of our data given that several predictors included in our final pneumonia-specific models have been associated with readmission in this population, including marital status,^13,30 income,^11,31 prior hospitalizations,^11,13 thrombocytosis,^32-34 and vital sign instabilities on discharge.¹⁷ Lastly, the discrimination of the CMS pneumonia model in our cohort (C statistic, 0.64) closely matched the discrimination observed in 4 independent cohorts (C statistic, 0.63), suggesting adequate generalizability of our study setting and population.^10,12

Our results should be interpreted in the context of several limitations. First, generalizability to other regions beyond north Texas is unknown. Second, although we included a diverse cohort of safety net, community, teaching, and nonteaching hospitals, the pneumonia-specific models were not externally validated in a separate cohort, which may lead to more optimistic estimates of model performance. Third, PSI and CURB-65 scores were modified to use diagnostic codes for altered mental status and pleural effusion, and omitted nursing home residence. Thus, the independent associations for the PSI and CURB-65 scores and their predictive ability are likely attenuated. Fourth, we were unable to include data on medications (antibiotics and steroid use) and outpatient visits, which may influence readmission risk.^2,9,13,35-40 Fifth, we included only the first pneumonia hospitalization per patient in this study. Had we included multiple hospitalizations per patient, we anticipate better model performance for the 2 pneumonia-specific EHR models since prior hospitalization was a robust predictor of readmission.

In conclusion, the full-stay pneumonia-specific EHR readmission risk-prediction model outperformed the first-day pneumonia-specific model, multi-condition EHR models, and the CMS pneumonia model. This suggests that: measures of clinical severity and stability at the time of discharge are important predictors for identifying patients at highest risk for readmission; and that EHR data routinely collected for clinical practice can be used to accurately predict risk of readmission among patients hospitalized for pneumonia.

Acknowledgments

The authors would like to acknowledge Ruben Amarasingham, MD, MBA, president and chief executive officer of Parkland Center for Clinical Innovation, and Ferdinand Velasco, MD, chief health information officer at Texas Health Resources for their assistance in assembling the 6-hospital cohort used in this study.

Disclosures

This work was supported by the Agency for Healthcare Research and Quality-funded UT Southwestern Center for Patient-Centered Outcomes Research (R24 HS022418-01); the Commonwealth Foundation (#20100323); the UT Southwestern KL2 Scholars Program supported by the National Institutes of Health (KL2 TR001103 to ANM and OKN); and the National Center for Advancing Translational Sciences at the National Institute of Health (U54 RFA-TR-12-006 to E.A.H.). The study sponsors had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no financial conflicts of interest to disclose

Pneumonia is a leading cause of hospitalizations in the U.S., accounting for more than 1.1 million discharges annually.¹ Pneumonia is frequently complicated by hospital readmission, which is costly and potentially avoidable.^2,3 Due to financial penalties imposed on hospitals for higher than expected 30-day readmission rates, there is increasing attention to implementing interventions to reduce readmissions in this population.^4,5 However, because these programs are resource-intensive, interventions are thought to be most cost-effective if they are targeted to high-risk individuals who are most likely to benefit.^6-8

Current pneumonia-specific readmission risk-prediction models that could enable identification of high-risk patients suffer from poor predictive ability, greatly limiting their use, and most were validated among older adults or by using data from single academic medical centers, limiting their generalizability.^9-14 A potential reason for poor predictive accuracy is the omission of known robust clinical predictors of pneumonia-related outcomes, including pneumonia severity of illness and stability on discharge.^15-17 Approaches using electronic health record (EHR) data, which include this clinically granular data, could enable hospitals to more accurately and pragmatically identify high-risk patients during the index hospitalization and enable interventions to be initiated prior to discharge.

An alternative strategy to identifying high-risk patients for readmission is to use a multi-condition risk-prediction model. Developing and implementing models for every condition may be time-consuming and costly. We have derived and validated 2 multi-condition risk-prediction models using EHR data—1 using data from the first day of hospital admission (‘first-day’ model), and the second incorporating data from the entire hospitalization (‘full-stay’ model) to reflect in-hospital complications and clinical stability at discharge.^18,19 However, it is unknown if a multi-condition model for pneumonia would perform as well as a disease-specific model.

This study aimed to develop 2 EHR-based pneumonia-specific readmission risk-prediction models using data routinely collected in clinical practice—a ‘first-day’ and a ‘full-stay’ model—and compare the performance of each model to: 1) one another; 2) the corresponding multi-condition EHR model; and 3) to other potentially useful models in predicting pneumonia readmissions (the Centers for Medicare and Medicaid Services [CMS] pneumonia model, and 2 commonly used pneumonia severity of illness scores validated for predicting mortality). We hypothesized that the pneumonia-specific EHR models would outperform other models; and the full-stay pneumonia-specific model would outperform the first-day pneumonia-specific model.

Study Design, Population, and Data Sources

We conducted an observational study using EHR data collected from 6 hospitals (including safety net, community, teaching, and nonteaching hospitals) in north Texas between November 2009 and October 2010, All hospitals used the Epic EHR (Epic Systems Corporation, Verona, WI). Details of this cohort have been published.^18,19

We included consecutive hospitalizations among adults 18 years and older discharged from any medicine service with principal discharge diagnoses of pneumonia (ICD-9-CM codes 480-483, 485, 486-487), sepsis (ICD-9-CM codes 038, 995.91, 995.92, 785.52), or respiratory failure (ICD-9-CM codes 518.81, 518.82, 518.84, 799.1) when the latter 2 were also accompanied by a secondary diagnosis of pneumonia.²⁰ For individuals with multiple hospitalizations during the study period, we included only the first hospitalization. We excluded individuals who died during the index hospitalization or within 30 days of discharge, were transferred to another acute care facility, or left against medical advice.

Outcomes

The primary outcome was all-cause 30-day readmission, defined as a nonelective hospitalization within 30 days of discharge to any of 75 acute care hospitals within a 100-mile radius of Dallas, ascertained from an all-payer regional hospitalization database.

Predictor Variables for the Pneumonia-Specific Readmission Models

The selection of candidate predictors was informed by our validated multi-condition risk-prediction models using EHR data available within 24 hours of admission (‘first-day’ multi-condition EHR model) or during the entire hospitalization (‘full-stay’ multi-condition EHR model).^18,19 For the pneumonia-specific models, we included all variables in our published multi-condition models as candidate predictors, including sociodemographics, prior utilization, Charlson Comorbidity Index, select laboratory and vital sign abnormalities, length of stay, hospital complications (eg, venous thromboembolism), vital sign instabilities, and disposition status (see Supplemental Table 1 for complete list of variables). We also assessed additional variables specific to pneumonia for inclusion that were: (1) available in the EHR of all participating hospitals; (2) routinely collected or available at the time of admission or discharge; and (3) plausible predictors of adverse outcomes based on literature and clinical expertise. These included select comorbidities (eg, psychiatric conditions, chronic lung disease, history of pneumonia),^10,11,21,22 the pneumonia severity index (PSI),^16,23,24 intensive care unit stay, and receipt of invasive or noninvasive ventilation. We used a modified PSI score because certain data elements were missing. The modified PSI (henceforth referred to as PSI) did not include nursing home residence and included diagnostic codes as proxies for the presence of pleural effusion (ICD-9-CM codes 510, 511.1, and 511.9) and altered mental status (ICD-9-CM codes 780.0X, 780.97, 293.0, 293.1, and 348.3X).

Statistical Analysis

Model Derivation. Candidate predictor variables were classified as available in the EHR within 24 hours of admission and/or at the time of discharge. For example, socioeconomic factors could be ascertained within the first day of hospitalization, whereas length of stay would not be available until the day of discharge. Predictors with missing values were assumed to be normal (less than 1% missing for each variable). Univariate relationships between readmission and each candidate predictor were assessed in the overall cohort using a pre-specified significance threshold of P ≤ 0.10. Significant variables were entered in the respective first-day and full-stay pneumonia-specific multivariable logistic regression models using stepwise-backward selection with a pre-specified significance threshold of P ≤ 0.05. In sensitivity analyses, we alternately derived our models using stepwise-forward selection, as well as stepwise-backward selection minimizing the Bayesian information criterion and Akaike information criterion separately. These alternate modeling strategies yielded identical predictors to our final models.

Model Validation. Model validation was performed using 5-fold cross-validation, with the overall cohort randomly divided into 5 equal-size subsets.²⁵ For each cycle, 4 subsets were used for training to estimate model coefficients, and the fifth subset was used for validation. This cycle was repeated 5 times with each randomly-divided subset used once as the validation set. We repeated this entire process 50 times and averaged the C statistic estimates to derive an optimism-corrected C statistic. Model calibration was assessed qualitatively by comparing predicted to observed probabilities of readmission by quintiles of predicted risk, and with the Hosmer-Lemeshow goodness-of-fit test.

Comparison to Other Models. The main comparisons of the first-day and full-stay pneumonia-specific EHR model performance were to each other and the corresponding multi-condition EHR model.^18,19 The multi-condition EHR models were separately derived and validated within the larger parent cohort from which this study cohort was derived, and outperformed the CMS all-cause model, the HOSPITAL model, and the LACE index.¹⁹ To further triangulate our findings, given the lack of other rigorously validated pneumonia-specific risk-prediction models for readmission,¹⁴ we compared the pneumonia-specific EHR models to the CMS pneumonia model derived from administrative claims data,¹⁰ and 2 commonly used risk-prediction scores for short-term mortality among patients with community-acquired pneumonia, the PSI and CURB-65 scores.¹⁶ Although derived and validated using patient-level data, the CMS model was developed to benchmark hospitals according to hospital-level readmission rates.¹⁰ The CURB-65 score in this study was also modified to include the same altered mental status diagnostic codes according to the modified PSI as a proxy for “confusion.” Both the PSI and CURB-65 scores were calculated using the most abnormal values within the first 24 hours of admission. The ‘updated’ PSI and the ‘updated’ CURB-65 were calculated using the most abnormal values within 24 hours prior to discharge, or the last known observation prior to discharge if no results were recorded within this time period. A complete list of variables for each of the comparison models are shown in Supplemental Table 1.

We assessed model performance by calculating the C statistic, integrated discrimination index, and net reclassification index (NRI) compared to our pneumonia-specific models. The integrated discrimination index is the difference in the mean predicted probability of readmission between patients who were and were not actually readmitted between 2 models, where more positive values suggest improvement in model performance compared to a reference model.²⁶ The NRI is defined as the sum of the net proportions of correctly reclassified persons with and without the event of interest.²⁷ Here, we calculated a category-based NRI to evaluate the performance of pneumonia-specific models in correctly classifying individuals with and without readmissions into the 2 highest readmission risk quintiles vs the lowest 3 risk quintiles compared to other models.²⁷ This pre-specified cutoff is relevant for hospitals interested in identifying the highest risk individuals for targeted intervention.⁷ Finally, we assessed calibration of comparator models in our cohort by comparing predicted probability to observed probability of readmission by quintiles of risk for each model. We conducted all analyses using Stata 12.1 (StataCorp, College Station, Texas). This study was approved by the University of Texas Southwestern Medical Center Institutional Review Board.

Of 1463 index hospitalizations (Supplemental Figure 1), the 30-day all-cause readmission rate was 13.6%. Individuals with a 30-day readmission had markedly different sociodemographic and clinical characteristics compared to those not readmitted (Table 1; see Supplemental Table 2 for additional clinical characteristics).

Derivation, Validation, and Performance of the Pneumonia-Specific Readmission Risk-Prediction Models

The final first-day pneumonia-specific EHR model included 7 variables, including sociodemographic characteristics; prior hospitalizations; thrombocytosis, and PSI (Table 2). The first-day pneumonia-specific model had adequate discrimination (C statistic, 0.695; optimism-corrected C statistic 0.675, 95% confidence interval [CI], 0.667-0.685; Table 3). It also effectively stratified individuals across a broad range of risk (average predicted decile of risk ranged from 4% to 33%; Table 3) and was well calibrated (Supplemental Table 3).

The final full-stay pneumonia-specific EHR readmission model included 8 predictors, including 3 variables from the first-day model (median income, thrombocytosis, and prior hospitalizations; Table 2). The full-stay pneumonia-specific EHR model also included vital sign instabilities on discharge, updated PSI, and disposition status (ie, being discharged with home health or to a post-acute care facility was associated with greater odds of readmission, and hospice with lower odds). The full-stay pneumonia-specific EHR model had good discrimination (C statistic, 0.731; optimism-corrected C statistic, 0.714; 95% CI, 0.706-0.720), and stratified individuals across a broad range of risk (average predicted decile of risk ranged from 3% to 37%; Table 3), and was also well calibrated (Supplemental Table 3).

First-Day Pneumonia-Specific EHR Model vs First-Day Multi-Condition EHR Model

The first-day pneumonia-specific EHR model outperformed the first-day multi-condition EHR model with better discrimination (P = 0.029) and more correctly classified individuals in the top 2 highest risk quintiles vs the bottom 3 risk quintiles (Table 3, Supplemental Table 4, and Supplemental Figure 2A). With respect to calibration, the first-day multi-condition EHR model overestimated risk among the highest quintile risk group compared to the first-day pneumonia-specific EHR model (Figure 1A, 1B).

Full-Stay Pneumonia-Specific EHR Model vs Other Models

The full-stay pneumonia-specific EHR model comparatively outperformed the corresponding full-stay multi-condition EHR model, as well as the first-day pneumonia-specific EHR model, the CMS pneumonia model, the updated PSI, and the updated CURB-65 (Table 3, Supplemental Table 5, Supplemental Table 6, and Supplemental Figures 2B and 2C). Compared to the full-stay multi-condition and first-day pneumonia-specific EHR models, the full-stay pneumonia-specific EHR model had better discrimination, better reclassification (NRI, 0.09 and 0.08, respectively), and was able to stratify individuals across a broader range of readmission risk (Table 3). It also had better calibration in the highest quintile risk group compared to the full-stay multi-condition EHR model (Figure 1C and 1D).

Updated vs First-Day Modified PSI and CURB-65 Scores

The updated PSI was more strongly predictive of readmission than the PSI calculated on the day of admission (Wald test, 9.83; P = 0.002). Each 10-point increase in the updated PSI was associated with a 22% increased odds of readmission vs an 11% increase for the PSI calculated upon admission (Table 2). The improved predictive ability of the updated PSI and CURB-65 scores was also reflected in the superior discrimination and calibration vs the respective first-day pneumonia severity of illness scores (Table 3).

Using routinely available EHR data from 6 diverse hospitals, we developed 2 pneumonia-specific readmission risk-prediction models that aimed to allow hospitals to identify patients hospitalized with pneumonia at high risk for readmission. Overall, we found that a pneumonia-specific model using EHR data from the entire hospitalization outperformed all other models—including the first-day pneumonia-specific model using data present only on admission, our own multi-condition EHR models, and the CMS pneumonia model based on administrative claims data—in all aspects of model performance (discrimination, calibration, and reclassification). We found that socioeconomic status, prior hospitalizations, thrombocytosis, and measures of clinical severity and stability were important predictors of 30-day all-cause readmissions among patients hospitalized with pneumonia. Additionally, an updated discharge PSI score was a stronger independent predictor of readmissions compared to the PSI score calculated upon admission; and inclusion of the updated PSI in our full-stay pneumonia model led to improved prediction of 30-day readmissions.

The marked improvement in performance of the full-stay pneumonia-specific EHR model compared to the first-day pneumonia-specific model suggests that clinical stability and trajectory during hospitalization (as modeled through disposition status, updated PSI, and vital sign instabilities at discharge) are important predictors of 30-day readmission among patients hospitalized for pneumonia, which was not the case for our EHR-based multi-condition models.¹⁹ With the inclusion of these measures, the full-stay pneumonia-specific model correctly reclassified an additional 8% of patients according to their true risk compared to the first-day pneumonia-specific model. One implication of these findings is that hospitals interested in targeting their highest risk individuals with pneumonia for transitional care interventions could do so using the first-day pneumonia-specific EHR model and could refine their targeted strategy at the time of discharge by using the full-stay pneumonia model. This staged risk-prediction strategy would enable hospitals to initiate transitional care interventions for high-risk individuals in the inpatient setting (ie, patient education).⁷ Then, hospitals could enroll both persistent and newly identified high-risk individuals for outpatient interventions (ie, follow-up telephone call) in the immediate post-discharge period, an interval characterized by heightened vulnerability for adverse events,²⁸ based on patients’ illness severity and stability at discharge. This approach can be implemented by hospitals by building these risk-prediction models directly into the EHR, or by extracting EHR data in near real time as our group has done successfully for heart failure.⁷

Another key implication of our study is that, for pneumonia, a disease-specific modeling approach has better predictive ability than using a multi-condition model. Compared to multi-condition models, the first-day and full-stay pneumonia-specific EHR models correctly reclassified an additional 6% and 9% of patients, respectively. Thus, hospitals interested in identifying the highest risk patients with pneumonia for targeted interventions should do so using the disease-specific models, if the costs and resources of doing so are within reach of the healthcare system.

An additional novel finding of our study is the added value of an updated PSI for predicting adverse events. Studies of pneumonia severity of illness scores have calculated the PSI and CURB-65 scores using data present only on admission.^16,24 While our study also confirms that the PSI calculated upon admission is a significant predictor of readmission,^23,29 this study extends this work by showing that an updated PSI score calculated at the time of discharge is an even stronger predictor for readmission, and its inclusion in the model significantly improves risk stratification and prognostication.

Our study was noteworthy for several strengths. First, we used data from a common EHR system, thus potentially allowing for the implementation of the pneumonia-specific models in real time across a number of hospitals. The use of routinely collected data for risk-prediction modeling makes this approach scalable and sustainable, because it obviates the need for burdensome data collection and entry. Second, to our knowledge, this is the first study to measure the additive influence of illness severity and stability at discharge on the readmission risk among patients hospitalized with pneumonia. Third, our study population was derived from 6 hospitals diverse in payer status, age, race/ethnicity, and socioeconomic status. Fourth, our models are less likely to be overfit to the idiosyncrasies of our data given that several predictors included in our final pneumonia-specific models have been associated with readmission in this population, including marital status,^13,30 income,^11,31 prior hospitalizations,^11,13 thrombocytosis,^32-34 and vital sign instabilities on discharge.¹⁷ Lastly, the discrimination of the CMS pneumonia model in our cohort (C statistic, 0.64) closely matched the discrimination observed in 4 independent cohorts (C statistic, 0.63), suggesting adequate generalizability of our study setting and population.^10,12

Our results should be interpreted in the context of several limitations. First, generalizability to other regions beyond north Texas is unknown. Second, although we included a diverse cohort of safety net, community, teaching, and nonteaching hospitals, the pneumonia-specific models were not externally validated in a separate cohort, which may lead to more optimistic estimates of model performance. Third, PSI and CURB-65 scores were modified to use diagnostic codes for altered mental status and pleural effusion, and omitted nursing home residence. Thus, the independent associations for the PSI and CURB-65 scores and their predictive ability are likely attenuated. Fourth, we were unable to include data on medications (antibiotics and steroid use) and outpatient visits, which may influence readmission risk.^2,9,13,35-40 Fifth, we included only the first pneumonia hospitalization per patient in this study. Had we included multiple hospitalizations per patient, we anticipate better model performance for the 2 pneumonia-specific EHR models since prior hospitalization was a robust predictor of readmission.

In conclusion, the full-stay pneumonia-specific EHR readmission risk-prediction model outperformed the first-day pneumonia-specific model, multi-condition EHR models, and the CMS pneumonia model. This suggests that: measures of clinical severity and stability at the time of discharge are important predictors for identifying patients at highest risk for readmission; and that EHR data routinely collected for clinical practice can be used to accurately predict risk of readmission among patients hospitalized for pneumonia.

Acknowledgments

The authors would like to acknowledge Ruben Amarasingham, MD, MBA, president and chief executive officer of Parkland Center for Clinical Innovation, and Ferdinand Velasco, MD, chief health information officer at Texas Health Resources for their assistance in assembling the 6-hospital cohort used in this study.

Disclosures

This work was supported by the Agency for Healthcare Research and Quality-funded UT Southwestern Center for Patient-Centered Outcomes Research (R24 HS022418-01); the Commonwealth Foundation (#20100323); the UT Southwestern KL2 Scholars Program supported by the National Institutes of Health (KL2 TR001103 to ANM and OKN); and the National Center for Advancing Translational Sciences at the National Institute of Health (U54 RFA-TR-12-006 to E.A.H.). The study sponsors had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no financial conflicts of interest to disclose

Patients typically show signs and symptoms of deterioration hours to days prior to cardiorespiratory arrest.^1,2 The rate of inhospital cardiorespiratory arrest (CRA) requiring cardiopulmonary resuscitation is estimated to be 0.174 per bed per year in the United States.³ After CRA, survival to discharge is estimated to be as low as 18%.^3,4 Efforts to predict and prevent arrest could prove beneficial.^1,2

Rapid response systems (RRS) have been proposed as a means of identifying clinical deterioration and facilitating a timely response. These systems were designed to bring clinicians with critical care expertise to the bedside to prevent unnecessary deaths. They typically include an afferent limb (detects deteriorating patients), an efferent limb (responds to calls and acts to avoid further deterioration), and administrative and data analysis limbs.^5,6 Automatic provision of recommendations and computer-based systems are desirable components of the afferent limb of the detection system.⁶ Both are independent predictors of improved clinical practices for clinical decision support systems.⁷ However, the existing early warning scores (EWS) may not be ready for automation due to low positive predictive values (PPV) and sensitivities.⁸

It is possible that the low discriminatory accuracy of the published EWS may be secondary to the use of aggregate patient populations for derivation of scores. We hypothesized that these EWS perform differently in medical and in surgical subpopulations. Also, the EWS need to be tested in a time-dependent manner to serve as a realistic clinical support tool for hospitalized patients.

STUDY AIM

The aim of this study was to evaluate the differential performance of widely used EWS in medical vs surgical patients.

Site

The study was conducted in an academic center with 2 hospitals in Southeastern Minnesota totaling approximately 1500 general care nonintensive care unit (ICU) beds. The Mayo Clinic Institutional Review Board approved the research proposal.

Subjects

Our retrospective cohort was comprised of all adult inpatients discharged from 2 academic hospitals between January 1, 2011 and December 31, 2011 who spent any time in a general care (non-ICU) unit. We excluded patients younger than 18 years, psychiatric or rehabilitation inpatients, those without research authorization, and patients admitted for research purposes.

Study patients were divided into medical and surgical cohorts. Hospitalizations were considered surgical if patients had surgery at any time during their hospital stay according to billing data. A trigger was an instance in which a patient met the conditions of a specific rule (score/vital sign exceeded the published/defined threshold).

A resuscitation call was defined as a call for cardiopulmonary resuscitation when a patient has a CRA.

An event was an occurrence of 1 of the following in a general care setting: unplanned transfer to the ICU, resuscitation call, or RRS activation.

The RRS activation criteria consisted of an “acute and persistent change” in any 1 or more of the following: oxygen saturations less than 90%, heart rate less than 40 or greater than 130 beats/minute, systolic blood pressure less than 90 mm Hg, or respiratory rate less than 10 or greater than 28 breaths/minute. The RRS activation requires health provider action; they are not electronically generated. Nurses and physicians may also activate the RRS if they are concerned about a patient, even if calling criteria are not met. This is in contrast to the EWS analyzed, which are aggregate composites of multiple parameters. However, whether or not a derangement in vital signs is considered an “acute and persistent change” still involves clinical judgment. Any movement from a general care bed to an ICU bed, or from a general care bed to a procedure area, and from there to an ICU, was considered unplanned. Transfers to the ICU directly from the emergency department or operating room (OR) were not considered as an unplanned transfer and were not included in the analyses.

Coverage time was the period observed for events after a rule was triggered. In this analysis, a coverage time of 24 hours was considered, with a 1-hour look-back. A trigger was counted as a true positive if an event occurred during the following 24 hours. The 1-hour look-back was included to take into account the nursing clinical process of prioritizing a call to the RRS followed by documentation of the altered vital signs that prompted the call.

An episode was the continuous time on the general care floor within a hospitalization, excluding times when a patient was in the OR or ICU. For example, if a patient was admitted to a general bed on a surgery floor, subsequently went to the OR, and then returned to the surgery floor, the 2 episodes were considered separate: the time on the floor before surgery, and the time on the floor after surgery.

Assessment of implementation of RRS in our hospitals showed a significant drop in the failure-to-rescue rate (issues considered related to delay or failure to identify or intervene appropriately when a patient was deteriorating, as identified through mortality review) and a decrease in non-ICU mortality.^9,10 This suggests that our current process captures many of the relevant episodes of acute deterioration when a rapid response team is needed and supports using RRS activation as outcomes.

Data Sources

We developed a time-stamped longitudinal database of patient data from the electronic health record, including vital signs, laboratory test results, demographics (age, sex), administrative data (including length of stay), comorbidities, resuscitation code status, location in hospital, and at the minute level throughout each patient’s hospital stay. Physiologically impossible values (eg, blood pressures of 1200 mm Hg) were considered entered in error and eliminated from the database. Time spent in the OR or ICU was excluded because RRS activation would not be applied in these already highly monitored areas. SAS Statistical software (SAS Institute Inc. Cary, North Carolina) was used for database creation.

We applied the current RRS calling criteria in our institution and calculated the Kirkland score,¹¹ along with some of the most widely used early warning scores:¹² Modified Early Warning System (MEWS),¹³ Standardized Early Warning Scoring System (SEWS),¹⁴ Global Modified Early Warning Score (GMEWS),¹⁵ Worthing physiologic scoring system,¹⁶ National Early Warning Score (NEWS),¹⁷ and VitaPAC Early Warning Score (ViEWS).¹⁸ Published thresholds for these scores were used to create rule triggers in the data. Once a trigger was created to calculate the number of false positives and true positives, all subsequent triggers were ignored until the end of the episode or until 24 hours elapsed. We calculated triggers in a rolling fashion throughout the episodes of care. The EWS score was updated every time a new parameter was entered into the analytical electronic health record, and the most recent value for each was used to calculate the score. SAS statistical software was used for calculation of scores and identification of outcomes.

For our analysis, events were treated as dependent variables, and triggers were independent variables. We calculated the score for each EWS to the minute level throughout our retrospective database. If the score for a specific EWS was higher than the published/recommended threshold for that EWS, an alert was considered to have been issued, and the patient was followed for 24 hours. If the patient had an event in the subsequent 24 hours, or 1 hour before (1-hour look-back), the alert was considered a true positive; if not, a false positive. Events that were not preceded by an alert were false negatives, and 24-hour intervals without either an alert or an event were considered true negatives. This simulation exercise was performed for each EWS in both subcohorts (medical and surgical). Clusters of RRS calls followed by transfers to the ICU within 3 hours were considered as a single adverse event (RRS calls, as it was the first event to occur) to avoid double counting. We have described how well this simulation methodology,⁸ correlates with results from prospective studies.¹⁹

Statistical Analysis

To calculate whether results were statistically significant for subgroups, a jackknife method of calculating variance²⁰ was used. The jackknife method calculates variance by repeating the calculations of the statistic leaving out 1 sample at a time. In our case, we repeated the calculation of sensitivity and PPV leaving out 1 patient at a time. Once the simulation method had been run and the false/true positives/negatives had been assigned, calculation of each metric (PPV and sensitivity) was repeated for n subsamples, each leaving out 1 patient. The variance was calculated and 2 Student t tests were performed for each EWS: 1 for PPV and another for sensitivity. SAS statistical software v 9.3 was used for the simulation analysis; R statistical software v 3.0.2 (The R Foundation, Vienna, Austria) was used for the calculation of the statistical significance of results. A univariable analysis was also performed to assess the sensitivity and PPVs for the published thresholds of the most common variables in each EWS: respiratory rate, systolic blood pressure, heart rate, temperature, and mental status as measured by the modified Richmond Agitation Sedation Score.²¹

The initial cohort included 60,020 hospitalizations, of which the following were excluded: 2751 because of a lack of appropriate research authorization; 6433 because the patients were younger than 18 years; 2129 as psychiatric admissions; 284 as rehabilitation admissions; 872 as research purposes-only admissions; and 1185 because the patient was never in a general care bed (eg, they were either admitted directly to the ICU, or they were admitted for an outpatient surgical procedure and spent time in the postanesthesia care unit).

Table 1 summarizes patient and trigger characteristics, overall and by subgroup. The final cohort included 75,240 total episodes in 46,366 hospitalizations, from 34,898 unique patients, of which 48.7% were male. There were 23,831 medical and 22,535 surgical hospitalizations. Median length of episode was 2 days both for medical and surgical patients. Median length of stay was 3 days, both for medical and for surgical patients.

We hypothesized that EWS may perform differently when applied to medical rather than surgical patients. Studies had not analyzed this in a time-dependent manner,^24-26 which limited the applicability of the results.⁸

All analyzed scores performed better in medical patients than in surgical patients (Figure). This could reflect a behavioral difference by the teams on surgical and medical floors in the decision to activate the RRS, or a bias of the clinicians who designed the scores (mostly nonsurgeons). The difference could also mean that physiological deteriorations are intrinsically different in patients who have undergone anesthesia and surgery. For example, in surgical patients, a bleeding episode is more likely to be the cause of their physiological deterioration, or the lingering effects of anesthesia could mask underlying deterioration. Such patients would benefit from scores where variables such as heart rate, blood pressure, or hemoglobin had more influence.

When comparing the different scores, it was much easier for a patient to meet the alerting score with the Worthing score than with GMEWS. In the Worthing score, a respiratory rate greater than 22 breaths per minute, or a systolic blood pressure less than 100 mm Hg, already meet alerting criteria. Similar vital signs result in 0 and 1 points (respectively) in GMEWS, far from its alerting score of 5. This reflects the intrinsic tradeoff of EWS: as the threshold for considering a patient “at risk” drops, not only does the number of true positives (and the sensitivity) increase, but also the number of false positives, thus lowering the PPV.

However, none of the scores analyzed were considered to perform well based on their PPV and sensitivity, particularly in the surgical subpopulation. Focusing on another metric, the area under the receiver operator curve can give misleadingly optimistic results.^24,27 However, the extremely low prevalence of acute physiological deterioration can produce low PPVs even when specificity seems acceptable, which is why it is important to evaluate PPV directly.²⁸

To use EWS effectively to activate RRS, they need to be combined with clinical judgment to avoid high levels of false alerts, particularly in surgical patients. It has been reported that RRS is activated only 30% of the time a patient meets RRS calling criteria.²⁹ While there may be cultural characteristics inhibiting the decision to call,³⁰ our study hints at another explanation: if RRS was activated every time a patient met calling criteria based on the scores analyzed, the number of RRS calls would be very high and difficult to manage. So health providers may be doing the right thing when “filtering” RRS calls and not applying the criteria strictly, but in conjunction with clinical judgment.

A limitation of any study like this is how to define “acute physiological deterioration.” We defined an event as recognized episodes of acute physiological deterioration that are signaled by escalations of care (eg, RRS, resuscitation calls, or transfers to an ICU) or unexpected death. By definition, our calculated PPV is affected by clinicians’ recognition of clinical deteriorations. This definition, common in the literature, has the limitation of potentially underestimating EWS’ performance by missing some events that are resolved by the primary care team without an escalation of care. However, we believe our interpretation is not unreasonable since the purpose of EWS is to trigger escalations of care in a timely fashion. Prospective studies could define an event in a way that is less affected by the clinicians’ judgment.

Regarding patient demographics, age was similar between the 2 groups (average, 58.2 years for medical vs 58.9 years for surgical), and there was only a small difference in gender ratios (45.1% male in the medical vs 51.4% in the surgical group). These differences are unlikely to have affected the results significantly, but unknown differences in demographics or other patient characteristics between groups may account for differences in score performance between surgical and medical patients.

Several of the EWS analyzed had overlapping trigger criteria with our own RRS activation criteria (although as single-parameter triggers and not as aggregate). To test how these potential biases could affect our results, we performed a post hoc sensitivity analysis eliminating calls to the RRS as an outcome (so using the alternative outcome of unexpected transfers to the ICU and resuscitation calls). The results are similar to those of our main analysis, with all analyzed scores having lower sensitivity and PPV in surgical hospitalizations when compared to medical hospitalizations.

Our study suggests that, to optimize detection of physiological deterioration events, EWS should try to take into account different patient types, with the most basic distinction being surgical vs medical. This tailoring will make EWS more complex, and less suited for paper-based calculation, but new electronic health records are increasingly able to incorporate decision support, and some EWS have been developed for electronic calculation only. Of particular interest in this regard is the score developed by Escobar et al,³¹ which groups patients into categories according to the reason for admission, and calculates a different subscore based on that category. While the score by Escobar et al. does not split patients based on medical or surgical status, a more general interpretation of our results suggests that a score may be more accurate if it classifies patients into subgroups with different subscores. This seems to be confirmed by the fact that the score by Escobar et al performs better than MEWS.²⁸ Unfortunately, the paper describing it does not provide enough detail to use it in our database.

A recent systematic review showed increasing evidence that RRS may be effective in reducing CRAs occurring in a non-ICU setting and, more important, overall inhospital mortality.³² While differing implementation strategies (eg, different length of the educational effort, changes in the frequency of vital signs monitoring) can impact the success of such an initiative, it has been speculated that the afferent limb (which often includes an EWS) might be the most critical part of the system.³³ Our results show that the most widely used EWS perform significantly worse on surgical patients, and suggest that a way to improve the accuracy of EWS would be to tailor the risk calculation to different patient subgroups (eg, medical and surgical patients). Plausible next steps would be to demonstrate that tailoring risk calculation to medical and surgical patients separately can improve risk predictions and accuracy of EWS.

Disclosure

The authors report no financial conflicts of interest.

Patients typically show signs and symptoms of deterioration hours to days prior to cardiorespiratory arrest.^1,2 The rate of inhospital cardiorespiratory arrest (CRA) requiring cardiopulmonary resuscitation is estimated to be 0.174 per bed per year in the United States.³ After CRA, survival to discharge is estimated to be as low as 18%.^3,4 Efforts to predict and prevent arrest could prove beneficial.^1,2

Rapid response systems (RRS) have been proposed as a means of identifying clinical deterioration and facilitating a timely response. These systems were designed to bring clinicians with critical care expertise to the bedside to prevent unnecessary deaths. They typically include an afferent limb (detects deteriorating patients), an efferent limb (responds to calls and acts to avoid further deterioration), and administrative and data analysis limbs.^5,6 Automatic provision of recommendations and computer-based systems are desirable components of the afferent limb of the detection system.⁶ Both are independent predictors of improved clinical practices for clinical decision support systems.⁷ However, the existing early warning scores (EWS) may not be ready for automation due to low positive predictive values (PPV) and sensitivities.⁸

It is possible that the low discriminatory accuracy of the published EWS may be secondary to the use of aggregate patient populations for derivation of scores. We hypothesized that these EWS perform differently in medical and in surgical subpopulations. Also, the EWS need to be tested in a time-dependent manner to serve as a realistic clinical support tool for hospitalized patients.

STUDY AIM

The aim of this study was to evaluate the differential performance of widely used EWS in medical vs surgical patients.

Site

The study was conducted in an academic center with 2 hospitals in Southeastern Minnesota totaling approximately 1500 general care nonintensive care unit (ICU) beds. The Mayo Clinic Institutional Review Board approved the research proposal.

Subjects

Our retrospective cohort was comprised of all adult inpatients discharged from 2 academic hospitals between January 1, 2011 and December 31, 2011 who spent any time in a general care (non-ICU) unit. We excluded patients younger than 18 years, psychiatric or rehabilitation inpatients, those without research authorization, and patients admitted for research purposes.

Study patients were divided into medical and surgical cohorts. Hospitalizations were considered surgical if patients had surgery at any time during their hospital stay according to billing data. A trigger was an instance in which a patient met the conditions of a specific rule (score/vital sign exceeded the published/defined threshold).

A resuscitation call was defined as a call for cardiopulmonary resuscitation when a patient has a CRA.

An event was an occurrence of 1 of the following in a general care setting: unplanned transfer to the ICU, resuscitation call, or RRS activation.

The RRS activation criteria consisted of an “acute and persistent change” in any 1 or more of the following: oxygen saturations less than 90%, heart rate less than 40 or greater than 130 beats/minute, systolic blood pressure less than 90 mm Hg, or respiratory rate less than 10 or greater than 28 breaths/minute. The RRS activation requires health provider action; they are not electronically generated. Nurses and physicians may also activate the RRS if they are concerned about a patient, even if calling criteria are not met. This is in contrast to the EWS analyzed, which are aggregate composites of multiple parameters. However, whether or not a derangement in vital signs is considered an “acute and persistent change” still involves clinical judgment. Any movement from a general care bed to an ICU bed, or from a general care bed to a procedure area, and from there to an ICU, was considered unplanned. Transfers to the ICU directly from the emergency department or operating room (OR) were not considered as an unplanned transfer and were not included in the analyses.

Coverage time was the period observed for events after a rule was triggered. In this analysis, a coverage time of 24 hours was considered, with a 1-hour look-back. A trigger was counted as a true positive if an event occurred during the following 24 hours. The 1-hour look-back was included to take into account the nursing clinical process of prioritizing a call to the RRS followed by documentation of the altered vital signs that prompted the call.

An episode was the continuous time on the general care floor within a hospitalization, excluding times when a patient was in the OR or ICU. For example, if a patient was admitted to a general bed on a surgery floor, subsequently went to the OR, and then returned to the surgery floor, the 2 episodes were considered separate: the time on the floor before surgery, and the time on the floor after surgery.

Assessment of implementation of RRS in our hospitals showed a significant drop in the failure-to-rescue rate (issues considered related to delay or failure to identify or intervene appropriately when a patient was deteriorating, as identified through mortality review) and a decrease in non-ICU mortality.^9,10 This suggests that our current process captures many of the relevant episodes of acute deterioration when a rapid response team is needed and supports using RRS activation as outcomes.

Data Sources

We developed a time-stamped longitudinal database of patient data from the electronic health record, including vital signs, laboratory test results, demographics (age, sex), administrative data (including length of stay), comorbidities, resuscitation code status, location in hospital, and at the minute level throughout each patient’s hospital stay. Physiologically impossible values (eg, blood pressures of 1200 mm Hg) were considered entered in error and eliminated from the database. Time spent in the OR or ICU was excluded because RRS activation would not be applied in these already highly monitored areas. SAS Statistical software (SAS Institute Inc. Cary, North Carolina) was used for database creation.

We applied the current RRS calling criteria in our institution and calculated the Kirkland score,¹¹ along with some of the most widely used early warning scores:¹² Modified Early Warning System (MEWS),¹³ Standardized Early Warning Scoring System (SEWS),¹⁴ Global Modified Early Warning Score (GMEWS),¹⁵ Worthing physiologic scoring system,¹⁶ National Early Warning Score (NEWS),¹⁷ and VitaPAC Early Warning Score (ViEWS).¹⁸ Published thresholds for these scores were used to create rule triggers in the data. Once a trigger was created to calculate the number of false positives and true positives, all subsequent triggers were ignored until the end of the episode or until 24 hours elapsed. We calculated triggers in a rolling fashion throughout the episodes of care. The EWS score was updated every time a new parameter was entered into the analytical electronic health record, and the most recent value for each was used to calculate the score. SAS statistical software was used for calculation of scores and identification of outcomes.

For our analysis, events were treated as dependent variables, and triggers were independent variables. We calculated the score for each EWS to the minute level throughout our retrospective database. If the score for a specific EWS was higher than the published/recommended threshold for that EWS, an alert was considered to have been issued, and the patient was followed for 24 hours. If the patient had an event in the subsequent 24 hours, or 1 hour before (1-hour look-back), the alert was considered a true positive; if not, a false positive. Events that were not preceded by an alert were false negatives, and 24-hour intervals without either an alert or an event were considered true negatives. This simulation exercise was performed for each EWS in both subcohorts (medical and surgical). Clusters of RRS calls followed by transfers to the ICU within 3 hours were considered as a single adverse event (RRS calls, as it was the first event to occur) to avoid double counting. We have described how well this simulation methodology,⁸ correlates with results from prospective studies.¹⁹

Statistical Analysis

To calculate whether results were statistically significant for subgroups, a jackknife method of calculating variance²⁰ was used. The jackknife method calculates variance by repeating the calculations of the statistic leaving out 1 sample at a time. In our case, we repeated the calculation of sensitivity and PPV leaving out 1 patient at a time. Once the simulation method had been run and the false/true positives/negatives had been assigned, calculation of each metric (PPV and sensitivity) was repeated for n subsamples, each leaving out 1 patient. The variance was calculated and 2 Student t tests were performed for each EWS: 1 for PPV and another for sensitivity. SAS statistical software v 9.3 was used for the simulation analysis; R statistical software v 3.0.2 (The R Foundation, Vienna, Austria) was used for the calculation of the statistical significance of results. A univariable analysis was also performed to assess the sensitivity and PPVs for the published thresholds of the most common variables in each EWS: respiratory rate, systolic blood pressure, heart rate, temperature, and mental status as measured by the modified Richmond Agitation Sedation Score.²¹

The initial cohort included 60,020 hospitalizations, of which the following were excluded: 2751 because of a lack of appropriate research authorization; 6433 because the patients were younger than 18 years; 2129 as psychiatric admissions; 284 as rehabilitation admissions; 872 as research purposes-only admissions; and 1185 because the patient was never in a general care bed (eg, they were either admitted directly to the ICU, or they were admitted for an outpatient surgical procedure and spent time in the postanesthesia care unit).

Table 1 summarizes patient and trigger characteristics, overall and by subgroup. The final cohort included 75,240 total episodes in 46,366 hospitalizations, from 34,898 unique patients, of which 48.7% were male. There were 23,831 medical and 22,535 surgical hospitalizations. Median length of episode was 2 days both for medical and surgical patients. Median length of stay was 3 days, both for medical and for surgical patients.

We hypothesized that EWS may perform differently when applied to medical rather than surgical patients. Studies had not analyzed this in a time-dependent manner,^24-26 which limited the applicability of the results.⁸

All analyzed scores performed better in medical patients than in surgical patients (Figure). This could reflect a behavioral difference by the teams on surgical and medical floors in the decision to activate the RRS, or a bias of the clinicians who designed the scores (mostly nonsurgeons). The difference could also mean that physiological deteriorations are intrinsically different in patients who have undergone anesthesia and surgery. For example, in surgical patients, a bleeding episode is more likely to be the cause of their physiological deterioration, or the lingering effects of anesthesia could mask underlying deterioration. Such patients would benefit from scores where variables such as heart rate, blood pressure, or hemoglobin had more influence.

When comparing the different scores, it was much easier for a patient to meet the alerting score with the Worthing score than with GMEWS. In the Worthing score, a respiratory rate greater than 22 breaths per minute, or a systolic blood pressure less than 100 mm Hg, already meet alerting criteria. Similar vital signs result in 0 and 1 points (respectively) in GMEWS, far from its alerting score of 5. This reflects the intrinsic tradeoff of EWS: as the threshold for considering a patient “at risk” drops, not only does the number of true positives (and the sensitivity) increase, but also the number of false positives, thus lowering the PPV.

However, none of the scores analyzed were considered to perform well based on their PPV and sensitivity, particularly in the surgical subpopulation. Focusing on another metric, the area under the receiver operator curve can give misleadingly optimistic results.^24,27 However, the extremely low prevalence of acute physiological deterioration can produce low PPVs even when specificity seems acceptable, which is why it is important to evaluate PPV directly.²⁸

To use EWS effectively to activate RRS, they need to be combined with clinical judgment to avoid high levels of false alerts, particularly in surgical patients. It has been reported that RRS is activated only 30% of the time a patient meets RRS calling criteria.²⁹ While there may be cultural characteristics inhibiting the decision to call,³⁰ our study hints at another explanation: if RRS was activated every time a patient met calling criteria based on the scores analyzed, the number of RRS calls would be very high and difficult to manage. So health providers may be doing the right thing when “filtering” RRS calls and not applying the criteria strictly, but in conjunction with clinical judgment.

A limitation of any study like this is how to define “acute physiological deterioration.” We defined an event as recognized episodes of acute physiological deterioration that are signaled by escalations of care (eg, RRS, resuscitation calls, or transfers to an ICU) or unexpected death. By definition, our calculated PPV is affected by clinicians’ recognition of clinical deteriorations. This definition, common in the literature, has the limitation of potentially underestimating EWS’ performance by missing some events that are resolved by the primary care team without an escalation of care. However, we believe our interpretation is not unreasonable since the purpose of EWS is to trigger escalations of care in a timely fashion. Prospective studies could define an event in a way that is less affected by the clinicians’ judgment.

Regarding patient demographics, age was similar between the 2 groups (average, 58.2 years for medical vs 58.9 years for surgical), and there was only a small difference in gender ratios (45.1% male in the medical vs 51.4% in the surgical group). These differences are unlikely to have affected the results significantly, but unknown differences in demographics or other patient characteristics between groups may account for differences in score performance between surgical and medical patients.

Several of the EWS analyzed had overlapping trigger criteria with our own RRS activation criteria (although as single-parameter triggers and not as aggregate). To test how these potential biases could affect our results, we performed a post hoc sensitivity analysis eliminating calls to the RRS as an outcome (so using the alternative outcome of unexpected transfers to the ICU and resuscitation calls). The results are similar to those of our main analysis, with all analyzed scores having lower sensitivity and PPV in surgical hospitalizations when compared to medical hospitalizations.

Our study suggests that, to optimize detection of physiological deterioration events, EWS should try to take into account different patient types, with the most basic distinction being surgical vs medical. This tailoring will make EWS more complex, and less suited for paper-based calculation, but new electronic health records are increasingly able to incorporate decision support, and some EWS have been developed for electronic calculation only. Of particular interest in this regard is the score developed by Escobar et al,³¹ which groups patients into categories according to the reason for admission, and calculates a different subscore based on that category. While the score by Escobar et al. does not split patients based on medical or surgical status, a more general interpretation of our results suggests that a score may be more accurate if it classifies patients into subgroups with different subscores. This seems to be confirmed by the fact that the score by Escobar et al performs better than MEWS.²⁸ Unfortunately, the paper describing it does not provide enough detail to use it in our database.

A recent systematic review showed increasing evidence that RRS may be effective in reducing CRAs occurring in a non-ICU setting and, more important, overall inhospital mortality.³² While differing implementation strategies (eg, different length of the educational effort, changes in the frequency of vital signs monitoring) can impact the success of such an initiative, it has been speculated that the afferent limb (which often includes an EWS) might be the most critical part of the system.³³ Our results show that the most widely used EWS perform significantly worse on surgical patients, and suggest that a way to improve the accuracy of EWS would be to tailor the risk calculation to different patient subgroups (eg, medical and surgical patients). Plausible next steps would be to demonstrate that tailoring risk calculation to medical and surgical patients separately can improve risk predictions and accuracy of EWS.

Disclosure

The authors report no financial conflicts of interest.

Patients typically show signs and symptoms of deterioration hours to days prior to cardiorespiratory arrest.^1,2 The rate of inhospital cardiorespiratory arrest (CRA) requiring cardiopulmonary resuscitation is estimated to be 0.174 per bed per year in the United States.³ After CRA, survival to discharge is estimated to be as low as 18%.^3,4 Efforts to predict and prevent arrest could prove beneficial.^1,2

Rapid response systems (RRS) have been proposed as a means of identifying clinical deterioration and facilitating a timely response. These systems were designed to bring clinicians with critical care expertise to the bedside to prevent unnecessary deaths. They typically include an afferent limb (detects deteriorating patients), an efferent limb (responds to calls and acts to avoid further deterioration), and administrative and data analysis limbs.^5,6 Automatic provision of recommendations and computer-based systems are desirable components of the afferent limb of the detection system.⁶ Both are independent predictors of improved clinical practices for clinical decision support systems.⁷ However, the existing early warning scores (EWS) may not be ready for automation due to low positive predictive values (PPV) and sensitivities.⁸

It is possible that the low discriminatory accuracy of the published EWS may be secondary to the use of aggregate patient populations for derivation of scores. We hypothesized that these EWS perform differently in medical and in surgical subpopulations. Also, the EWS need to be tested in a time-dependent manner to serve as a realistic clinical support tool for hospitalized patients.

STUDY AIM

The aim of this study was to evaluate the differential performance of widely used EWS in medical vs surgical patients.

Site

The study was conducted in an academic center with 2 hospitals in Southeastern Minnesota totaling approximately 1500 general care nonintensive care unit (ICU) beds. The Mayo Clinic Institutional Review Board approved the research proposal.

Subjects

Our retrospective cohort was comprised of all adult inpatients discharged from 2 academic hospitals between January 1, 2011 and December 31, 2011 who spent any time in a general care (non-ICU) unit. We excluded patients younger than 18 years, psychiatric or rehabilitation inpatients, those without research authorization, and patients admitted for research purposes.

Study patients were divided into medical and surgical cohorts. Hospitalizations were considered surgical if patients had surgery at any time during their hospital stay according to billing data. A trigger was an instance in which a patient met the conditions of a specific rule (score/vital sign exceeded the published/defined threshold).

A resuscitation call was defined as a call for cardiopulmonary resuscitation when a patient has a CRA.

An event was an occurrence of 1 of the following in a general care setting: unplanned transfer to the ICU, resuscitation call, or RRS activation.

The RRS activation criteria consisted of an “acute and persistent change” in any 1 or more of the following: oxygen saturations less than 90%, heart rate less than 40 or greater than 130 beats/minute, systolic blood pressure less than 90 mm Hg, or respiratory rate less than 10 or greater than 28 breaths/minute. The RRS activation requires health provider action; they are not electronically generated. Nurses and physicians may also activate the RRS if they are concerned about a patient, even if calling criteria are not met. This is in contrast to the EWS analyzed, which are aggregate composites of multiple parameters. However, whether or not a derangement in vital signs is considered an “acute and persistent change” still involves clinical judgment. Any movement from a general care bed to an ICU bed, or from a general care bed to a procedure area, and from there to an ICU, was considered unplanned. Transfers to the ICU directly from the emergency department or operating room (OR) were not considered as an unplanned transfer and were not included in the analyses.

Coverage time was the period observed for events after a rule was triggered. In this analysis, a coverage time of 24 hours was considered, with a 1-hour look-back. A trigger was counted as a true positive if an event occurred during the following 24 hours. The 1-hour look-back was included to take into account the nursing clinical process of prioritizing a call to the RRS followed by documentation of the altered vital signs that prompted the call.

An episode was the continuous time on the general care floor within a hospitalization, excluding times when a patient was in the OR or ICU. For example, if a patient was admitted to a general bed on a surgery floor, subsequently went to the OR, and then returned to the surgery floor, the 2 episodes were considered separate: the time on the floor before surgery, and the time on the floor after surgery.

Assessment of implementation of RRS in our hospitals showed a significant drop in the failure-to-rescue rate (issues considered related to delay or failure to identify or intervene appropriately when a patient was deteriorating, as identified through mortality review) and a decrease in non-ICU mortality.^9,10 This suggests that our current process captures many of the relevant episodes of acute deterioration when a rapid response team is needed and supports using RRS activation as outcomes.

Data Sources

We developed a time-stamped longitudinal database of patient data from the electronic health record, including vital signs, laboratory test results, demographics (age, sex), administrative data (including length of stay), comorbidities, resuscitation code status, location in hospital, and at the minute level throughout each patient’s hospital stay. Physiologically impossible values (eg, blood pressures of 1200 mm Hg) were considered entered in error and eliminated from the database. Time spent in the OR or ICU was excluded because RRS activation would not be applied in these already highly monitored areas. SAS Statistical software (SAS Institute Inc. Cary, North Carolina) was used for database creation.

We applied the current RRS calling criteria in our institution and calculated the Kirkland score,¹¹ along with some of the most widely used early warning scores:¹² Modified Early Warning System (MEWS),¹³ Standardized Early Warning Scoring System (SEWS),¹⁴ Global Modified Early Warning Score (GMEWS),¹⁵ Worthing physiologic scoring system,¹⁶ National Early Warning Score (NEWS),¹⁷ and VitaPAC Early Warning Score (ViEWS).¹⁸ Published thresholds for these scores were used to create rule triggers in the data. Once a trigger was created to calculate the number of false positives and true positives, all subsequent triggers were ignored until the end of the episode or until 24 hours elapsed. We calculated triggers in a rolling fashion throughout the episodes of care. The EWS score was updated every time a new parameter was entered into the analytical electronic health record, and the most recent value for each was used to calculate the score. SAS statistical software was used for calculation of scores and identification of outcomes.

For our analysis, events were treated as dependent variables, and triggers were independent variables. We calculated the score for each EWS to the minute level throughout our retrospective database. If the score for a specific EWS was higher than the published/recommended threshold for that EWS, an alert was considered to have been issued, and the patient was followed for 24 hours. If the patient had an event in the subsequent 24 hours, or 1 hour before (1-hour look-back), the alert was considered a true positive; if not, a false positive. Events that were not preceded by an alert were false negatives, and 24-hour intervals without either an alert or an event were considered true negatives. This simulation exercise was performed for each EWS in both subcohorts (medical and surgical). Clusters of RRS calls followed by transfers to the ICU within 3 hours were considered as a single adverse event (RRS calls, as it was the first event to occur) to avoid double counting. We have described how well this simulation methodology,⁸ correlates with results from prospective studies.¹⁹

Statistical Analysis

To calculate whether results were statistically significant for subgroups, a jackknife method of calculating variance²⁰ was used. The jackknife method calculates variance by repeating the calculations of the statistic leaving out 1 sample at a time. In our case, we repeated the calculation of sensitivity and PPV leaving out 1 patient at a time. Once the simulation method had been run and the false/true positives/negatives had been assigned, calculation of each metric (PPV and sensitivity) was repeated for n subsamples, each leaving out 1 patient. The variance was calculated and 2 Student t tests were performed for each EWS: 1 for PPV and another for sensitivity. SAS statistical software v 9.3 was used for the simulation analysis; R statistical software v 3.0.2 (The R Foundation, Vienna, Austria) was used for the calculation of the statistical significance of results. A univariable analysis was also performed to assess the sensitivity and PPVs for the published thresholds of the most common variables in each EWS: respiratory rate, systolic blood pressure, heart rate, temperature, and mental status as measured by the modified Richmond Agitation Sedation Score.²¹

The initial cohort included 60,020 hospitalizations, of which the following were excluded: 2751 because of a lack of appropriate research authorization; 6433 because the patients were younger than 18 years; 2129 as psychiatric admissions; 284 as rehabilitation admissions; 872 as research purposes-only admissions; and 1185 because the patient was never in a general care bed (eg, they were either admitted directly to the ICU, or they were admitted for an outpatient surgical procedure and spent time in the postanesthesia care unit).

Table 1 summarizes patient and trigger characteristics, overall and by subgroup. The final cohort included 75,240 total episodes in 46,366 hospitalizations, from 34,898 unique patients, of which 48.7% were male. There were 23,831 medical and 22,535 surgical hospitalizations. Median length of episode was 2 days both for medical and surgical patients. Median length of stay was 3 days, both for medical and for surgical patients.

We hypothesized that EWS may perform differently when applied to medical rather than surgical patients. Studies had not analyzed this in a time-dependent manner,^24-26 which limited the applicability of the results.⁸

All analyzed scores performed better in medical patients than in surgical patients (Figure). This could reflect a behavioral difference by the teams on surgical and medical floors in the decision to activate the RRS, or a bias of the clinicians who designed the scores (mostly nonsurgeons). The difference could also mean that physiological deteriorations are intrinsically different in patients who have undergone anesthesia and surgery. For example, in surgical patients, a bleeding episode is more likely to be the cause of their physiological deterioration, or the lingering effects of anesthesia could mask underlying deterioration. Such patients would benefit from scores where variables such as heart rate, blood pressure, or hemoglobin had more influence.

When comparing the different scores, it was much easier for a patient to meet the alerting score with the Worthing score than with GMEWS. In the Worthing score, a respiratory rate greater than 22 breaths per minute, or a systolic blood pressure less than 100 mm Hg, already meet alerting criteria. Similar vital signs result in 0 and 1 points (respectively) in GMEWS, far from its alerting score of 5. This reflects the intrinsic tradeoff of EWS: as the threshold for considering a patient “at risk” drops, not only does the number of true positives (and the sensitivity) increase, but also the number of false positives, thus lowering the PPV.

However, none of the scores analyzed were considered to perform well based on their PPV and sensitivity, particularly in the surgical subpopulation. Focusing on another metric, the area under the receiver operator curve can give misleadingly optimistic results.^24,27 However, the extremely low prevalence of acute physiological deterioration can produce low PPVs even when specificity seems acceptable, which is why it is important to evaluate PPV directly.²⁸

To use EWS effectively to activate RRS, they need to be combined with clinical judgment to avoid high levels of false alerts, particularly in surgical patients. It has been reported that RRS is activated only 30% of the time a patient meets RRS calling criteria.²⁹ While there may be cultural characteristics inhibiting the decision to call,³⁰ our study hints at another explanation: if RRS was activated every time a patient met calling criteria based on the scores analyzed, the number of RRS calls would be very high and difficult to manage. So health providers may be doing the right thing when “filtering” RRS calls and not applying the criteria strictly, but in conjunction with clinical judgment.

A limitation of any study like this is how to define “acute physiological deterioration.” We defined an event as recognized episodes of acute physiological deterioration that are signaled by escalations of care (eg, RRS, resuscitation calls, or transfers to an ICU) or unexpected death. By definition, our calculated PPV is affected by clinicians’ recognition of clinical deteriorations. This definition, common in the literature, has the limitation of potentially underestimating EWS’ performance by missing some events that are resolved by the primary care team without an escalation of care. However, we believe our interpretation is not unreasonable since the purpose of EWS is to trigger escalations of care in a timely fashion. Prospective studies could define an event in a way that is less affected by the clinicians’ judgment.

Regarding patient demographics, age was similar between the 2 groups (average, 58.2 years for medical vs 58.9 years for surgical), and there was only a small difference in gender ratios (45.1% male in the medical vs 51.4% in the surgical group). These differences are unlikely to have affected the results significantly, but unknown differences in demographics or other patient characteristics between groups may account for differences in score performance between surgical and medical patients.

Several of the EWS analyzed had overlapping trigger criteria with our own RRS activation criteria (although as single-parameter triggers and not as aggregate). To test how these potential biases could affect our results, we performed a post hoc sensitivity analysis eliminating calls to the RRS as an outcome (so using the alternative outcome of unexpected transfers to the ICU and resuscitation calls). The results are similar to those of our main analysis, with all analyzed scores having lower sensitivity and PPV in surgical hospitalizations when compared to medical hospitalizations.

Our study suggests that, to optimize detection of physiological deterioration events, EWS should try to take into account different patient types, with the most basic distinction being surgical vs medical. This tailoring will make EWS more complex, and less suited for paper-based calculation, but new electronic health records are increasingly able to incorporate decision support, and some EWS have been developed for electronic calculation only. Of particular interest in this regard is the score developed by Escobar et al,³¹ which groups patients into categories according to the reason for admission, and calculates a different subscore based on that category. While the score by Escobar et al. does not split patients based on medical or surgical status, a more general interpretation of our results suggests that a score may be more accurate if it classifies patients into subgroups with different subscores. This seems to be confirmed by the fact that the score by Escobar et al performs better than MEWS.²⁸ Unfortunately, the paper describing it does not provide enough detail to use it in our database.

A recent systematic review showed increasing evidence that RRS may be effective in reducing CRAs occurring in a non-ICU setting and, more important, overall inhospital mortality.³² While differing implementation strategies (eg, different length of the educational effort, changes in the frequency of vital signs monitoring) can impact the success of such an initiative, it has been speculated that the afferent limb (which often includes an EWS) might be the most critical part of the system.³³ Our results show that the most widely used EWS perform significantly worse on surgical patients, and suggest that a way to improve the accuracy of EWS would be to tailor the risk calculation to different patient subgroups (eg, medical and surgical patients). Plausible next steps would be to demonstrate that tailoring risk calculation to medical and surgical patients separately can improve risk predictions and accuracy of EWS.

Disclosure

The authors report no financial conflicts of interest.

Take-Home Points

• The location of a bicortical defect in proximity to a tibia plate does not appear to affect the torsional stiffness or torsional failure strength of the bone.

• External fixator pin placement should be based on considerations other than the potential for creating a distal stress riser after definitive fracture management.

A stress riser in cortical bone may be considered any abrupt change in the contour or consistency of the hollow structure, such as a surface defect, that not only weakens the bone but concentrates stresses at that transition point.¹ A cortical defect that is 20% of the bone diameter is associated with a 34% decrease in torsional strength, thus representing a “stress riser.”² High-energy and complex tibia fractures are often provisionally stabilized with external fixation that gives the soft tissues time to recover before definitive fracture fixation. Pin diameter for a medium-size tibia external fixator typically is 5.0 mm, resulting in a 10-mm defect in bicortical placement. Therefore, any tibia with a diameter of <50 mm is at risk for a stress riser fracture.

Although it had been established that sizable cortical defects can decrease the torsional strength of long bone,² the effect of a plate in close proximity to a defect secondary to an empty external-fixator pin site on torsional strength has not been determined. We conducted a study to evaluate this effect. The null hypothesis was there would be no difference in tibia torsional strength attributable to varying the proximity of a tibia midshaft plate to a 5.0-mm bicortical defect.

Methods

Forty fourth-generation, medium-size left composite tibias (Pacific Research Laboratories) were divided into 8 groups of 5 bones (Figure 1).

Torsion testing to failure was performed for all specimens in a manner similar to that described by Gardner and colleagues.³ Impression molds for the composite tibia constructed from polymethylmethacrylate encased the superior and distal ends, leaving 25.5 cm of exposed midshaft. This allowed the composites to be rigidly clamped into a materials testing system (858 Mini-Bionix; MTS) equipped with a 100.0-Nm torsional load cell (Figure 2).

Results

Graphical results for torsional stiffness are presented in Figure 3. R² for all stiffness calculations was >0.99.

Discussion

Many tibia fractures require provisional stabilization with an external fixator that spans the knee, because of the high-energy nature of the injury or other, higher-priority polytrauma concerns. When the patient or injury is suitable for definitive fixation, the external fixator typically is removed in favor of internal fixation with a plate and screws. Depending on the nature and location of the fracture and the subsequent plate, the empty cortical pin-site defects, often lying at varying distances from the distal end of the plate, can potentially serve as stress risers for fracture.⁴

Other studies have evaluated long-bone cortical defects biomechanically^1,2,4 and clinically,^5-7 and multiple studies have been conducted on the effects of plates on long-bone strength for fracture stabilization.^8-13 The present study evaluated the torsional strength of long bones in the presence of a bicortical defect and the proximity of the defect to a plate. There were no differences in stiffness or failure load between any of the groups of plated and unplated fourth-generation composite tibias tested to failure in torsion with varying distal bicortical defects. Hypothetically, one would expect the torsional stiffness of these specimens to increase with the mere addition of a metallic diaphyseal plate. However, this study demonstrated that the addition of a plate did not affect the torsional stiffness or strength of the tibias. Clinically, it is common practice to place external fixator pins as far as possible outside the planned incision site for definitive fracture fixation. Thus, we also hypothesized that the presence of a bicortical pin-site defect and its proximity to the plate would alter the torsional strength of the tibia specimens, and that the distal pin-site defect’s location farthest from the plate would exhibit greater strength, but this did not occur. Although other studies have shown that the presence of bicortical defects decreases the strength of long bones, we were unable to quantify this decrease because the 2 intact groups of composites, plated and unplated, survived failure testing.

This study had several limitations, first being the use of composite tibias as opposed to human cadaver bone. Although fourth-generation composite bone models have been validated as a suitable and accurate biomechanical substitute for cadaver specimens,¹⁴ anatomical variations in cadaver tibias may transfer forces differently through plates, screws, and distal pin sites. In order to test plated specimens against the unplated controls, we did not simulate a mid-shaft fracture in any of the tibias. The pin-site defects were intended to reflect the mechanical effects of bicortical defects immediately after pin removal and in the absence of any degree of bone healing. Finally, this study focused on pin-site defects that were distal to a midshaft plate and that may not represent the effects of bicortical pin-site defects proximal to the plate.

Given the results of this biomechanical study in composite tibias, varying the proximity of a bicortical defect to a plate does not affect the torsional stiffness or torsional failure strength of the bone. Placement of an intended bicortical defect should be based on considerations other than the potential for creating a distal stress riser after definitive fracture management.

Am J Orthop. 2017;46(2):E108-E111. Copyright Frontline Medical Communications Inc. 2017. All rights reserved.

Take-Home Points

• The location of a bicortical defect in proximity to a tibia plate does not appear to affect the torsional stiffness or torsional failure strength of the bone.

• External fixator pin placement should be based on considerations other than the potential for creating a distal stress riser after definitive fracture management.

A stress riser in cortical bone may be considered any abrupt change in the contour or consistency of the hollow structure, such as a surface defect, that not only weakens the bone but concentrates stresses at that transition point.¹ A cortical defect that is 20% of the bone diameter is associated with a 34% decrease in torsional strength, thus representing a “stress riser.”² High-energy and complex tibia fractures are often provisionally stabilized with external fixation that gives the soft tissues time to recover before definitive fracture fixation. Pin diameter for a medium-size tibia external fixator typically is 5.0 mm, resulting in a 10-mm defect in bicortical placement. Therefore, any tibia with a diameter of <50 mm is at risk for a stress riser fracture.

Although it had been established that sizable cortical defects can decrease the torsional strength of long bone,² the effect of a plate in close proximity to a defect secondary to an empty external-fixator pin site on torsional strength has not been determined. We conducted a study to evaluate this effect. The null hypothesis was there would be no difference in tibia torsional strength attributable to varying the proximity of a tibia midshaft plate to a 5.0-mm bicortical defect.

Methods

Forty fourth-generation, medium-size left composite tibias (Pacific Research Laboratories) were divided into 8 groups of 5 bones (Figure 1).

Torsion testing to failure was performed for all specimens in a manner similar to that described by Gardner and colleagues.³ Impression molds for the composite tibia constructed from polymethylmethacrylate encased the superior and distal ends, leaving 25.5 cm of exposed midshaft. This allowed the composites to be rigidly clamped into a materials testing system (858 Mini-Bionix; MTS) equipped with a 100.0-Nm torsional load cell (Figure 2).

Results

Graphical results for torsional stiffness are presented in Figure 3. R² for all stiffness calculations was >0.99.

Discussion

Many tibia fractures require provisional stabilization with an external fixator that spans the knee, because of the high-energy nature of the injury or other, higher-priority polytrauma concerns. When the patient or injury is suitable for definitive fixation, the external fixator typically is removed in favor of internal fixation with a plate and screws. Depending on the nature and location of the fracture and the subsequent plate, the empty cortical pin-site defects, often lying at varying distances from the distal end of the plate, can potentially serve as stress risers for fracture.⁴

Other studies have evaluated long-bone cortical defects biomechanically^1,2,4 and clinically,^5-7 and multiple studies have been conducted on the effects of plates on long-bone strength for fracture stabilization.^8-13 The present study evaluated the torsional strength of long bones in the presence of a bicortical defect and the proximity of the defect to a plate. There were no differences in stiffness or failure load between any of the groups of plated and unplated fourth-generation composite tibias tested to failure in torsion with varying distal bicortical defects. Hypothetically, one would expect the torsional stiffness of these specimens to increase with the mere addition of a metallic diaphyseal plate. However, this study demonstrated that the addition of a plate did not affect the torsional stiffness or strength of the tibias. Clinically, it is common practice to place external fixator pins as far as possible outside the planned incision site for definitive fracture fixation. Thus, we also hypothesized that the presence of a bicortical pin-site defect and its proximity to the plate would alter the torsional strength of the tibia specimens, and that the distal pin-site defect’s location farthest from the plate would exhibit greater strength, but this did not occur. Although other studies have shown that the presence of bicortical defects decreases the strength of long bones, we were unable to quantify this decrease because the 2 intact groups of composites, plated and unplated, survived failure testing.

This study had several limitations, first being the use of composite tibias as opposed to human cadaver bone. Although fourth-generation composite bone models have been validated as a suitable and accurate biomechanical substitute for cadaver specimens,¹⁴ anatomical variations in cadaver tibias may transfer forces differently through plates, screws, and distal pin sites. In order to test plated specimens against the unplated controls, we did not simulate a mid-shaft fracture in any of the tibias. The pin-site defects were intended to reflect the mechanical effects of bicortical defects immediately after pin removal and in the absence of any degree of bone healing. Finally, this study focused on pin-site defects that were distal to a midshaft plate and that may not represent the effects of bicortical pin-site defects proximal to the plate.

Given the results of this biomechanical study in composite tibias, varying the proximity of a bicortical defect to a plate does not affect the torsional stiffness or torsional failure strength of the bone. Placement of an intended bicortical defect should be based on considerations other than the potential for creating a distal stress riser after definitive fracture management.

Am J Orthop. 2017;46(2):E108-E111. Copyright Frontline Medical Communications Inc. 2017. All rights reserved.

Take-Home Points

• The location of a bicortical defect in proximity to a tibia plate does not appear to affect the torsional stiffness or torsional failure strength of the bone.

• External fixator pin placement should be based on considerations other than the potential for creating a distal stress riser after definitive fracture management.

A stress riser in cortical bone may be considered any abrupt change in the contour or consistency of the hollow structure, such as a surface defect, that not only weakens the bone but concentrates stresses at that transition point.¹ A cortical defect that is 20% of the bone diameter is associated with a 34% decrease in torsional strength, thus representing a “stress riser.”² High-energy and complex tibia fractures are often provisionally stabilized with external fixation that gives the soft tissues time to recover before definitive fracture fixation. Pin diameter for a medium-size tibia external fixator typically is 5.0 mm, resulting in a 10-mm defect in bicortical placement. Therefore, any tibia with a diameter of <50 mm is at risk for a stress riser fracture.

Although it had been established that sizable cortical defects can decrease the torsional strength of long bone,² the effect of a plate in close proximity to a defect secondary to an empty external-fixator pin site on torsional strength has not been determined. We conducted a study to evaluate this effect. The null hypothesis was there would be no difference in tibia torsional strength attributable to varying the proximity of a tibia midshaft plate to a 5.0-mm bicortical defect.

Methods

Forty fourth-generation, medium-size left composite tibias (Pacific Research Laboratories) were divided into 8 groups of 5 bones (Figure 1).

Torsion testing to failure was performed for all specimens in a manner similar to that described by Gardner and colleagues.³ Impression molds for the composite tibia constructed from polymethylmethacrylate encased the superior and distal ends, leaving 25.5 cm of exposed midshaft. This allowed the composites to be rigidly clamped into a materials testing system (858 Mini-Bionix; MTS) equipped with a 100.0-Nm torsional load cell (Figure 2).

Results

Graphical results for torsional stiffness are presented in Figure 3. R² for all stiffness calculations was >0.99.

Discussion

Many tibia fractures require provisional stabilization with an external fixator that spans the knee, because of the high-energy nature of the injury or other, higher-priority polytrauma concerns. When the patient or injury is suitable for definitive fixation, the external fixator typically is removed in favor of internal fixation with a plate and screws. Depending on the nature and location of the fracture and the subsequent plate, the empty cortical pin-site defects, often lying at varying distances from the distal end of the plate, can potentially serve as stress risers for fracture.⁴

Other studies have evaluated long-bone cortical defects biomechanically^1,2,4 and clinically,^5-7 and multiple studies have been conducted on the effects of plates on long-bone strength for fracture stabilization.^8-13 The present study evaluated the torsional strength of long bones in the presence of a bicortical defect and the proximity of the defect to a plate. There were no differences in stiffness or failure load between any of the groups of plated and unplated fourth-generation composite tibias tested to failure in torsion with varying distal bicortical defects. Hypothetically, one would expect the torsional stiffness of these specimens to increase with the mere addition of a metallic diaphyseal plate. However, this study demonstrated that the addition of a plate did not affect the torsional stiffness or strength of the tibias. Clinically, it is common practice to place external fixator pins as far as possible outside the planned incision site for definitive fracture fixation. Thus, we also hypothesized that the presence of a bicortical pin-site defect and its proximity to the plate would alter the torsional strength of the tibia specimens, and that the distal pin-site defect’s location farthest from the plate would exhibit greater strength, but this did not occur. Although other studies have shown that the presence of bicortical defects decreases the strength of long bones, we were unable to quantify this decrease because the 2 intact groups of composites, plated and unplated, survived failure testing.

This study had several limitations, first being the use of composite tibias as opposed to human cadaver bone. Although fourth-generation composite bone models have been validated as a suitable and accurate biomechanical substitute for cadaver specimens,¹⁴ anatomical variations in cadaver tibias may transfer forces differently through plates, screws, and distal pin sites. In order to test plated specimens against the unplated controls, we did not simulate a mid-shaft fracture in any of the tibias. The pin-site defects were intended to reflect the mechanical effects of bicortical defects immediately after pin removal and in the absence of any degree of bone healing. Finally, this study focused on pin-site defects that were distal to a midshaft plate and that may not represent the effects of bicortical pin-site defects proximal to the plate.

Given the results of this biomechanical study in composite tibias, varying the proximity of a bicortical defect to a plate does not affect the torsional stiffness or torsional failure strength of the bone. Placement of an intended bicortical defect should be based on considerations other than the potential for creating a distal stress riser after definitive fracture management.

Am J Orthop. 2017;46(2):E108-E111. Copyright Frontline Medical Communications Inc. 2017. All rights reserved.