Original Research

According to the American College of Radiology Appropriateness Criteria, renal ultrasound (RUS) is the most appropriate imaging examination for evaluating patients with acute kidney injury (AKI), with a rating score of 9, representing the strongest level of recommendation.[1, 2] However, recent studies suggest that RUS may be performed in patients with certain risk factors for ureteral obstruction,[1] which would lead to important reductions in the use of medical imaging. Licurse developed a risk stratification framework to help clinicians identify patients in whom RUS was most likely to be beneficial.[2] The model was built based on clinical predictors that included race, recent exposure to inpatient nephrotoxic medications, history of hydronephrosis, recurrent urinary tract infections, benign prostatic hyperplasia, abdominal or pelvic cancer, neurogenic bladder, single functional kidney, previous pelvic surgery, congestive heart failure, and prerenal AKI. It was found, using a cross‐sectional study design that included derivation and validation samples, that a low‐risk population could be identified based on demographic and clinical risk factors; in this population, the prevalence of hydronephrosis, as well as the rate of hydronephrosis requiring an intervention, was only <1%.

However, due to several study limitations, including that it was performed at a single center,[3] the stratification prediction rule has yet to be adopted broadly. Although at least 1 other study has similarly found that RUS may not be efficacious in patients with no suggestive history and with other more likely causes for renal failure,[1] to the best of our knowledge, no large, external, prospective trial to validate the selective use of RUS in patients with AKI has been reported. Therefore, the aim of this study was to evaluate the accuracy and usefulness of the Licurse renal ultrasonography risk stratification model for hospitalized patients with AKI.

METHODS

RESULTS

During the 23‐month study period, a total of 961 RUS studies were completed for inpatients with AKI; 778 unique studies met our inclusion criteria (Figure 1).

Figure 1

Based on the manual review of objective clinical data from the random sample of 80 medical records for 480 clinical attributes, overall, there was 90.2% (433/480) concordance rate between the structured data entry and that captured in free text in the clinical notes. There were some variations in the concordance rates for each clinical attribute, ranging from 78.8% (63/80) for exposure to nephrotoxic drugs to 95% for history of congestive heart failure.

On univariate analysis, patients with past medical history of hydronephrosis had a 5‐fold higher likelihood of developing a recurrence of hydronephrosis (45.9% [50/109] vs 8.4% [56/669], P < 0.001). Similarly, they also had a 9.5‐fold higher likelihood of requiring urologic interventions related to the hydronephrosis (12.8% [14/109] vs 1.4% [9/669], P < 0.001). Having diagnoses predisposing the patient for urinary obstruction (benign prostate hyperplasia, abdominal/pelvic cancer, neurogenic bladder, single functional kidney, and history of pelvic surgery) was correlated with the likelihood of both hydronephrosis and the need for urologic intervention. Of the patients with a diagnosis predisposing the patient for urinary obstructions, 22.1% (59/267) had hydronephrosis on imaging, whereas 9.2% (47/511) of patients without such a diagnosis had hydronephrosis (P < 0.001).

Conversely, having a recent exposure to nephrotoxic medications was negatively correlated with the likelihood of both hydronephrosis and the need for urologic intervention. Of the patients with recent exposure to nephrotoxic medications, 7.1% (20/280) had hydronephrosis on imaging, whereas the prevalence of hydronephrosis was 17.3% (86/498) in patients without such an exposure (P < 0.001) (Table 1).

Adjusted for other covariates, the multiple variable model showed that a diagnosis predisposing patients for obstruction (odds ratio [OR]: 2.0, P = 0.004), history of hydronephrosis (OR: 7.4, P < 0.001), absence of a history of congestive heart failure (OR: 2.7, P = 0.009), and lack of exposure to nephrotoxic medications (OR: 1.9, P = 0.022) were statistically significant predictors for hydronephrosis (Table 2).

After applying the Licurse renal ultrasonography risk stratification model, 176 (22.6%), 190 (24.4%), and 412 (53.0%) patients were classified as low risk, medium risk, and high risk for hydronephrosis, respectively. The incidence rates for hydronephrosis in the pretest probability risk groups were 4.0%, 6.8%, and 20.9% for low‐, medium‐, and high‐risk patients, respectively (P < 0.0001). The rates for urologic interventions were 1.1%, 0.5%, and 4.9% in the risk groups from low to high (P < 0.0001) (Figure 2).

Figure 2

Overall, the Licurse model, using a cutoff between low‐risk and medium/high‐risk patients, had sensitivity of 91.3% (95% confidence interval [CI]: 73.2%‐97.6%) for HNRI and 93.4% (95% CI: 87.0%‐96.8%) for presence of HN. Specificity was low for both HNRI (23.0% [95% CI: 20.2%‐26.2%]) and HN (25.1% [95% CI: 22.0%‐28.6%]). The estimated potential reduction in renal ultrasound for hospitalized patients with AKI, defined as the rate of imaging performed in the low‐risk group, was 22.6%. In the low‐risk group, the number needed to screen to find 1 case of HN was 25, and to find 1 case of HNRI it was 88. The negative predictive value for hydronephrosis was 96.0% (95% CI: 92.0%‐98.1%) and 98.9% for HNRI (95% CI: 96.0%‐99.7%) (Table 3).

Supporting Table 1, in the online version of this article, shows a sensitivity analysis using different cutoff thresholds in the Licurse model for classifying low‐risk patients. A lower threshold cutoff (ie, a cutoff of <1) significantly increases the sensitivity (98.1% [95% CI: 93.4%‐99.5%] for HN; 100% [95% CI: 85.7%‐100%]) for HNRI, but at the cost of a lower specificity (7.6% [95% CI: 5.8%‐9.8%] for HN and 7.0% [95% CI: 5.4%‐9.1%] for HNRI). The estimated potential reduction in renal ultrasound for hospitalized patients with AKI would be 6.0%, the number needed to screen to find 1 case of HN would be 26, and 1 case of HNRI would be infinity.

DISCUSSION

In this prospective observational study, we found that the Licurse risk stratification model, using a cutoff between low‐ risk and medium/high‐risk patients, had 91.3% (95% CI: 73.2%‐97.6%) sensitivity for predicting patients who would require urologic intervention and 93.4% (95% CI: 87.0%‐96.8%) sensitivity for identifying patients with hydronephrosis. These findings were comparable to those found in the original validation cohort of the model, which showed sensitivity rates of 96.3% and 91.8%, respectively.[2] The negative predictive value for hydronephrosis and HNRI were sufficiently high, at 96.0% (95% CI: 92.0‐98.1) and 98.9% (95% CI: 96.0‐99.7), respectively.

Our results suggest that the Licurse model may be sufficient to rule out HN in the inpatient setting at our institution. The slight differences between the findings of our and the original studies may be due to differences in data extraction methodologies. In the original study, all data were retrospectively abstracted from medical records (discharge summaries and clinical notes) by 4 trained reviewers. However, such methodology is dependent on the quality of unstructured EHR data, which as noted in previous research, can be highly variable. Hogan and Wagner found that the correctness of EHR data can range from 44% to 100% and completeness from 1.1% to 100%, depending on the clinical concepts being studied.[4] Similarly, Thiru et al. found that the sensitivity of different types of EHR data ranged from 0.25 to 1.0.[5] Medical chart review can be labor intensive and time consuming. The lack of standardized methods for structured data capture has been a major limitation in decreasing research costs and speeding the rate of new medical discoveries through the secondary use of EHR data. By modifying our institutional clinical decision support (CDS) system to enable the necessary granular clinical data collection, we were able to obviate the need for resource intensive retrospective chart reviews. To our knowledge, this is the second example of a CDS tool specifically designed for capture of discrete data to validate a decision rule.[6] A similar process may also be useful to accelerate generation of new decision rules. With secondary use of EHR data becoming an increasingly important topic,[7] CDS may serve as an alternative method in the context of data reuse for clinical research. Based on a randomly selected chart review, it was noted that clinicians, overall, do try to communicate to the interpreting radiologists the clinical picture as accurately as they can, and rarely do providers drop their orders due to data entry.

Despite our data confirming Licurse's initial findings, it is important to note that as with any clinical prediction rules, there is a trade‐off between cost savings and potential missed diagnoses. Even the most accepted clinical decision rules, such as the Well's criteria for pulmonary embolism and deep vein thrombosis, has their inherent acceptable rates of false negative. What is considered to be acceptable may differ among providers and patients. Thus, a shared decision‐making model, in which the patient and provider actively engage in sharing of information regarding risks and benefits of both performing and bypassing the diagnostic testing, is preferred. For providers/patients who are more risk‐adverse, one could consider using a more sensitive cutoff (for example, using the <1 threshold), essentially increasing the sensitivity from 91.3% to 100% for HNRI and from 93.4% to 98.1% for HN.

Although one would not want to miss a hydronephrosis in a patient, a too aggressive imaging strategy is not without economic and downstream risks. At an estimated cost of $200 per renal ultrasonography,[2] a 22.6% reduction would result in an annual savings of nearly $20,000 at our institution. The financial costs of forgoing ultrasound studies at the risk of missing 1 case of HN or 1 case of HNRI would be $5000 and $17,600, respectively.

Data‐driven decision rules are becoming more commonly used in the current environment of increased emphasis on evidence‐based medicine.[8, 9, 10, 11, 12, 13] When applied appropriately, such prediction models can result in more efficient use of medical imaging while increasing value of care.[14, 15] However, prior to implementation in clinical practice, these models need to be externally validated across multiple institutions and in various practice settings. This is the largest study of which we are aware to validate the utility of a prediction model for AKI in the inpatient setting. Although we did find slightly smaller differences in hydronephrosis in inpatients across the low, moderate, and high pretest probability groups, this may be explained by the differences in methodology.

Our study has several limitations. First, it was performed at a single academic medical center, a similar setting as that of the original work. Thus, the generalizability of our findings in other settings is unclear. Second, it is possible that our ordering providers did not thoroughly and accurately enter data into the structured CPOE form. However, we randomly selected a sample for chart review and found 90% concordance between data captured and those in the EHR. Due to selection of our cohort that included only patients with AKI who underwent RUS, it is possible that some patients who were not imaged or imaged with other cross‐sectional modalities were excluded, resulting in differential test ordering bias. Finally, we did not include the potential benefits of RUS in affecting nonsurgical interventions of hydronephrosis (eg, Foley catheter insertion).

CONCLUSION

We found that the Licurse renal ultrasonography risk stratification model was sufficiently accurate in classifying patients at risk for ureteral obstruction among hospitalized patients with AKI.

According to the American College of Radiology Appropriateness Criteria, renal ultrasound (RUS) is the most appropriate imaging examination for evaluating patients with acute kidney injury (AKI), with a rating score of 9, representing the strongest level of recommendation.[1, 2] However, recent studies suggest that RUS may be performed in patients with certain risk factors for ureteral obstruction,[1] which would lead to important reductions in the use of medical imaging. Licurse developed a risk stratification framework to help clinicians identify patients in whom RUS was most likely to be beneficial.[2] The model was built based on clinical predictors that included race, recent exposure to inpatient nephrotoxic medications, history of hydronephrosis, recurrent urinary tract infections, benign prostatic hyperplasia, abdominal or pelvic cancer, neurogenic bladder, single functional kidney, previous pelvic surgery, congestive heart failure, and prerenal AKI. It was found, using a cross‐sectional study design that included derivation and validation samples, that a low‐risk population could be identified based on demographic and clinical risk factors; in this population, the prevalence of hydronephrosis, as well as the rate of hydronephrosis requiring an intervention, was only <1%.

However, due to several study limitations, including that it was performed at a single center,[3] the stratification prediction rule has yet to be adopted broadly. Although at least 1 other study has similarly found that RUS may not be efficacious in patients with no suggestive history and with other more likely causes for renal failure,[1] to the best of our knowledge, no large, external, prospective trial to validate the selective use of RUS in patients with AKI has been reported. Therefore, the aim of this study was to evaluate the accuracy and usefulness of the Licurse renal ultrasonography risk stratification model for hospitalized patients with AKI.

METHODS

RESULTS

During the 23‐month study period, a total of 961 RUS studies were completed for inpatients with AKI; 778 unique studies met our inclusion criteria (Figure 1).

Figure 1

Based on the manual review of objective clinical data from the random sample of 80 medical records for 480 clinical attributes, overall, there was 90.2% (433/480) concordance rate between the structured data entry and that captured in free text in the clinical notes. There were some variations in the concordance rates for each clinical attribute, ranging from 78.8% (63/80) for exposure to nephrotoxic drugs to 95% for history of congestive heart failure.

On univariate analysis, patients with past medical history of hydronephrosis had a 5‐fold higher likelihood of developing a recurrence of hydronephrosis (45.9% [50/109] vs 8.4% [56/669], P < 0.001). Similarly, they also had a 9.5‐fold higher likelihood of requiring urologic interventions related to the hydronephrosis (12.8% [14/109] vs 1.4% [9/669], P < 0.001). Having diagnoses predisposing the patient for urinary obstruction (benign prostate hyperplasia, abdominal/pelvic cancer, neurogenic bladder, single functional kidney, and history of pelvic surgery) was correlated with the likelihood of both hydronephrosis and the need for urologic intervention. Of the patients with a diagnosis predisposing the patient for urinary obstructions, 22.1% (59/267) had hydronephrosis on imaging, whereas 9.2% (47/511) of patients without such a diagnosis had hydronephrosis (P < 0.001).

Conversely, having a recent exposure to nephrotoxic medications was negatively correlated with the likelihood of both hydronephrosis and the need for urologic intervention. Of the patients with recent exposure to nephrotoxic medications, 7.1% (20/280) had hydronephrosis on imaging, whereas the prevalence of hydronephrosis was 17.3% (86/498) in patients without such an exposure (P < 0.001) (Table 1).

Adjusted for other covariates, the multiple variable model showed that a diagnosis predisposing patients for obstruction (odds ratio [OR]: 2.0, P = 0.004), history of hydronephrosis (OR: 7.4, P < 0.001), absence of a history of congestive heart failure (OR: 2.7, P = 0.009), and lack of exposure to nephrotoxic medications (OR: 1.9, P = 0.022) were statistically significant predictors for hydronephrosis (Table 2).

After applying the Licurse renal ultrasonography risk stratification model, 176 (22.6%), 190 (24.4%), and 412 (53.0%) patients were classified as low risk, medium risk, and high risk for hydronephrosis, respectively. The incidence rates for hydronephrosis in the pretest probability risk groups were 4.0%, 6.8%, and 20.9% for low‐, medium‐, and high‐risk patients, respectively (P < 0.0001). The rates for urologic interventions were 1.1%, 0.5%, and 4.9% in the risk groups from low to high (P < 0.0001) (Figure 2).

Figure 2

Overall, the Licurse model, using a cutoff between low‐risk and medium/high‐risk patients, had sensitivity of 91.3% (95% confidence interval [CI]: 73.2%‐97.6%) for HNRI and 93.4% (95% CI: 87.0%‐96.8%) for presence of HN. Specificity was low for both HNRI (23.0% [95% CI: 20.2%‐26.2%]) and HN (25.1% [95% CI: 22.0%‐28.6%]). The estimated potential reduction in renal ultrasound for hospitalized patients with AKI, defined as the rate of imaging performed in the low‐risk group, was 22.6%. In the low‐risk group, the number needed to screen to find 1 case of HN was 25, and to find 1 case of HNRI it was 88. The negative predictive value for hydronephrosis was 96.0% (95% CI: 92.0%‐98.1%) and 98.9% for HNRI (95% CI: 96.0%‐99.7%) (Table 3).

Supporting Table 1, in the online version of this article, shows a sensitivity analysis using different cutoff thresholds in the Licurse model for classifying low‐risk patients. A lower threshold cutoff (ie, a cutoff of <1) significantly increases the sensitivity (98.1% [95% CI: 93.4%‐99.5%] for HN; 100% [95% CI: 85.7%‐100%]) for HNRI, but at the cost of a lower specificity (7.6% [95% CI: 5.8%‐9.8%] for HN and 7.0% [95% CI: 5.4%‐9.1%] for HNRI). The estimated potential reduction in renal ultrasound for hospitalized patients with AKI would be 6.0%, the number needed to screen to find 1 case of HN would be 26, and 1 case of HNRI would be infinity.

DISCUSSION

In this prospective observational study, we found that the Licurse risk stratification model, using a cutoff between low‐ risk and medium/high‐risk patients, had 91.3% (95% CI: 73.2%‐97.6%) sensitivity for predicting patients who would require urologic intervention and 93.4% (95% CI: 87.0%‐96.8%) sensitivity for identifying patients with hydronephrosis. These findings were comparable to those found in the original validation cohort of the model, which showed sensitivity rates of 96.3% and 91.8%, respectively.[2] The negative predictive value for hydronephrosis and HNRI were sufficiently high, at 96.0% (95% CI: 92.0‐98.1) and 98.9% (95% CI: 96.0‐99.7), respectively.

Our results suggest that the Licurse model may be sufficient to rule out HN in the inpatient setting at our institution. The slight differences between the findings of our and the original studies may be due to differences in data extraction methodologies. In the original study, all data were retrospectively abstracted from medical records (discharge summaries and clinical notes) by 4 trained reviewers. However, such methodology is dependent on the quality of unstructured EHR data, which as noted in previous research, can be highly variable. Hogan and Wagner found that the correctness of EHR data can range from 44% to 100% and completeness from 1.1% to 100%, depending on the clinical concepts being studied.[4] Similarly, Thiru et al. found that the sensitivity of different types of EHR data ranged from 0.25 to 1.0.[5] Medical chart review can be labor intensive and time consuming. The lack of standardized methods for structured data capture has been a major limitation in decreasing research costs and speeding the rate of new medical discoveries through the secondary use of EHR data. By modifying our institutional clinical decision support (CDS) system to enable the necessary granular clinical data collection, we were able to obviate the need for resource intensive retrospective chart reviews. To our knowledge, this is the second example of a CDS tool specifically designed for capture of discrete data to validate a decision rule.[6] A similar process may also be useful to accelerate generation of new decision rules. With secondary use of EHR data becoming an increasingly important topic,[7] CDS may serve as an alternative method in the context of data reuse for clinical research. Based on a randomly selected chart review, it was noted that clinicians, overall, do try to communicate to the interpreting radiologists the clinical picture as accurately as they can, and rarely do providers drop their orders due to data entry.

Despite our data confirming Licurse's initial findings, it is important to note that as with any clinical prediction rules, there is a trade‐off between cost savings and potential missed diagnoses. Even the most accepted clinical decision rules, such as the Well's criteria for pulmonary embolism and deep vein thrombosis, has their inherent acceptable rates of false negative. What is considered to be acceptable may differ among providers and patients. Thus, a shared decision‐making model, in which the patient and provider actively engage in sharing of information regarding risks and benefits of both performing and bypassing the diagnostic testing, is preferred. For providers/patients who are more risk‐adverse, one could consider using a more sensitive cutoff (for example, using the <1 threshold), essentially increasing the sensitivity from 91.3% to 100% for HNRI and from 93.4% to 98.1% for HN.

Although one would not want to miss a hydronephrosis in a patient, a too aggressive imaging strategy is not without economic and downstream risks. At an estimated cost of $200 per renal ultrasonography,[2] a 22.6% reduction would result in an annual savings of nearly $20,000 at our institution. The financial costs of forgoing ultrasound studies at the risk of missing 1 case of HN or 1 case of HNRI would be $5000 and $17,600, respectively.

Data‐driven decision rules are becoming more commonly used in the current environment of increased emphasis on evidence‐based medicine.[8, 9, 10, 11, 12, 13] When applied appropriately, such prediction models can result in more efficient use of medical imaging while increasing value of care.[14, 15] However, prior to implementation in clinical practice, these models need to be externally validated across multiple institutions and in various practice settings. This is the largest study of which we are aware to validate the utility of a prediction model for AKI in the inpatient setting. Although we did find slightly smaller differences in hydronephrosis in inpatients across the low, moderate, and high pretest probability groups, this may be explained by the differences in methodology.

Our study has several limitations. First, it was performed at a single academic medical center, a similar setting as that of the original work. Thus, the generalizability of our findings in other settings is unclear. Second, it is possible that our ordering providers did not thoroughly and accurately enter data into the structured CPOE form. However, we randomly selected a sample for chart review and found 90% concordance between data captured and those in the EHR. Due to selection of our cohort that included only patients with AKI who underwent RUS, it is possible that some patients who were not imaged or imaged with other cross‐sectional modalities were excluded, resulting in differential test ordering bias. Finally, we did not include the potential benefits of RUS in affecting nonsurgical interventions of hydronephrosis (eg, Foley catheter insertion).

CONCLUSION

We found that the Licurse renal ultrasonography risk stratification model was sufficiently accurate in classifying patients at risk for ureteral obstruction among hospitalized patients with AKI.

According to the American College of Radiology Appropriateness Criteria, renal ultrasound (RUS) is the most appropriate imaging examination for evaluating patients with acute kidney injury (AKI), with a rating score of 9, representing the strongest level of recommendation.[1, 2] However, recent studies suggest that RUS may be performed in patients with certain risk factors for ureteral obstruction,[1] which would lead to important reductions in the use of medical imaging. Licurse developed a risk stratification framework to help clinicians identify patients in whom RUS was most likely to be beneficial.[2] The model was built based on clinical predictors that included race, recent exposure to inpatient nephrotoxic medications, history of hydronephrosis, recurrent urinary tract infections, benign prostatic hyperplasia, abdominal or pelvic cancer, neurogenic bladder, single functional kidney, previous pelvic surgery, congestive heart failure, and prerenal AKI. It was found, using a cross‐sectional study design that included derivation and validation samples, that a low‐risk population could be identified based on demographic and clinical risk factors; in this population, the prevalence of hydronephrosis, as well as the rate of hydronephrosis requiring an intervention, was only <1%.

However, due to several study limitations, including that it was performed at a single center,[3] the stratification prediction rule has yet to be adopted broadly. Although at least 1 other study has similarly found that RUS may not be efficacious in patients with no suggestive history and with other more likely causes for renal failure,[1] to the best of our knowledge, no large, external, prospective trial to validate the selective use of RUS in patients with AKI has been reported. Therefore, the aim of this study was to evaluate the accuracy and usefulness of the Licurse renal ultrasonography risk stratification model for hospitalized patients with AKI.

METHODS

RESULTS

During the 23‐month study period, a total of 961 RUS studies were completed for inpatients with AKI; 778 unique studies met our inclusion criteria (Figure 1).

Figure 1

Based on the manual review of objective clinical data from the random sample of 80 medical records for 480 clinical attributes, overall, there was 90.2% (433/480) concordance rate between the structured data entry and that captured in free text in the clinical notes. There were some variations in the concordance rates for each clinical attribute, ranging from 78.8% (63/80) for exposure to nephrotoxic drugs to 95% for history of congestive heart failure.

On univariate analysis, patients with past medical history of hydronephrosis had a 5‐fold higher likelihood of developing a recurrence of hydronephrosis (45.9% [50/109] vs 8.4% [56/669], P < 0.001). Similarly, they also had a 9.5‐fold higher likelihood of requiring urologic interventions related to the hydronephrosis (12.8% [14/109] vs 1.4% [9/669], P < 0.001). Having diagnoses predisposing the patient for urinary obstruction (benign prostate hyperplasia, abdominal/pelvic cancer, neurogenic bladder, single functional kidney, and history of pelvic surgery) was correlated with the likelihood of both hydronephrosis and the need for urologic intervention. Of the patients with a diagnosis predisposing the patient for urinary obstructions, 22.1% (59/267) had hydronephrosis on imaging, whereas 9.2% (47/511) of patients without such a diagnosis had hydronephrosis (P < 0.001).

Conversely, having a recent exposure to nephrotoxic medications was negatively correlated with the likelihood of both hydronephrosis and the need for urologic intervention. Of the patients with recent exposure to nephrotoxic medications, 7.1% (20/280) had hydronephrosis on imaging, whereas the prevalence of hydronephrosis was 17.3% (86/498) in patients without such an exposure (P < 0.001) (Table 1).

Adjusted for other covariates, the multiple variable model showed that a diagnosis predisposing patients for obstruction (odds ratio [OR]: 2.0, P = 0.004), history of hydronephrosis (OR: 7.4, P < 0.001), absence of a history of congestive heart failure (OR: 2.7, P = 0.009), and lack of exposure to nephrotoxic medications (OR: 1.9, P = 0.022) were statistically significant predictors for hydronephrosis (Table 2).

After applying the Licurse renal ultrasonography risk stratification model, 176 (22.6%), 190 (24.4%), and 412 (53.0%) patients were classified as low risk, medium risk, and high risk for hydronephrosis, respectively. The incidence rates for hydronephrosis in the pretest probability risk groups were 4.0%, 6.8%, and 20.9% for low‐, medium‐, and high‐risk patients, respectively (P < 0.0001). The rates for urologic interventions were 1.1%, 0.5%, and 4.9% in the risk groups from low to high (P < 0.0001) (Figure 2).

Figure 2

Overall, the Licurse model, using a cutoff between low‐risk and medium/high‐risk patients, had sensitivity of 91.3% (95% confidence interval [CI]: 73.2%‐97.6%) for HNRI and 93.4% (95% CI: 87.0%‐96.8%) for presence of HN. Specificity was low for both HNRI (23.0% [95% CI: 20.2%‐26.2%]) and HN (25.1% [95% CI: 22.0%‐28.6%]). The estimated potential reduction in renal ultrasound for hospitalized patients with AKI, defined as the rate of imaging performed in the low‐risk group, was 22.6%. In the low‐risk group, the number needed to screen to find 1 case of HN was 25, and to find 1 case of HNRI it was 88. The negative predictive value for hydronephrosis was 96.0% (95% CI: 92.0%‐98.1%) and 98.9% for HNRI (95% CI: 96.0%‐99.7%) (Table 3).

Supporting Table 1, in the online version of this article, shows a sensitivity analysis using different cutoff thresholds in the Licurse model for classifying low‐risk patients. A lower threshold cutoff (ie, a cutoff of <1) significantly increases the sensitivity (98.1% [95% CI: 93.4%‐99.5%] for HN; 100% [95% CI: 85.7%‐100%]) for HNRI, but at the cost of a lower specificity (7.6% [95% CI: 5.8%‐9.8%] for HN and 7.0% [95% CI: 5.4%‐9.1%] for HNRI). The estimated potential reduction in renal ultrasound for hospitalized patients with AKI would be 6.0%, the number needed to screen to find 1 case of HN would be 26, and 1 case of HNRI would be infinity.

DISCUSSION

In this prospective observational study, we found that the Licurse risk stratification model, using a cutoff between low‐ risk and medium/high‐risk patients, had 91.3% (95% CI: 73.2%‐97.6%) sensitivity for predicting patients who would require urologic intervention and 93.4% (95% CI: 87.0%‐96.8%) sensitivity for identifying patients with hydronephrosis. These findings were comparable to those found in the original validation cohort of the model, which showed sensitivity rates of 96.3% and 91.8%, respectively.[2] The negative predictive value for hydronephrosis and HNRI were sufficiently high, at 96.0% (95% CI: 92.0‐98.1) and 98.9% (95% CI: 96.0‐99.7), respectively.

Our results suggest that the Licurse model may be sufficient to rule out HN in the inpatient setting at our institution. The slight differences between the findings of our and the original studies may be due to differences in data extraction methodologies. In the original study, all data were retrospectively abstracted from medical records (discharge summaries and clinical notes) by 4 trained reviewers. However, such methodology is dependent on the quality of unstructured EHR data, which as noted in previous research, can be highly variable. Hogan and Wagner found that the correctness of EHR data can range from 44% to 100% and completeness from 1.1% to 100%, depending on the clinical concepts being studied.[4] Similarly, Thiru et al. found that the sensitivity of different types of EHR data ranged from 0.25 to 1.0.[5] Medical chart review can be labor intensive and time consuming. The lack of standardized methods for structured data capture has been a major limitation in decreasing research costs and speeding the rate of new medical discoveries through the secondary use of EHR data. By modifying our institutional clinical decision support (CDS) system to enable the necessary granular clinical data collection, we were able to obviate the need for resource intensive retrospective chart reviews. To our knowledge, this is the second example of a CDS tool specifically designed for capture of discrete data to validate a decision rule.[6] A similar process may also be useful to accelerate generation of new decision rules. With secondary use of EHR data becoming an increasingly important topic,[7] CDS may serve as an alternative method in the context of data reuse for clinical research. Based on a randomly selected chart review, it was noted that clinicians, overall, do try to communicate to the interpreting radiologists the clinical picture as accurately as they can, and rarely do providers drop their orders due to data entry.

Despite our data confirming Licurse's initial findings, it is important to note that as with any clinical prediction rules, there is a trade‐off between cost savings and potential missed diagnoses. Even the most accepted clinical decision rules, such as the Well's criteria for pulmonary embolism and deep vein thrombosis, has their inherent acceptable rates of false negative. What is considered to be acceptable may differ among providers and patients. Thus, a shared decision‐making model, in which the patient and provider actively engage in sharing of information regarding risks and benefits of both performing and bypassing the diagnostic testing, is preferred. For providers/patients who are more risk‐adverse, one could consider using a more sensitive cutoff (for example, using the <1 threshold), essentially increasing the sensitivity from 91.3% to 100% for HNRI and from 93.4% to 98.1% for HN.

Although one would not want to miss a hydronephrosis in a patient, a too aggressive imaging strategy is not without economic and downstream risks. At an estimated cost of $200 per renal ultrasonography,[2] a 22.6% reduction would result in an annual savings of nearly $20,000 at our institution. The financial costs of forgoing ultrasound studies at the risk of missing 1 case of HN or 1 case of HNRI would be $5000 and $17,600, respectively.

Data‐driven decision rules are becoming more commonly used in the current environment of increased emphasis on evidence‐based medicine.[8, 9, 10, 11, 12, 13] When applied appropriately, such prediction models can result in more efficient use of medical imaging while increasing value of care.[14, 15] However, prior to implementation in clinical practice, these models need to be externally validated across multiple institutions and in various practice settings. This is the largest study of which we are aware to validate the utility of a prediction model for AKI in the inpatient setting. Although we did find slightly smaller differences in hydronephrosis in inpatients across the low, moderate, and high pretest probability groups, this may be explained by the differences in methodology.

Our study has several limitations. First, it was performed at a single academic medical center, a similar setting as that of the original work. Thus, the generalizability of our findings in other settings is unclear. Second, it is possible that our ordering providers did not thoroughly and accurately enter data into the structured CPOE form. However, we randomly selected a sample for chart review and found 90% concordance between data captured and those in the EHR. Due to selection of our cohort that included only patients with AKI who underwent RUS, it is possible that some patients who were not imaged or imaged with other cross‐sectional modalities were excluded, resulting in differential test ordering bias. Finally, we did not include the potential benefits of RUS in affecting nonsurgical interventions of hydronephrosis (eg, Foley catheter insertion).

CONCLUSION

We found that the Licurse renal ultrasonography risk stratification model was sufficiently accurate in classifying patients at risk for ureteral obstruction among hospitalized patients with AKI.

Frailty is a state of vulnerability that encompasses a heterogeneous group of people.[1] Because it lacks a precise definition, multiple tools have been developed to identify frailty in both clinical and research settings.[2, 3, 4] Prevalence of frailty depends on the frailty assessment tool used and the population studied, ranging from 4% to 17% when the Fried score[5, 6, 7] is used and from 5% to 44%[5, 7, 8] when cumulative deficit models like the Frailty Index are utilized, with the lower prevalences being in younger community‐dwelling elderly populations and the higher proportions being in older institutionalized populations.

The Frailty Index, also called the Burden or Cumulative Deficit Model, comprises 70 domains that include mobility, mood, function, cognitive impairment, and disease states. It is multidimensional and allows for patients to be categorized on a continuum of frailty, but it is extremely difficult to apply in clinical practice. Recognizing this, Rockwood et al.[9] developed and validated the Clinical Frailty Scale (CFS) in the Canadian Study of Health and Aging. The CFS classifies patients into 1 of 9 categories: very fit, well, managing well, vulnerable, mildly frail (needs help with at least 1 instrumental activity of daily living such as shopping, finances, meal preparation, or housework), moderately frail (needs help with 1 or 2 activities of daily living such as bathing and dressing), severely frail (dependent for personal care), very severely frail (bedbound), and terminally ill. Although this tool is easy to use in clinical practice, it reflects a gestalt impression and requires some clinical judgement.

The Fried score[6] is a prototypical phenotype tool based on 5 criteria that include weight loss, self‐reported exhaustion, low energy expenditure, slowness of gait, and weakness. Recent evidence has suggested that slow gait (or dysmobility) alone may also be a potential screening test for frailty.[10] A recent systematic review[11] demonstrated an association between slow gait (dysmobility) and increased mortality. Dysmobility negatively impacts quality of life and has a strong association with disability resulting in the need for an increased level of care.[12] The Timed Up and Go Test (TUGT) is one method of assessing mobility which is relatively easy to perform, does not require special equipment, and is feasible to use in clinical settings.[13] However, whether impaired mobility predicts outcomes within the first 30 days after hospital discharge (a timeframe highlighted in the Affordable Care Act and used by the Centers for Medicare and Medicaid Services as an important hospital quality indicator) is still uncertain.

The aim of this study was to compare frailty assessments using the CFS and 2 of the most commonly used phenotypic tools (a modified Fried score and the TUGT as a proxy for mobility assessment) to determine which tools best predict postdischarge outcomes.

METHODS

RESULTS

Of 1124 potentially eligible patients, 626 were excluded because of patient refusal (n = 227); transfer to/from another hospital, long‐term care facility, or out of province (n = 189); moderate to severe cognitive impairment (n = 88); language barriers (n = 71); or foreshortened life expectancy (n = 51). Another 3 patients withdrew consent prior to outcome assessment. The 495 patients we recruited and had outcome data for had a mean age of 64 years, 19.6% were older than 80 years, 50% were women, and the patients had a mean of 4.2 comorbidities and mean Charlson score of 2.4. The 4 most common reasons for hospital admission were heart failure, pneumonia, chronic obstructive pulmonary disease, and urinary tract infection, and the median length of stay was 5 days (interquartile range: 49 days).

Prevalence of Frailty According to Different Definitions

Although the CFS assessment resulted in 162 (33%) patients being deemed frail, only 82 (51%) of those patients also met the phenotype frailty definition using either the Fried model or the TUGT, and 49 (10%) patients who were not classified as frail on the CFS met either of the phenotypic definitions of frailty (Figure 1). Overall, 211 (43%) patients were frail according to at least 1 assessment, and 46 (9%) met all 3 frailty definitions. In the subgroup of 245 patients older than 65 years, 137 (56%) were frail according to at least 1 assessment, 38 (16%) met all 3 frailty definitions, and 27 (11%) of those patients classified as not frail on the CFS met either phenotypic definition of frailty. Agreement between TUGT and CFS or CFS and Fried was relatively poor with kappas of 0.31 (95% confidence interval [CI]: 0.23‐0.40) and 0.33 (95% CI: 0.25‐0.42), respectively. It is noteworthy that some patients deemed nonfrail on the CFS had slow gait speeds, and most CFS‐frail patients had gait speeds in the nonfrail range (Figure 2).

Figure 1

Figure 2

DISCUSSION

In summary, we found that of patients being discharged from general medical wards who were frail according to at least 1 of the 3 tools we used, only 22% met all 3 frailty case definitions (including only 28% of elderly patients deemed frail by at least 1 definition). There was surprisingly poor correlation between phenotypic markers of frailty such as poor mobility (slow TUGT) or the modified Fried Index and the CFS, even amongt elderly patients. The most clinically useful of the frailty assessment tools (both overall and in those patients who are elderly) appears to be the CFS, because it more accurately identifies those at higher risk of adverse outcomes after discharge, does not require special equipment to conduct, and is faster to do than the phenotypic assessment models we tested. We have also previously demonstrated that the CFS, after a brief training period identical to that used in this study, is reproducible between observers[19] and remains an independent predictor of adverse 30‐day outcomes even after adjusting for age, sex, comorbidities, and the LACE (length of stay, acuity of the admission, comorbidity, emergency room visits during the previous 6 months) score.[14]

Although some[10] have advocated for the use of mobility assessments (such as gait speed) as a frailty marker due to its ease of measurement and objectivity, we found that slow TUGT (which is a marker for mobility and not just slow gait speed) was not an independent prognostic marker for postdischarge outcomes. We hypothesize that the phenotypic models of frailty performed less well than the CFS as they focus on the measurement of particular physical attributes and do not take into account cognitive or psychosocial characteristics or comorbidity burden that also influence postdischarge outcomes. As well, the CFS captures the patients' baseline status prior to acute illness, whereas the phenotypic measures were assessed just prior to discharge and thus may provide less information about eventual recovery potential. Some have suggested that repeating phenotype measures postdischarge might be more informative,[20] but this would reduce clinical applicability a great deal. Certainly, an analysis[21] of the Cardiovascular Health Study cohort demonstrated that cumulative deficit models of frailty (for which the CFS is an accurate proxy[9, 15]) better predicted risk of death than phenotypic models.

Although a number of published studies have shown similar results to ours in that frail patients are at greater risk for death and/or hospitalization,[22, 23, 24] there is surprisingly little literature on the comparative predictive performance of the different frailty instruments and the extent to which they overlap. Cigolle et al.[25] compared 3 frailty scales (the Functional Domain Model, the Burden Model, and the Fried score) in the Health and Retirement Study and, similarly to us, found that although 30.2% were frail on at least 1 of these scales, only 3.1% were deemed frail by all 3. The Conselice Study of Brain Aging[5] also reported that a deficit accumulation model defined a much higher prevalence of frailty (37.6%) than the 11.6% identified using the phenotypic Study of Osteoporotic Fractures (SOF) index based on weight loss, mobility, and level of energy. Another study[26] reported that risk models incorporating either the SOF index or the Fried score exhibited C statistics of only 0.61 for predicting falls in elderly females. A cohort study[27] from 2 English general medical units also found that none of the 5 frailty models was particularly accurate at predicting risk of readmission at 3 months, with C statistics ranging between 0.52 and 0.57. Although frailty assessment at time of hospital admission predicted in‐hospital mortality and length of stay in another English study, it was not independently associated with 30‐day outcomes after adjusting for age, sex, and comorbidities including dementia.[27] To our knowledge, these latter 2 are the only other studies reported to date performed in hospitalized patients to assess whether frailty assessment helps predict postdischarge outcomes. Thus, the poor C statistics we found for all of our frailty tools confirms prior literature that frailty assessment alone is inadequate to accurately identify those patients at highest risk for poor outcomes in the first 30 days after discharge. However, frailty assessment together with consideration of each individual's comorbidities, cognitive status, psychosocial circumstances, and environment can be useful to flag those individuals who may need extra attention postdischarge to optimize outcomes.

Frailty is a state of vulnerability that encompasses a heterogeneous group of people.[1] Because it lacks a precise definition, multiple tools have been developed to identify frailty in both clinical and research settings.[2, 3, 4] Prevalence of frailty depends on the frailty assessment tool used and the population studied, ranging from 4% to 17% when the Fried score[5, 6, 7] is used and from 5% to 44%[5, 7, 8] when cumulative deficit models like the Frailty Index are utilized, with the lower prevalences being in younger community‐dwelling elderly populations and the higher proportions being in older institutionalized populations.

The Frailty Index, also called the Burden or Cumulative Deficit Model, comprises 70 domains that include mobility, mood, function, cognitive impairment, and disease states. It is multidimensional and allows for patients to be categorized on a continuum of frailty, but it is extremely difficult to apply in clinical practice. Recognizing this, Rockwood et al.[9] developed and validated the Clinical Frailty Scale (CFS) in the Canadian Study of Health and Aging. The CFS classifies patients into 1 of 9 categories: very fit, well, managing well, vulnerable, mildly frail (needs help with at least 1 instrumental activity of daily living such as shopping, finances, meal preparation, or housework), moderately frail (needs help with 1 or 2 activities of daily living such as bathing and dressing), severely frail (dependent for personal care), very severely frail (bedbound), and terminally ill. Although this tool is easy to use in clinical practice, it reflects a gestalt impression and requires some clinical judgement.

The Fried score[6] is a prototypical phenotype tool based on 5 criteria that include weight loss, self‐reported exhaustion, low energy expenditure, slowness of gait, and weakness. Recent evidence has suggested that slow gait (or dysmobility) alone may also be a potential screening test for frailty.[10] A recent systematic review[11] demonstrated an association between slow gait (dysmobility) and increased mortality. Dysmobility negatively impacts quality of life and has a strong association with disability resulting in the need for an increased level of care.[12] The Timed Up and Go Test (TUGT) is one method of assessing mobility which is relatively easy to perform, does not require special equipment, and is feasible to use in clinical settings.[13] However, whether impaired mobility predicts outcomes within the first 30 days after hospital discharge (a timeframe highlighted in the Affordable Care Act and used by the Centers for Medicare and Medicaid Services as an important hospital quality indicator) is still uncertain.

The aim of this study was to compare frailty assessments using the CFS and 2 of the most commonly used phenotypic tools (a modified Fried score and the TUGT as a proxy for mobility assessment) to determine which tools best predict postdischarge outcomes.

METHODS

RESULTS

Of 1124 potentially eligible patients, 626 were excluded because of patient refusal (n = 227); transfer to/from another hospital, long‐term care facility, or out of province (n = 189); moderate to severe cognitive impairment (n = 88); language barriers (n = 71); or foreshortened life expectancy (n = 51). Another 3 patients withdrew consent prior to outcome assessment. The 495 patients we recruited and had outcome data for had a mean age of 64 years, 19.6% were older than 80 years, 50% were women, and the patients had a mean of 4.2 comorbidities and mean Charlson score of 2.4. The 4 most common reasons for hospital admission were heart failure, pneumonia, chronic obstructive pulmonary disease, and urinary tract infection, and the median length of stay was 5 days (interquartile range: 49 days).

Prevalence of Frailty According to Different Definitions

Although the CFS assessment resulted in 162 (33%) patients being deemed frail, only 82 (51%) of those patients also met the phenotype frailty definition using either the Fried model or the TUGT, and 49 (10%) patients who were not classified as frail on the CFS met either of the phenotypic definitions of frailty (Figure 1). Overall, 211 (43%) patients were frail according to at least 1 assessment, and 46 (9%) met all 3 frailty definitions. In the subgroup of 245 patients older than 65 years, 137 (56%) were frail according to at least 1 assessment, 38 (16%) met all 3 frailty definitions, and 27 (11%) of those patients classified as not frail on the CFS met either phenotypic definition of frailty. Agreement between TUGT and CFS or CFS and Fried was relatively poor with kappas of 0.31 (95% confidence interval [CI]: 0.23‐0.40) and 0.33 (95% CI: 0.25‐0.42), respectively. It is noteworthy that some patients deemed nonfrail on the CFS had slow gait speeds, and most CFS‐frail patients had gait speeds in the nonfrail range (Figure 2).

Figure 1

Figure 2

DISCUSSION

In summary, we found that of patients being discharged from general medical wards who were frail according to at least 1 of the 3 tools we used, only 22% met all 3 frailty case definitions (including only 28% of elderly patients deemed frail by at least 1 definition). There was surprisingly poor correlation between phenotypic markers of frailty such as poor mobility (slow TUGT) or the modified Fried Index and the CFS, even amongt elderly patients. The most clinically useful of the frailty assessment tools (both overall and in those patients who are elderly) appears to be the CFS, because it more accurately identifies those at higher risk of adverse outcomes after discharge, does not require special equipment to conduct, and is faster to do than the phenotypic assessment models we tested. We have also previously demonstrated that the CFS, after a brief training period identical to that used in this study, is reproducible between observers[19] and remains an independent predictor of adverse 30‐day outcomes even after adjusting for age, sex, comorbidities, and the LACE (length of stay, acuity of the admission, comorbidity, emergency room visits during the previous 6 months) score.[14]

Although some[10] have advocated for the use of mobility assessments (such as gait speed) as a frailty marker due to its ease of measurement and objectivity, we found that slow TUGT (which is a marker for mobility and not just slow gait speed) was not an independent prognostic marker for postdischarge outcomes. We hypothesize that the phenotypic models of frailty performed less well than the CFS as they focus on the measurement of particular physical attributes and do not take into account cognitive or psychosocial characteristics or comorbidity burden that also influence postdischarge outcomes. As well, the CFS captures the patients' baseline status prior to acute illness, whereas the phenotypic measures were assessed just prior to discharge and thus may provide less information about eventual recovery potential. Some have suggested that repeating phenotype measures postdischarge might be more informative,[20] but this would reduce clinical applicability a great deal. Certainly, an analysis[21] of the Cardiovascular Health Study cohort demonstrated that cumulative deficit models of frailty (for which the CFS is an accurate proxy[9, 15]) better predicted risk of death than phenotypic models.

Although a number of published studies have shown similar results to ours in that frail patients are at greater risk for death and/or hospitalization,[22, 23, 24] there is surprisingly little literature on the comparative predictive performance of the different frailty instruments and the extent to which they overlap. Cigolle et al.[25] compared 3 frailty scales (the Functional Domain Model, the Burden Model, and the Fried score) in the Health and Retirement Study and, similarly to us, found that although 30.2% were frail on at least 1 of these scales, only 3.1% were deemed frail by all 3. The Conselice Study of Brain Aging[5] also reported that a deficit accumulation model defined a much higher prevalence of frailty (37.6%) than the 11.6% identified using the phenotypic Study of Osteoporotic Fractures (SOF) index based on weight loss, mobility, and level of energy. Another study[26] reported that risk models incorporating either the SOF index or the Fried score exhibited C statistics of only 0.61 for predicting falls in elderly females. A cohort study[27] from 2 English general medical units also found that none of the 5 frailty models was particularly accurate at predicting risk of readmission at 3 months, with C statistics ranging between 0.52 and 0.57. Although frailty assessment at time of hospital admission predicted in‐hospital mortality and length of stay in another English study, it was not independently associated with 30‐day outcomes after adjusting for age, sex, and comorbidities including dementia.[27] To our knowledge, these latter 2 are the only other studies reported to date performed in hospitalized patients to assess whether frailty assessment helps predict postdischarge outcomes. Thus, the poor C statistics we found for all of our frailty tools confirms prior literature that frailty assessment alone is inadequate to accurately identify those patients at highest risk for poor outcomes in the first 30 days after discharge. However, frailty assessment together with consideration of each individual's comorbidities, cognitive status, psychosocial circumstances, and environment can be useful to flag those individuals who may need extra attention postdischarge to optimize outcomes.

Frailty is a state of vulnerability that encompasses a heterogeneous group of people.[1] Because it lacks a precise definition, multiple tools have been developed to identify frailty in both clinical and research settings.[2, 3, 4] Prevalence of frailty depends on the frailty assessment tool used and the population studied, ranging from 4% to 17% when the Fried score[5, 6, 7] is used and from 5% to 44%[5, 7, 8] when cumulative deficit models like the Frailty Index are utilized, with the lower prevalences being in younger community‐dwelling elderly populations and the higher proportions being in older institutionalized populations.

The Frailty Index, also called the Burden or Cumulative Deficit Model, comprises 70 domains that include mobility, mood, function, cognitive impairment, and disease states. It is multidimensional and allows for patients to be categorized on a continuum of frailty, but it is extremely difficult to apply in clinical practice. Recognizing this, Rockwood et al.[9] developed and validated the Clinical Frailty Scale (CFS) in the Canadian Study of Health and Aging. The CFS classifies patients into 1 of 9 categories: very fit, well, managing well, vulnerable, mildly frail (needs help with at least 1 instrumental activity of daily living such as shopping, finances, meal preparation, or housework), moderately frail (needs help with 1 or 2 activities of daily living such as bathing and dressing), severely frail (dependent for personal care), very severely frail (bedbound), and terminally ill. Although this tool is easy to use in clinical practice, it reflects a gestalt impression and requires some clinical judgement.

The Fried score[6] is a prototypical phenotype tool based on 5 criteria that include weight loss, self‐reported exhaustion, low energy expenditure, slowness of gait, and weakness. Recent evidence has suggested that slow gait (or dysmobility) alone may also be a potential screening test for frailty.[10] A recent systematic review[11] demonstrated an association between slow gait (dysmobility) and increased mortality. Dysmobility negatively impacts quality of life and has a strong association with disability resulting in the need for an increased level of care.[12] The Timed Up and Go Test (TUGT) is one method of assessing mobility which is relatively easy to perform, does not require special equipment, and is feasible to use in clinical settings.[13] However, whether impaired mobility predicts outcomes within the first 30 days after hospital discharge (a timeframe highlighted in the Affordable Care Act and used by the Centers for Medicare and Medicaid Services as an important hospital quality indicator) is still uncertain.

The aim of this study was to compare frailty assessments using the CFS and 2 of the most commonly used phenotypic tools (a modified Fried score and the TUGT as a proxy for mobility assessment) to determine which tools best predict postdischarge outcomes.

METHODS

RESULTS

Of 1124 potentially eligible patients, 626 were excluded because of patient refusal (n = 227); transfer to/from another hospital, long‐term care facility, or out of province (n = 189); moderate to severe cognitive impairment (n = 88); language barriers (n = 71); or foreshortened life expectancy (n = 51). Another 3 patients withdrew consent prior to outcome assessment. The 495 patients we recruited and had outcome data for had a mean age of 64 years, 19.6% were older than 80 years, 50% were women, and the patients had a mean of 4.2 comorbidities and mean Charlson score of 2.4. The 4 most common reasons for hospital admission were heart failure, pneumonia, chronic obstructive pulmonary disease, and urinary tract infection, and the median length of stay was 5 days (interquartile range: 49 days).

Prevalence of Frailty According to Different Definitions

Although the CFS assessment resulted in 162 (33%) patients being deemed frail, only 82 (51%) of those patients also met the phenotype frailty definition using either the Fried model or the TUGT, and 49 (10%) patients who were not classified as frail on the CFS met either of the phenotypic definitions of frailty (Figure 1). Overall, 211 (43%) patients were frail according to at least 1 assessment, and 46 (9%) met all 3 frailty definitions. In the subgroup of 245 patients older than 65 years, 137 (56%) were frail according to at least 1 assessment, 38 (16%) met all 3 frailty definitions, and 27 (11%) of those patients classified as not frail on the CFS met either phenotypic definition of frailty. Agreement between TUGT and CFS or CFS and Fried was relatively poor with kappas of 0.31 (95% confidence interval [CI]: 0.23‐0.40) and 0.33 (95% CI: 0.25‐0.42), respectively. It is noteworthy that some patients deemed nonfrail on the CFS had slow gait speeds, and most CFS‐frail patients had gait speeds in the nonfrail range (Figure 2).

Figure 1

Figure 2

DISCUSSION

In summary, we found that of patients being discharged from general medical wards who were frail according to at least 1 of the 3 tools we used, only 22% met all 3 frailty case definitions (including only 28% of elderly patients deemed frail by at least 1 definition). There was surprisingly poor correlation between phenotypic markers of frailty such as poor mobility (slow TUGT) or the modified Fried Index and the CFS, even amongt elderly patients. The most clinically useful of the frailty assessment tools (both overall and in those patients who are elderly) appears to be the CFS, because it more accurately identifies those at higher risk of adverse outcomes after discharge, does not require special equipment to conduct, and is faster to do than the phenotypic assessment models we tested. We have also previously demonstrated that the CFS, after a brief training period identical to that used in this study, is reproducible between observers[19] and remains an independent predictor of adverse 30‐day outcomes even after adjusting for age, sex, comorbidities, and the LACE (length of stay, acuity of the admission, comorbidity, emergency room visits during the previous 6 months) score.[14]

Although some[10] have advocated for the use of mobility assessments (such as gait speed) as a frailty marker due to its ease of measurement and objectivity, we found that slow TUGT (which is a marker for mobility and not just slow gait speed) was not an independent prognostic marker for postdischarge outcomes. We hypothesize that the phenotypic models of frailty performed less well than the CFS as they focus on the measurement of particular physical attributes and do not take into account cognitive or psychosocial characteristics or comorbidity burden that also influence postdischarge outcomes. As well, the CFS captures the patients' baseline status prior to acute illness, whereas the phenotypic measures were assessed just prior to discharge and thus may provide less information about eventual recovery potential. Some have suggested that repeating phenotype measures postdischarge might be more informative,[20] but this would reduce clinical applicability a great deal. Certainly, an analysis[21] of the Cardiovascular Health Study cohort demonstrated that cumulative deficit models of frailty (for which the CFS is an accurate proxy[9, 15]) better predicted risk of death than phenotypic models.

Although a number of published studies have shown similar results to ours in that frail patients are at greater risk for death and/or hospitalization,[22, 23, 24] there is surprisingly little literature on the comparative predictive performance of the different frailty instruments and the extent to which they overlap. Cigolle et al.[25] compared 3 frailty scales (the Functional Domain Model, the Burden Model, and the Fried score) in the Health and Retirement Study and, similarly to us, found that although 30.2% were frail on at least 1 of these scales, only 3.1% were deemed frail by all 3. The Conselice Study of Brain Aging[5] also reported that a deficit accumulation model defined a much higher prevalence of frailty (37.6%) than the 11.6% identified using the phenotypic Study of Osteoporotic Fractures (SOF) index based on weight loss, mobility, and level of energy. Another study[26] reported that risk models incorporating either the SOF index or the Fried score exhibited C statistics of only 0.61 for predicting falls in elderly females. A cohort study[27] from 2 English general medical units also found that none of the 5 frailty models was particularly accurate at predicting risk of readmission at 3 months, with C statistics ranging between 0.52 and 0.57. Although frailty assessment at time of hospital admission predicted in‐hospital mortality and length of stay in another English study, it was not independently associated with 30‐day outcomes after adjusting for age, sex, and comorbidities including dementia.[27] To our knowledge, these latter 2 are the only other studies reported to date performed in hospitalized patients to assess whether frailty assessment helps predict postdischarge outcomes. Thus, the poor C statistics we found for all of our frailty tools confirms prior literature that frailty assessment alone is inadequate to accurately identify those patients at highest risk for poor outcomes in the first 30 days after discharge. However, frailty assessment together with consideration of each individual's comorbidities, cognitive status, psychosocial circumstances, and environment can be useful to flag those individuals who may need extra attention postdischarge to optimize outcomes.

The hospital‐wide all‐cause 30‐day readmission rate is a key quality measure associated with patient outcomes, cost of care, and wasted hospital resources.[1] The estimated 20% readmission rate of Medicare patients and the associated $17 billion annual cost of readmissions led the Centers for Medicare and Medicaid Services (CMS) to implement policies that limit reimbursement for 30‐day unplanned readmissions and thus place hospitals with high readmission rates at financial risk.[1, 2]

The variation in readmission rates between hospitals is well documented in the literature.[3, 4] Singh et al. found that 9.3% of the variation in readmissions can be explained by hospital characteristics.[4] Hospital factors associated with lower readmission rates include not‐for‐profit ownership, hospital size, and nursing staffing levels.[5, 6, 7] Other studies found an association between environmental factors such as the percent of patients living under the poverty line and higher readmission rates.[7] The recent publicly available CMS data on readmission rates allows us to further our understanding of hospital characteristics that explain the variation in readmission rates. In this article, we are specifically interested in hospitalist staffing levels and hospital‐physician arrangements such as physician integration level and physician ownership. Moreover, we are interested in novel organizational variables, specifically, the adoption of a medical home model, which has been ignored by previous research. Medical homes are associated with better quality[8]; hospitals that already adopted the medical home model might be better equipped to coordinate care after the patients are discharged.

In recent years, the number of hospitals relying on hospitalists to provide inpatient care has been on the rise. As more hospitals employ hospitalists, it is important to understand how hospitalist staffing levels are associated with quality. Previous studies have linked hospitalists with lower hospital mortality rates,[7] lower cost of care,[9, 10] and lower readmission rates.[10, 11] Goodrich et al., on the other hand, did not find a significant relationship between the presence of hospitalists and mortality or readmission rates.[12] In a recent study, hospitalists indicated that heavy workloads limited the time they had available to communicate with patients, which negatively influenced quality and patient satisfaction, and resulted in delayed admissions and discharges.[13]

The main objective of this article was therefore to study the association between hospitalist staffing levels and hospital‐wide all‐cause readmission rates. Most empirical studies examining the relationship between hospitalist staffing and quality of inpatient care have predominantly focused on whether the presence of hospitalists who provided care at a hospital influenced mortality or readmissions.[11, 12] In this article, we contribute to the literature by examining how staffing levels measured by the ratio of hospitalists to general medical and surgical beds is associated with 30‐day readmission rates. We predict that there is a positive association between readmission rates and hospitalists per bed.

Hospitals have a broad range of contractual arrangements or integration levels with physicians, with employment being the highest level. A hospital can rely on physicians who have admitting privileges but are not salaried employees of the hospital to treat a large portion of its inpatient population. In the past few years, with the passage of the Patient Protection and Affordable Care Act (2010) and the shift in reimbursement towards Value Based Purchasing (VBP), more hospitals are choosing to ensure that physicians are strongly integrated within the hospital by adopting an employment‐based model. Moreover, hospitals view physician employment as a strategic move that will help ensure or expand their market share.[14] For instance, the number of surgeons who identified as self‐employed dropped from 48% in 2001 to 28% in 2011, and this reduction is attributed to the shift toward hospital employment of physicians.[15] Despite the evolving models of hospital‐physician arrangements, little is understood on how the adoption of the integrated salary model, in addition to the equity and foundation models, which are classified by Baker et al. as the highest level of integration, influence quality.[16] Therefore, another objective of this article was to examine the association between hospital‐physician arrangements and all‐cause unplanned readmission rates.

METHODS

RESULTS

Among our sample of 1756 hospitals, the median 30‐day all‐cause readmission rate was 16%, with the middle 50% of hospitals with readmission rates between 15.2% and 16.5%. All of the hospitals in this study reported that hospitalists provide care at the hospitals. The median Medicare share of hospital admissions was 48.46%, and the median Medicaid share of hospital admissions was 16.4%. Fifty‐one percent of the hospitals in our sample were fully integrated. Fifty percent of hospitals had 9 or fewer hospitalists per 100 general medical and surgical beds. Only 5% of the hospitals had partial or full physician ownership. Twenty‐nine percent of hospitals had an established medical home program. Table 1 provides summary statistics and the data sources of all the variables included in the study.

To compare readmission rates, we created a dummy variable that divided the sample into 2 categories: hospitals with low hospitalist staffing levels (hospitalists per general medical and surgical beds is less than the median) and high hospitalist staffing (hospitalists per general medical and surgical bed ratio is more than the median). We then used t tests to compare all‐cause readmission rates between hospitals with low and high hospitalist staffing levels, physician owned versus nonphysician owned, and fully integrated versus not fully integrated. We also used single‐factor analysis of variance (ANOVA) to compare readmission rates between nonteaching, minor teaching, and major teaching hospitals. Results are displayed in Table 2. There was a significant difference in the mean readmission rates between hospitals with low hospitalist staffing levels (mean readmission rate = 16.06%) versus high staffing levels (mean readmission rate = 15.72%). The mean readmission rate for physician‐owned hospitals was significantly lower than for nonphysician‐owned hospitals (15.46% vs 15.9%). Also, fully integrated hospitals had a lower readmission rate than hospitals where physicians were not fully integrated (15.93% vs 15.86%). Based on the ANOVA results, there was a significant difference between teaching levels. According to a Tukey honest significant difference post hoc test, there was no significant difference between nonteaching and minor teaching hospitals, but the readmission rate was significantly higher in major teaching hospitals (nonteaching = 15.83%, minor teaching = 15.76%, major teaching = 16.9%).

The OLS regression model was significant and explained 16% of the variability in readmission rates (Table 3). Higher hospitalists staffing levels were associated with lower 30‐day all cause readmission rates (P = 0.00). The addition of 1 hospitalist per general and surgical bed was associated with a 0.77 percentage points decrease in adjusted readmission rates. In terms of hospital‐physician arrangements, fully integrated hospitals had adjusted 30‐day all‐cause readmission rates 0.09 percentage points lower than nonfully integrated hospitals (P = 0.08). Physician partial or full ownership was significantly associated with lower readmission rates (P = 0.00); hospitals partially or fully owned by physicians had adjusted readmission rates 0.36 percentage points lower than nonphysician‐owned hospitals.

Based on the regression analysis, major teaching hospitals on average had adjusted readmission rates 1.03 percentage point higher than nonteaching hospitals (P = 0.000), whereas there was no significant difference between minor and nonteaching hospitals (P > 0.1). As the number of registered nurses (RNs) per 100 inpatient days increased by 1, readmission rates dropped by 0.17 (P = 0.00). Hospitals with higher Medicaid shares of admission had significantly higher readmission rates (P < 0.05). Hospitals located in counties with higher unemployment rates also had higher readmission rates (P = 0.000), whereas market competition had no significant association with readmissions. For‐profit hospitals had adjusted readmission rates 0.21 percentage points higher than not‐for‐profit hospitals (P = 0.01). Finally, hospitals that have adopted a medical home model had significantly lower readmission rates (P = 0.02); hospitals with an established medical home model had adjusted readmission rates 0.17 percentage points lower than their counterparts.

DISCUSSION

In the era of VBP and mounting pressures on hospitals to improve quality and lower cost, it is important to understand the association between modifiable hospital characteristics, such as hospitalist staffing levels, and unmodifiable characteristics, such as teaching status and size, with quality of care. There are many factors that can contribute to higher readmission rates. Some of these factors are hospital related and others are patient related, such as the environment in which a patient resides. Benbassat and Taragin argue that 9% to 48% of hospital readmissions are avoidable and are related to factors such as inadequate resolution of the problem the patient was admitted for and poor discharge care.[22] In this article, we have focused on hospital and market factors. Our main variables of interest were hospitalist staffing level, physician full integration, physician ownership, and the adoption of the medical home model at the hospital. Moreover, we examined the association between the hospital environment, specifically, market competition, and the patient environment, specifically, unemployment rates, with readmission rates.

Hospitalists' provision of inpatient care has been on the rise. From 1997 to 2006, the likelihood of receiving inpatient care from a hospitalist grew by 29.2% per year.[23] Based on AHA (2013) data, 65% of hospitals reported that hospitalists provided care at the hospital. The main driver behind the adoption of the hospitalists' model is the positive role hospitalists play in improving hospital efficiency and their familiarity and specialization in hospital care.[24] However, concerns exist that hospitalists might negatively influence patient outcomes given the discontinuity of care that occurs once the patient is discharged from the hospital and back to the care of their primary care physician.[25] Based on our analysis though, higher hospitalist staffing levels were associated with lower readmission rates. Therefore, to better understand the relationship between hospitalists and quality, it is important to account for staffing levels, not merely whether hospitalists provide care at the hospital or not. Higher patient load per hospitalist might still improve hospital efficiency by lowering costs, but is it likely to impede the quality of care provided by hospitalists. This is not surprising given similar findings, including in this article, which document a similar positive relationship between nursing staffing levels and quality.

Hospitals utilize various arrangements with physicians that range from employment to more relaxed arrangements such as physicians with privileges who are neither employed by the hospital nor under individual or group contracts. Historically, the main incentive for hospitals to integrate physicians was referrals to hospital services and specialties.[16, 26] The Affordable Care Act, however, provided further incentives, such as ease of care coordination, physicians' involvement, and commitment to quality improvement and cost‐containment efforts. Based on this study, hospitals that were classified as fully integrated had lower readmission rates. Also, hospitals partially or fully owned by physicians had better readmission rates. These findings indicate that hospital‐physician arrangements play a significant role not only in influencing efficiency and market share but also patient outcomes. Physician integration and physician ownership align physicians' financial incentives with those of the hospital. For instance, given the recent changes in reimbursement and the shift toward VBP, physician income in physician‐owned hospitals is at risk if the hospital has poor patient outcomes.

Other significant predictors of readmission rates included the adoption of the medical home model and RN staffing levels. Hospitals that adopted a medical home model and had a higher registered nurse‐to‐inpatient days ratio had significantly better readmission rates. The finding on the adoption of the medical home model is especially important. Previous research indicates that patient‐centered medical homes are associated with lower emergency room visits but not necessarily lower admissions.[27] Our findings indicate that medical homes might play a role in lowering readmission rates, and therefore this outcome needs to be included in studies examining the performance of medical homes. Critical access hospitals and those with higher admissions share of Medicaid patients had worst readmission rates. Finally, hospitals located in counties with higher unemployment rates also had the worst readmission rates. This finding is not surprising and is consistent with previous research, which indicates that the patients' environment and social risk factors play a significant role.

This article contributes to our understanding of readmission rates despite its several limitations, which include the measurement of hospitalist staffing levels based on general medical and surgical beds rather than general medicine admissions. Moreover, some hospitals had missing data on key variables, which warranted their exclusion from this study. In conclusion, many structural, operational and market‐level factors influence all‐cause readmission rates. However, some of these variables are modifiable and can thus be adjusted by a hospital to improve readmission rates. These variables include hospitalists and registered nurse staffing levels; physician integration through the salaried, equity, or foundation model; and the adoption of a medical home model.

The hospital‐wide all‐cause 30‐day readmission rate is a key quality measure associated with patient outcomes, cost of care, and wasted hospital resources.[1] The estimated 20% readmission rate of Medicare patients and the associated $17 billion annual cost of readmissions led the Centers for Medicare and Medicaid Services (CMS) to implement policies that limit reimbursement for 30‐day unplanned readmissions and thus place hospitals with high readmission rates at financial risk.[1, 2]

The variation in readmission rates between hospitals is well documented in the literature.[3, 4] Singh et al. found that 9.3% of the variation in readmissions can be explained by hospital characteristics.[4] Hospital factors associated with lower readmission rates include not‐for‐profit ownership, hospital size, and nursing staffing levels.[5, 6, 7] Other studies found an association between environmental factors such as the percent of patients living under the poverty line and higher readmission rates.[7] The recent publicly available CMS data on readmission rates allows us to further our understanding of hospital characteristics that explain the variation in readmission rates. In this article, we are specifically interested in hospitalist staffing levels and hospital‐physician arrangements such as physician integration level and physician ownership. Moreover, we are interested in novel organizational variables, specifically, the adoption of a medical home model, which has been ignored by previous research. Medical homes are associated with better quality[8]; hospitals that already adopted the medical home model might be better equipped to coordinate care after the patients are discharged.

In recent years, the number of hospitals relying on hospitalists to provide inpatient care has been on the rise. As more hospitals employ hospitalists, it is important to understand how hospitalist staffing levels are associated with quality. Previous studies have linked hospitalists with lower hospital mortality rates,[7] lower cost of care,[9, 10] and lower readmission rates.[10, 11] Goodrich et al., on the other hand, did not find a significant relationship between the presence of hospitalists and mortality or readmission rates.[12] In a recent study, hospitalists indicated that heavy workloads limited the time they had available to communicate with patients, which negatively influenced quality and patient satisfaction, and resulted in delayed admissions and discharges.[13]

The main objective of this article was therefore to study the association between hospitalist staffing levels and hospital‐wide all‐cause readmission rates. Most empirical studies examining the relationship between hospitalist staffing and quality of inpatient care have predominantly focused on whether the presence of hospitalists who provided care at a hospital influenced mortality or readmissions.[11, 12] In this article, we contribute to the literature by examining how staffing levels measured by the ratio of hospitalists to general medical and surgical beds is associated with 30‐day readmission rates. We predict that there is a positive association between readmission rates and hospitalists per bed.

Hospitals have a broad range of contractual arrangements or integration levels with physicians, with employment being the highest level. A hospital can rely on physicians who have admitting privileges but are not salaried employees of the hospital to treat a large portion of its inpatient population. In the past few years, with the passage of the Patient Protection and Affordable Care Act (2010) and the shift in reimbursement towards Value Based Purchasing (VBP), more hospitals are choosing to ensure that physicians are strongly integrated within the hospital by adopting an employment‐based model. Moreover, hospitals view physician employment as a strategic move that will help ensure or expand their market share.[14] For instance, the number of surgeons who identified as self‐employed dropped from 48% in 2001 to 28% in 2011, and this reduction is attributed to the shift toward hospital employment of physicians.[15] Despite the evolving models of hospital‐physician arrangements, little is understood on how the adoption of the integrated salary model, in addition to the equity and foundation models, which are classified by Baker et al. as the highest level of integration, influence quality.[16] Therefore, another objective of this article was to examine the association between hospital‐physician arrangements and all‐cause unplanned readmission rates.

METHODS

RESULTS

Among our sample of 1756 hospitals, the median 30‐day all‐cause readmission rate was 16%, with the middle 50% of hospitals with readmission rates between 15.2% and 16.5%. All of the hospitals in this study reported that hospitalists provide care at the hospitals. The median Medicare share of hospital admissions was 48.46%, and the median Medicaid share of hospital admissions was 16.4%. Fifty‐one percent of the hospitals in our sample were fully integrated. Fifty percent of hospitals had 9 or fewer hospitalists per 100 general medical and surgical beds. Only 5% of the hospitals had partial or full physician ownership. Twenty‐nine percent of hospitals had an established medical home program. Table 1 provides summary statistics and the data sources of all the variables included in the study.

To compare readmission rates, we created a dummy variable that divided the sample into 2 categories: hospitals with low hospitalist staffing levels (hospitalists per general medical and surgical beds is less than the median) and high hospitalist staffing (hospitalists per general medical and surgical bed ratio is more than the median). We then used t tests to compare all‐cause readmission rates between hospitals with low and high hospitalist staffing levels, physician owned versus nonphysician owned, and fully integrated versus not fully integrated. We also used single‐factor analysis of variance (ANOVA) to compare readmission rates between nonteaching, minor teaching, and major teaching hospitals. Results are displayed in Table 2. There was a significant difference in the mean readmission rates between hospitals with low hospitalist staffing levels (mean readmission rate = 16.06%) versus high staffing levels (mean readmission rate = 15.72%). The mean readmission rate for physician‐owned hospitals was significantly lower than for nonphysician‐owned hospitals (15.46% vs 15.9%). Also, fully integrated hospitals had a lower readmission rate than hospitals where physicians were not fully integrated (15.93% vs 15.86%). Based on the ANOVA results, there was a significant difference between teaching levels. According to a Tukey honest significant difference post hoc test, there was no significant difference between nonteaching and minor teaching hospitals, but the readmission rate was significantly higher in major teaching hospitals (nonteaching = 15.83%, minor teaching = 15.76%, major teaching = 16.9%).

The OLS regression model was significant and explained 16% of the variability in readmission rates (Table 3). Higher hospitalists staffing levels were associated with lower 30‐day all cause readmission rates (P = 0.00). The addition of 1 hospitalist per general and surgical bed was associated with a 0.77 percentage points decrease in adjusted readmission rates. In terms of hospital‐physician arrangements, fully integrated hospitals had adjusted 30‐day all‐cause readmission rates 0.09 percentage points lower than nonfully integrated hospitals (P = 0.08). Physician partial or full ownership was significantly associated with lower readmission rates (P = 0.00); hospitals partially or fully owned by physicians had adjusted readmission rates 0.36 percentage points lower than nonphysician‐owned hospitals.

Based on the regression analysis, major teaching hospitals on average had adjusted readmission rates 1.03 percentage point higher than nonteaching hospitals (P = 0.000), whereas there was no significant difference between minor and nonteaching hospitals (P > 0.1). As the number of registered nurses (RNs) per 100 inpatient days increased by 1, readmission rates dropped by 0.17 (P = 0.00). Hospitals with higher Medicaid shares of admission had significantly higher readmission rates (P < 0.05). Hospitals located in counties with higher unemployment rates also had higher readmission rates (P = 0.000), whereas market competition had no significant association with readmissions. For‐profit hospitals had adjusted readmission rates 0.21 percentage points higher than not‐for‐profit hospitals (P = 0.01). Finally, hospitals that have adopted a medical home model had significantly lower readmission rates (P = 0.02); hospitals with an established medical home model had adjusted readmission rates 0.17 percentage points lower than their counterparts.

DISCUSSION

In the era of VBP and mounting pressures on hospitals to improve quality and lower cost, it is important to understand the association between modifiable hospital characteristics, such as hospitalist staffing levels, and unmodifiable characteristics, such as teaching status and size, with quality of care. There are many factors that can contribute to higher readmission rates. Some of these factors are hospital related and others are patient related, such as the environment in which a patient resides. Benbassat and Taragin argue that 9% to 48% of hospital readmissions are avoidable and are related to factors such as inadequate resolution of the problem the patient was admitted for and poor discharge care.[22] In this article, we have focused on hospital and market factors. Our main variables of interest were hospitalist staffing level, physician full integration, physician ownership, and the adoption of the medical home model at the hospital. Moreover, we examined the association between the hospital environment, specifically, market competition, and the patient environment, specifically, unemployment rates, with readmission rates.

Hospitalists' provision of inpatient care has been on the rise. From 1997 to 2006, the likelihood of receiving inpatient care from a hospitalist grew by 29.2% per year.[23] Based on AHA (2013) data, 65% of hospitals reported that hospitalists provided care at the hospital. The main driver behind the adoption of the hospitalists' model is the positive role hospitalists play in improving hospital efficiency and their familiarity and specialization in hospital care.[24] However, concerns exist that hospitalists might negatively influence patient outcomes given the discontinuity of care that occurs once the patient is discharged from the hospital and back to the care of their primary care physician.[25] Based on our analysis though, higher hospitalist staffing levels were associated with lower readmission rates. Therefore, to better understand the relationship between hospitalists and quality, it is important to account for staffing levels, not merely whether hospitalists provide care at the hospital or not. Higher patient load per hospitalist might still improve hospital efficiency by lowering costs, but is it likely to impede the quality of care provided by hospitalists. This is not surprising given similar findings, including in this article, which document a similar positive relationship between nursing staffing levels and quality.

Hospitals utilize various arrangements with physicians that range from employment to more relaxed arrangements such as physicians with privileges who are neither employed by the hospital nor under individual or group contracts. Historically, the main incentive for hospitals to integrate physicians was referrals to hospital services and specialties.[16, 26] The Affordable Care Act, however, provided further incentives, such as ease of care coordination, physicians' involvement, and commitment to quality improvement and cost‐containment efforts. Based on this study, hospitals that were classified as fully integrated had lower readmission rates. Also, hospitals partially or fully owned by physicians had better readmission rates. These findings indicate that hospital‐physician arrangements play a significant role not only in influencing efficiency and market share but also patient outcomes. Physician integration and physician ownership align physicians' financial incentives with those of the hospital. For instance, given the recent changes in reimbursement and the shift toward VBP, physician income in physician‐owned hospitals is at risk if the hospital has poor patient outcomes.

Other significant predictors of readmission rates included the adoption of the medical home model and RN staffing levels. Hospitals that adopted a medical home model and had a higher registered nurse‐to‐inpatient days ratio had significantly better readmission rates. The finding on the adoption of the medical home model is especially important. Previous research indicates that patient‐centered medical homes are associated with lower emergency room visits but not necessarily lower admissions.[27] Our findings indicate that medical homes might play a role in lowering readmission rates, and therefore this outcome needs to be included in studies examining the performance of medical homes. Critical access hospitals and those with higher admissions share of Medicaid patients had worst readmission rates. Finally, hospitals located in counties with higher unemployment rates also had the worst readmission rates. This finding is not surprising and is consistent with previous research, which indicates that the patients' environment and social risk factors play a significant role.

This article contributes to our understanding of readmission rates despite its several limitations, which include the measurement of hospitalist staffing levels based on general medical and surgical beds rather than general medicine admissions. Moreover, some hospitals had missing data on key variables, which warranted their exclusion from this study. In conclusion, many structural, operational and market‐level factors influence all‐cause readmission rates. However, some of these variables are modifiable and can thus be adjusted by a hospital to improve readmission rates. These variables include hospitalists and registered nurse staffing levels; physician integration through the salaried, equity, or foundation model; and the adoption of a medical home model.

The hospital‐wide all‐cause 30‐day readmission rate is a key quality measure associated with patient outcomes, cost of care, and wasted hospital resources.[1] The estimated 20% readmission rate of Medicare patients and the associated $17 billion annual cost of readmissions led the Centers for Medicare and Medicaid Services (CMS) to implement policies that limit reimbursement for 30‐day unplanned readmissions and thus place hospitals with high readmission rates at financial risk.[1, 2]

The variation in readmission rates between hospitals is well documented in the literature.[3, 4] Singh et al. found that 9.3% of the variation in readmissions can be explained by hospital characteristics.[4] Hospital factors associated with lower readmission rates include not‐for‐profit ownership, hospital size, and nursing staffing levels.[5, 6, 7] Other studies found an association between environmental factors such as the percent of patients living under the poverty line and higher readmission rates.[7] The recent publicly available CMS data on readmission rates allows us to further our understanding of hospital characteristics that explain the variation in readmission rates. In this article, we are specifically interested in hospitalist staffing levels and hospital‐physician arrangements such as physician integration level and physician ownership. Moreover, we are interested in novel organizational variables, specifically, the adoption of a medical home model, which has been ignored by previous research. Medical homes are associated with better quality[8]; hospitals that already adopted the medical home model might be better equipped to coordinate care after the patients are discharged.

In recent years, the number of hospitals relying on hospitalists to provide inpatient care has been on the rise. As more hospitals employ hospitalists, it is important to understand how hospitalist staffing levels are associated with quality. Previous studies have linked hospitalists with lower hospital mortality rates,[7] lower cost of care,[9, 10] and lower readmission rates.[10, 11] Goodrich et al., on the other hand, did not find a significant relationship between the presence of hospitalists and mortality or readmission rates.[12] In a recent study, hospitalists indicated that heavy workloads limited the time they had available to communicate with patients, which negatively influenced quality and patient satisfaction, and resulted in delayed admissions and discharges.[13]

The main objective of this article was therefore to study the association between hospitalist staffing levels and hospital‐wide all‐cause readmission rates. Most empirical studies examining the relationship between hospitalist staffing and quality of inpatient care have predominantly focused on whether the presence of hospitalists who provided care at a hospital influenced mortality or readmissions.[11, 12] In this article, we contribute to the literature by examining how staffing levels measured by the ratio of hospitalists to general medical and surgical beds is associated with 30‐day readmission rates. We predict that there is a positive association between readmission rates and hospitalists per bed.

Hospitals have a broad range of contractual arrangements or integration levels with physicians, with employment being the highest level. A hospital can rely on physicians who have admitting privileges but are not salaried employees of the hospital to treat a large portion of its inpatient population. In the past few years, with the passage of the Patient Protection and Affordable Care Act (2010) and the shift in reimbursement towards Value Based Purchasing (VBP), more hospitals are choosing to ensure that physicians are strongly integrated within the hospital by adopting an employment‐based model. Moreover, hospitals view physician employment as a strategic move that will help ensure or expand their market share.[14] For instance, the number of surgeons who identified as self‐employed dropped from 48% in 2001 to 28% in 2011, and this reduction is attributed to the shift toward hospital employment of physicians.[15] Despite the evolving models of hospital‐physician arrangements, little is understood on how the adoption of the integrated salary model, in addition to the equity and foundation models, which are classified by Baker et al. as the highest level of integration, influence quality.[16] Therefore, another objective of this article was to examine the association between hospital‐physician arrangements and all‐cause unplanned readmission rates.

METHODS

RESULTS

Among our sample of 1756 hospitals, the median 30‐day all‐cause readmission rate was 16%, with the middle 50% of hospitals with readmission rates between 15.2% and 16.5%. All of the hospitals in this study reported that hospitalists provide care at the hospitals. The median Medicare share of hospital admissions was 48.46%, and the median Medicaid share of hospital admissions was 16.4%. Fifty‐one percent of the hospitals in our sample were fully integrated. Fifty percent of hospitals had 9 or fewer hospitalists per 100 general medical and surgical beds. Only 5% of the hospitals had partial or full physician ownership. Twenty‐nine percent of hospitals had an established medical home program. Table 1 provides summary statistics and the data sources of all the variables included in the study.

To compare readmission rates, we created a dummy variable that divided the sample into 2 categories: hospitals with low hospitalist staffing levels (hospitalists per general medical and surgical beds is less than the median) and high hospitalist staffing (hospitalists per general medical and surgical bed ratio is more than the median). We then used t tests to compare all‐cause readmission rates between hospitals with low and high hospitalist staffing levels, physician owned versus nonphysician owned, and fully integrated versus not fully integrated. We also used single‐factor analysis of variance (ANOVA) to compare readmission rates between nonteaching, minor teaching, and major teaching hospitals. Results are displayed in Table 2. There was a significant difference in the mean readmission rates between hospitals with low hospitalist staffing levels (mean readmission rate = 16.06%) versus high staffing levels (mean readmission rate = 15.72%). The mean readmission rate for physician‐owned hospitals was significantly lower than for nonphysician‐owned hospitals (15.46% vs 15.9%). Also, fully integrated hospitals had a lower readmission rate than hospitals where physicians were not fully integrated (15.93% vs 15.86%). Based on the ANOVA results, there was a significant difference between teaching levels. According to a Tukey honest significant difference post hoc test, there was no significant difference between nonteaching and minor teaching hospitals, but the readmission rate was significantly higher in major teaching hospitals (nonteaching = 15.83%, minor teaching = 15.76%, major teaching = 16.9%).

The OLS regression model was significant and explained 16% of the variability in readmission rates (Table 3). Higher hospitalists staffing levels were associated with lower 30‐day all cause readmission rates (P = 0.00). The addition of 1 hospitalist per general and surgical bed was associated with a 0.77 percentage points decrease in adjusted readmission rates. In terms of hospital‐physician arrangements, fully integrated hospitals had adjusted 30‐day all‐cause readmission rates 0.09 percentage points lower than nonfully integrated hospitals (P = 0.08). Physician partial or full ownership was significantly associated with lower readmission rates (P = 0.00); hospitals partially or fully owned by physicians had adjusted readmission rates 0.36 percentage points lower than nonphysician‐owned hospitals.

Based on the regression analysis, major teaching hospitals on average had adjusted readmission rates 1.03 percentage point higher than nonteaching hospitals (P = 0.000), whereas there was no significant difference between minor and nonteaching hospitals (P > 0.1). As the number of registered nurses (RNs) per 100 inpatient days increased by 1, readmission rates dropped by 0.17 (P = 0.00). Hospitals with higher Medicaid shares of admission had significantly higher readmission rates (P < 0.05). Hospitals located in counties with higher unemployment rates also had higher readmission rates (P = 0.000), whereas market competition had no significant association with readmissions. For‐profit hospitals had adjusted readmission rates 0.21 percentage points higher than not‐for‐profit hospitals (P = 0.01). Finally, hospitals that have adopted a medical home model had significantly lower readmission rates (P = 0.02); hospitals with an established medical home model had adjusted readmission rates 0.17 percentage points lower than their counterparts.

DISCUSSION

In the era of VBP and mounting pressures on hospitals to improve quality and lower cost, it is important to understand the association between modifiable hospital characteristics, such as hospitalist staffing levels, and unmodifiable characteristics, such as teaching status and size, with quality of care. There are many factors that can contribute to higher readmission rates. Some of these factors are hospital related and others are patient related, such as the environment in which a patient resides. Benbassat and Taragin argue that 9% to 48% of hospital readmissions are avoidable and are related to factors such as inadequate resolution of the problem the patient was admitted for and poor discharge care.[22] In this article, we have focused on hospital and market factors. Our main variables of interest were hospitalist staffing level, physician full integration, physician ownership, and the adoption of the medical home model at the hospital. Moreover, we examined the association between the hospital environment, specifically, market competition, and the patient environment, specifically, unemployment rates, with readmission rates.

Hospitalists' provision of inpatient care has been on the rise. From 1997 to 2006, the likelihood of receiving inpatient care from a hospitalist grew by 29.2% per year.[23] Based on AHA (2013) data, 65% of hospitals reported that hospitalists provided care at the hospital. The main driver behind the adoption of the hospitalists' model is the positive role hospitalists play in improving hospital efficiency and their familiarity and specialization in hospital care.[24] However, concerns exist that hospitalists might negatively influence patient outcomes given the discontinuity of care that occurs once the patient is discharged from the hospital and back to the care of their primary care physician.[25] Based on our analysis though, higher hospitalist staffing levels were associated with lower readmission rates. Therefore, to better understand the relationship between hospitalists and quality, it is important to account for staffing levels, not merely whether hospitalists provide care at the hospital or not. Higher patient load per hospitalist might still improve hospital efficiency by lowering costs, but is it likely to impede the quality of care provided by hospitalists. This is not surprising given similar findings, including in this article, which document a similar positive relationship between nursing staffing levels and quality.

Hospitals utilize various arrangements with physicians that range from employment to more relaxed arrangements such as physicians with privileges who are neither employed by the hospital nor under individual or group contracts. Historically, the main incentive for hospitals to integrate physicians was referrals to hospital services and specialties.[16, 26] The Affordable Care Act, however, provided further incentives, such as ease of care coordination, physicians' involvement, and commitment to quality improvement and cost‐containment efforts. Based on this study, hospitals that were classified as fully integrated had lower readmission rates. Also, hospitals partially or fully owned by physicians had better readmission rates. These findings indicate that hospital‐physician arrangements play a significant role not only in influencing efficiency and market share but also patient outcomes. Physician integration and physician ownership align physicians' financial incentives with those of the hospital. For instance, given the recent changes in reimbursement and the shift toward VBP, physician income in physician‐owned hospitals is at risk if the hospital has poor patient outcomes.

Other significant predictors of readmission rates included the adoption of the medical home model and RN staffing levels. Hospitals that adopted a medical home model and had a higher registered nurse‐to‐inpatient days ratio had significantly better readmission rates. The finding on the adoption of the medical home model is especially important. Previous research indicates that patient‐centered medical homes are associated with lower emergency room visits but not necessarily lower admissions.[27] Our findings indicate that medical homes might play a role in lowering readmission rates, and therefore this outcome needs to be included in studies examining the performance of medical homes. Critical access hospitals and those with higher admissions share of Medicaid patients had worst readmission rates. Finally, hospitals located in counties with higher unemployment rates also had the worst readmission rates. This finding is not surprising and is consistent with previous research, which indicates that the patients' environment and social risk factors play a significant role.

This article contributes to our understanding of readmission rates despite its several limitations, which include the measurement of hospitalist staffing levels based on general medical and surgical beds rather than general medicine admissions. Moreover, some hospitals had missing data on key variables, which warranted their exclusion from this study. In conclusion, many structural, operational and market‐level factors influence all‐cause readmission rates. However, some of these variables are modifiable and can thus be adjusted by a hospital to improve readmission rates. These variables include hospitalists and registered nurse staffing levels; physician integration through the salaried, equity, or foundation model; and the adoption of a medical home model.