Patient Safety Indicator-12 Rarely Identifies Problems with Quality of Care in Perioperative Venous Thromboembolism

Perioperative venous thromboembolism (VTE) is a major contributor to the morbidity and mortality of hospitalized patients. The historical incidence of postoperative VTE varied between 15%-80% depending on the type of procedure and monitoring strategies. Higher incidences of VTE occurred with major surgery (15%-40%), knee or hip arthroplasty (40%-60%), and trauma (60%-80%).¹ The use of VTE prophylaxis with subcutaneous heparin reduced DVTs by 70% and PEs by 50%.² A recent study from Olmstead County showed improved adherence to inpatient VTE prophylaxis from 2005 to 2010 but no difference in VTE incidence. However, 52% of VTE events were associated with hospitalization.³ As such, VTE continues to be a healthcare-associated adverse event, and surgery remains a significant risk factor for thrombosis.⁴

The Agency for Healthcare Research and Quality (AHRQ) released Patient Safety Indicators (PSI) in 2003 to provide a means of screening for adverse events.⁵ Over time, PSIs have been adopted as a measure of hospital performance and are utilized in several pay-for-performance programs. PSI-90 is a composite measure of several other PSIs and is a core metric in the Centers for Medicare and Medicaid Services (CMS) Hospital-Acquired Condition Reduction Program and the Hospital Value-Based Purchasing Program which impacts up to 2% of a hospital’s Medicare payments.^6,7 One component of PSI-90 is PSI-12, which captures perioperative VTE. PSI-12 events are identified using software that screens medical records based on International Classification of Diseases (ICD)-9/10 codes for thrombosis and procedure codes at time of discharge.⁸

Over the last several years, there has been concern regarding the validity of including PSI-12 in pay-for-performance metrics. Common areas for concern include PSI-12’s accuracy in detecting true postoperative VTE⁹ in addition to surveillance bias.^10,11 However, some note that PSI-12 is useful when applied with its original intent: as a screening tool for hospitals to identify specific areas to implement improvements.^9,12The aim of our study was to review all PSI-12 events at our institution to evaluate the accuracy of PSI-12 and identify areas for improvement to prevent VTE events in surgical patients. While several other studies have looked at the positive predictive values, accuracy, or surveillance bias of PSI-12, to the best of our knowledge, few, if any, previous studies have reported PSI-12 events in relation to their timing, type of prophylaxis used, and mitigating factors to identify areas for quality improvement.

METHODS

PSI-12 events were identified between June 2015 and June 2017 using AHRQ software (version 5). Cases were also identified through Vizient and reviewed to ensure congruence between the methods. Patients’ electronic medical records were reviewed for patient demographics, type of VTE event, platelet count at VTE diagnosis, procedure type, and both the timing and type of VTE prophylaxis. Summary statistics were calculated.

 

We considered perioperative VTE pharmacologic prophylaxis appropriate if started within 24 hours of a low bleeding risk procedure or 72 hours of a high-bleeding risk procedure.¹³ Mechanical prophylaxis was considered appropriate if pharmacologic prophylaxis was not used because of procedure risk, thrombocytopenia, or active bleeding. The medication administration record was reviewed to determine if prophylaxis was ordered, given, and/or refused.

RESULTS

During the two-year period, 18,084 surgeries were performed, and 161 cases of VTE events were identified. A detailed chart review and correction of documentation led to the exclusion of seven cases (4%) because the VTE event occurred prior to admission (n = 5) or were incidental findings that did not meet the Uniform Hospital Discharge Data Set definition for reporting (n = 2). In total, 154 (0.9% of all surgeries) cases were considered PSI-12 events. Pulmonary embolism (PE) occurred in most cases (n = 97, 62.9%), followed by deep vein thrombosis (DVT) (n = 37, 24.0%). Twenty cases (12.9%) experienced concurrent PE and DVT. Within the PE group, 16 cases (14%) were subsegmental PE only. Eight patients (14% of DVT cases) had only a distal DVT. The mean age of patients was 56 years (+/− 16 years), and the majority (59%) were male. The clinical specialties with the most events included neurosurgery (21%), orthopedics (14%), general surgery (13%), and trauma (11%). Fourteen patients (9%) died during the hospitalization, and of these, six (43%) had either sudden death or death attributed to PE (Table 1).

Cases were also reviewed for the type of VTE prophylactic strategy administered at the time of the event. The top three prophylactic strategies were subcutaneous unfractionated heparin (61%), mechanical prophylaxis only (51%), and enoxaparin (31%). Nine cases of VTE occurred during therapeutic anticoagulation (6%; Table 1).

We also evaluated the timing of VTE in relation to hospitalization and procedure. Overall, the median length of hospital stay was 21 days (range: 11-39 days). VTE occurred early in the hospitalization; 21% of cases of VTE occurred within three days of admission, and 43.5% occurred within seven days of admission (Figure 1). With regard to VTE timing in relation to the procedure, 4.5% of cases of VTE occurred prior to the procedure, 33% occurred within three days of the procedure, and 53% occurred within seven days (Figure 2).

Absence of guideline-appropriate VTE prophylaxis was identified in only nine (6%) cases: seven patients had delayed initiation of pharmacologic prophylaxis, and two had pharmacologic prophylaxis held for unknown reasons. When accounting for pharmacologic prophylaxis missed based on patient refusal (n = 10 patients), the number of patients without guideline- appropriate VTE prophylaxis increased to 17 cases (11%), as two of the cases with patient refusal were found to have other quality issues present. Pharmacologic prophylaxis was given to 125 patients during their hospitalization. A median of 8% of ordered doses was refused, and an additional 8% of doses were held for a procedure (Table 2). We evaluated other factors that could have influenced the rate or type of VTE prophylactic strategies toward the use of mechanical prophylaxis, including thrombocytopenia and trauma. Although 11% of cases were treated primarily by the trauma teams (Table 1), a trauma-related procedure accounted for 29% of PSI-12 cases. Thrombocytopenia (platelets of less than 100,000) occurred in 53 cases (34%), with 27 patients (18%) having a platelet count of less than 50,000.

 

DISCUSSION

Utilizing AHRQ version 5 software for PSI-12, our institution identified 154 cases of perioperative VTE. Most of these cases were a pulmonary embolism, occurred within a week of admission, and were associated with surgical specialties that portend a higher risk of VTE. Very few of these cases were deficient in guideline-directed VTE prophylaxis, and several cases had associated factors such as trauma and thrombocytopenia that may have appropriately influenced the decision to use mechanical only prophylaxis. Sixteen percent of pharmacologic VTE prophylactic doses were refused or held for a procedure, a known but rarely quantified influence on rates of pharmacologic prophylaxis. The use of patient-level data and adjudication of the clinical decision that affected the administration of VTE prophylaxis is a major strength of this work. In all, our data raise several questions about the accuracy of PSI-12 in identifying preventable postoperative VTE, especially as it is utilized as a marker for pay-for-performance measures in addition to identifying further areas for research and improvement.

Our results align with previous studies that suggest that PSI-12 is an inaccurate measure of performance quality. A study by Bilimoria et al. in 2013 noted that surveillance biases associated with PSI-12, showing that hospitals with higher compliance to appropriate VTE prophylaxis paradoxically had worse outcomes.¹⁰ Furthermore, Blay et al. recently published a study evaluating PSI-90 scores for hospitals with and without the VTE measure included. Their results indicated that larger hospitals (teaching hospitals, level I trauma centers, etc) caring for sicker patients were noted to improve by 8%-25% when the VTE measure was removed.¹¹ Similarly, our data indicate that the PSI-12 may not be an accurate measure of quality performance, as only 6% of cases were noted to be deficient in appropriate guideline-directed prophylaxis. Even when accounting for the refusal of doses of pharmacologic prophylaxis, this figure only increased to 11%.

Procedures included in the current PSI-12 algorithm also vary in the risk they pose to developing VTE. For example, the current version of PSI-12 includes surgical Medicare Severity Diagnosis Related Groups (MS-DRGs) for procedures with a high risk of VTE such as orthopedic, abdominal, or thoracic procedures.^14,15, However, it is worth noting that procedures such as tracheostomy and ocular surgery are also included.¹⁴ The variation in risk for development of VTE is reflected in the Caprini score, a perioperative risk stratification tool. These latter procedures only contribute one point, whereas trauma would add five points each to the total score, and make the patient a high VTE risk from the procedure alone.¹⁶ With regard to PSI-12, in theory, scores could vary significantly between centers even if the quality of care is the same, based on the volume and risk of procedures performed.

While most of our cases of VTE occurred within higher risk surgical subspecialties, 15 (10%) of our cases were within the clinical specialties of interventional radiology, cardiology, gastroenterology, and bone marrow transplant. One procedure of notable conflict includes bronchoalveolar lavage (BAL), which is included as an operating room procedure per the current version of PSI-12 software,¹⁷ but is no longer recognized by CMS as a surgical MS-DRG for reimbursement. The PSI-12 observed-to-expected rates take the DRG and comorbidity codes into consideration, but which DRG is selected does not always reflect the procedure type or the risk of VTE associated with the procedure.

Regarding the type of VTE event, our data revealed that PE was the predominant event, accounting for 62% of cases. This is different compared with other studies, such as the population-based studies in which PE and DVT account for approximately 40%-42% of events, respectively, and approximately 15% with concurrent PE and DVT. ^9,18 Borzecki et al. however, noted similar rates to our study, with 55% as PE only, 38% as DVT only, and 8% had both PE and DVT. This study also found a positive correlation between PSI-12 rates and VTE imaging rates, such that if more CT scans were completed, more PEs could be found.¹⁹ Our data identified 16 of the 97 cases of PE were subsegmental PE, a subset of VTE whose clinical significance has been questioned.²⁰ Excluding these cases brings the percentage closer to 52%. Also, screening ultrasounds are not performed at our institution. Asymptomatic or minimally symptomatic DVTs could be missed.

Further, this study also highlights that the reliability of PSI-12 is dependent on accurate documentation and coding. This is most evident when reviewing studies that evaluated the positive predictive value (PPV) of this measure.^9,12,21 A study of 28 Veteran’s Affairs hospitals from 2003 to 2007 used the AHRQ version 3 software to assess the PPV of PSI-12. Out of the 112 cases flagged by the AHRQ PSI software, only 48 were true events of postoperative DVT, yielding a PPV of only 43%. False-positive results were primarily patients with VTE present on admission and cases that were diagnosed after admission but prior to the index procedure. They also noted that coding inaccuracies were present in 38% of cases.⁹ Similarly, Henderson et al. conducted a retrospective review of 112 postsurgical discharges noting a PPV of 54%, with most false-positives resulting from superficial clots identified by PSI-12 related to coding ambiguity.¹² Our data similarly showed false-positive results as seven cases were excluded based on chart review and documentation correction, and 4.5% were preprocedural. However, there is some discordance with previous studies, as our data yields a PPV of 91% for VTE when accounting for preprocedural events as well as those excluded based on chart review. One explanation is an improvement in documentation strategies and coding, as our institution has adopted strategies to review cases and clarify documentation prior to billing to improve accuracy, and inclusion of a checkbox to indicate that a diagnosis was present on admission. Another possibility is improved accuracy with ICD-10 coding, as shown in a paper by Quan et al. that found a 79% PPV of PSI-12, which improved to 89% when cases present on admission were excluded.²¹ Lastly, we used newer versions of the AHRQ software, which could have improved the accuracy of detection. Despite these improvements in PSI-12 identifying true cases of postoperative VTE, as our data show, there is still much to be desired in terms of identifying issues with the quality of care and inclusion of PSI-12 in pay-for-performance.

Our study has several limitations. As a retrospective review, a major limiting factor is that data obtained are subject to the accuracy of documentation within the provider and nursing notes. As such, we focused on broad topics such as VTE events, the type of VTE event, and timing of the event in relation to admission and procedure. We actively review cases at discharge prior to billing to correct documentation if required. However, a prolonged hospital stay could lead to a review of the case weeks to months later. A real-time alert for a VTE event in a patient with surgery could improve documentation.

Further, although mechanical only prophylaxis was present on chart review, it is impossible to know both the rate of compliance with this method and whether it contributed to some events. Lastly, our study evaluated primarily the PPV of PSI-12. We did not directly evaluate the negative predictive value of PSI-12, which could mean there are cases of preventable VTE that are missed entirely.

Despite these limitations, our goals of evaluating the validity of PSI-12 and identifying areas for improved measurement and techniques for DVT prophylaxis were met. As previously discussed, our data suggest that PSI-12 has several limitations in identifying quality of care issues with the prevention of DVT. For example, PSI-12 included many cases in which pharmacologic prophylaxis was appropriately not given. Approximately 50% of cases occurred in patients who were thrombocytopenic (platelets < 100,000), and 29% of events occurred in trauma patients. In addition, 33% of cases identified were within three days of the procedure, which, in cases of high-bleeding risk procedures, is an appropriate time to refrain from pharmacologic anticoagulation.¹³ In cases such as these, it may be worth evaluating alternative forms of mechanical only prophylaxis, or other strategies to mitigate this risk.

This study also highlights areas for further research. Many of the VTE events in this study occurred while patients were treated with appropriate prophylaxis, as other studies have also shown.^22,23 Gangireddy et al. looked at demographic and clinical information surrounding postoperative VTE and found several other risk factors that incur a higher risk of VTE, including steroid use, infections, and myocardial infarction.²⁴ Perhaps future studies could target these groups as potential points for increased prophylactic strategies. As noted by Lau et al. appropriate VTE prophylaxis involves risk stratification, ordering the appropriate prophylaxis by clinicians, patient acceptance of prescribed therapy, and nursing administration of prescribed therapy.²⁵ As exemplified by our data, despite appropriate prophylaxis being ordered, eight events were associated solely with the refusal of pharmacologic prophylaxis. An ideal VTE metric would identify patients with a VTE who had a defective VTE prophylactic process,²⁵ but the cost associated with manual data abstraction may limit the inclusion of a process metric into pay-for-performance. Additional research also needs to identify whether modifications to PSI-12 can improve its accuracy and predictive value, such as the exclusion of VTE prior to a procedure, modifications to what counts as a procedure, or utilizing markers to assess adherence to VTE prophylactic rates. These points are especially important to consider if PSI-12 is to remain as one of the key factors in pay-for-performance outcome measures. Lastly, understanding the effectiveness and patient adherence to oral VTE prophylactic regimens could also decrease the rates of refusal of VTE prophylaxis.

Overall, VTE remains a large contributor to morbidity and mortality among hospitalized surgical patients. While there is utility in PSI-12 as a screening tool to identify these events and potential areas for improved processes to decrease VTE, its usefulness for detection of true quality issues and its utilization in pay-for-performance is questionable. The universally high rates of VTE prophylaxis question the need for a VTE prophylactic metric. However, if these metrics are going to continue to be used in determining payments, modifications to the current processes and algorithms are needed to improve accuracy in identifying issues in quality of care.

 

Perioperative venous thromboembolism (VTE) is a major contributor to the morbidity and mortality of hospitalized patients. The historical incidence of postoperative VTE varied between 15%-80% depending on the type of procedure and monitoring strategies. Higher incidences of VTE occurred with major surgery (15%-40%), knee or hip arthroplasty (40%-60%), and trauma (60%-80%).¹ The use of VTE prophylaxis with subcutaneous heparin reduced DVTs by 70% and PEs by 50%.² A recent study from Olmstead County showed improved adherence to inpatient VTE prophylaxis from 2005 to 2010 but no difference in VTE incidence. However, 52% of VTE events were associated with hospitalization.³ As such, VTE continues to be a healthcare-associated adverse event, and surgery remains a significant risk factor for thrombosis.⁴

The Agency for Healthcare Research and Quality (AHRQ) released Patient Safety Indicators (PSI) in 2003 to provide a means of screening for adverse events.⁵ Over time, PSIs have been adopted as a measure of hospital performance and are utilized in several pay-for-performance programs. PSI-90 is a composite measure of several other PSIs and is a core metric in the Centers for Medicare and Medicaid Services (CMS) Hospital-Acquired Condition Reduction Program and the Hospital Value-Based Purchasing Program which impacts up to 2% of a hospital’s Medicare payments.^6,7 One component of PSI-90 is PSI-12, which captures perioperative VTE. PSI-12 events are identified using software that screens medical records based on International Classification of Diseases (ICD)-9/10 codes for thrombosis and procedure codes at time of discharge.⁸

Over the last several years, there has been concern regarding the validity of including PSI-12 in pay-for-performance metrics. Common areas for concern include PSI-12’s accuracy in detecting true postoperative VTE⁹ in addition to surveillance bias.^10,11 However, some note that PSI-12 is useful when applied with its original intent: as a screening tool for hospitals to identify specific areas to implement improvements.^9,12The aim of our study was to review all PSI-12 events at our institution to evaluate the accuracy of PSI-12 and identify areas for improvement to prevent VTE events in surgical patients. While several other studies have looked at the positive predictive values, accuracy, or surveillance bias of PSI-12, to the best of our knowledge, few, if any, previous studies have reported PSI-12 events in relation to their timing, type of prophylaxis used, and mitigating factors to identify areas for quality improvement.

METHODS

PSI-12 events were identified between June 2015 and June 2017 using AHRQ software (version 5). Cases were also identified through Vizient and reviewed to ensure congruence between the methods. Patients’ electronic medical records were reviewed for patient demographics, type of VTE event, platelet count at VTE diagnosis, procedure type, and both the timing and type of VTE prophylaxis. Summary statistics were calculated.

 

We considered perioperative VTE pharmacologic prophylaxis appropriate if started within 24 hours of a low bleeding risk procedure or 72 hours of a high-bleeding risk procedure.¹³ Mechanical prophylaxis was considered appropriate if pharmacologic prophylaxis was not used because of procedure risk, thrombocytopenia, or active bleeding. The medication administration record was reviewed to determine if prophylaxis was ordered, given, and/or refused.

RESULTS

During the two-year period, 18,084 surgeries were performed, and 161 cases of VTE events were identified. A detailed chart review and correction of documentation led to the exclusion of seven cases (4%) because the VTE event occurred prior to admission (n = 5) or were incidental findings that did not meet the Uniform Hospital Discharge Data Set definition for reporting (n = 2). In total, 154 (0.9% of all surgeries) cases were considered PSI-12 events. Pulmonary embolism (PE) occurred in most cases (n = 97, 62.9%), followed by deep vein thrombosis (DVT) (n = 37, 24.0%). Twenty cases (12.9%) experienced concurrent PE and DVT. Within the PE group, 16 cases (14%) were subsegmental PE only. Eight patients (14% of DVT cases) had only a distal DVT. The mean age of patients was 56 years (+/− 16 years), and the majority (59%) were male. The clinical specialties with the most events included neurosurgery (21%), orthopedics (14%), general surgery (13%), and trauma (11%). Fourteen patients (9%) died during the hospitalization, and of these, six (43%) had either sudden death or death attributed to PE (Table 1).

Cases were also reviewed for the type of VTE prophylactic strategy administered at the time of the event. The top three prophylactic strategies were subcutaneous unfractionated heparin (61%), mechanical prophylaxis only (51%), and enoxaparin (31%). Nine cases of VTE occurred during therapeutic anticoagulation (6%; Table 1).

We also evaluated the timing of VTE in relation to hospitalization and procedure. Overall, the median length of hospital stay was 21 days (range: 11-39 days). VTE occurred early in the hospitalization; 21% of cases of VTE occurred within three days of admission, and 43.5% occurred within seven days of admission (Figure 1). With regard to VTE timing in relation to the procedure, 4.5% of cases of VTE occurred prior to the procedure, 33% occurred within three days of the procedure, and 53% occurred within seven days (Figure 2).

Absence of guideline-appropriate VTE prophylaxis was identified in only nine (6%) cases: seven patients had delayed initiation of pharmacologic prophylaxis, and two had pharmacologic prophylaxis held for unknown reasons. When accounting for pharmacologic prophylaxis missed based on patient refusal (n = 10 patients), the number of patients without guideline- appropriate VTE prophylaxis increased to 17 cases (11%), as two of the cases with patient refusal were found to have other quality issues present. Pharmacologic prophylaxis was given to 125 patients during their hospitalization. A median of 8% of ordered doses was refused, and an additional 8% of doses were held for a procedure (Table 2). We evaluated other factors that could have influenced the rate or type of VTE prophylactic strategies toward the use of mechanical prophylaxis, including thrombocytopenia and trauma. Although 11% of cases were treated primarily by the trauma teams (Table 1), a trauma-related procedure accounted for 29% of PSI-12 cases. Thrombocytopenia (platelets of less than 100,000) occurred in 53 cases (34%), with 27 patients (18%) having a platelet count of less than 50,000.

 

DISCUSSION

Utilizing AHRQ version 5 software for PSI-12, our institution identified 154 cases of perioperative VTE. Most of these cases were a pulmonary embolism, occurred within a week of admission, and were associated with surgical specialties that portend a higher risk of VTE. Very few of these cases were deficient in guideline-directed VTE prophylaxis, and several cases had associated factors such as trauma and thrombocytopenia that may have appropriately influenced the decision to use mechanical only prophylaxis. Sixteen percent of pharmacologic VTE prophylactic doses were refused or held for a procedure, a known but rarely quantified influence on rates of pharmacologic prophylaxis. The use of patient-level data and adjudication of the clinical decision that affected the administration of VTE prophylaxis is a major strength of this work. In all, our data raise several questions about the accuracy of PSI-12 in identifying preventable postoperative VTE, especially as it is utilized as a marker for pay-for-performance measures in addition to identifying further areas for research and improvement.

Our results align with previous studies that suggest that PSI-12 is an inaccurate measure of performance quality. A study by Bilimoria et al. in 2013 noted that surveillance biases associated with PSI-12, showing that hospitals with higher compliance to appropriate VTE prophylaxis paradoxically had worse outcomes.¹⁰ Furthermore, Blay et al. recently published a study evaluating PSI-90 scores for hospitals with and without the VTE measure included. Their results indicated that larger hospitals (teaching hospitals, level I trauma centers, etc) caring for sicker patients were noted to improve by 8%-25% when the VTE measure was removed.¹¹ Similarly, our data indicate that the PSI-12 may not be an accurate measure of quality performance, as only 6% of cases were noted to be deficient in appropriate guideline-directed prophylaxis. Even when accounting for the refusal of doses of pharmacologic prophylaxis, this figure only increased to 11%.

Procedures included in the current PSI-12 algorithm also vary in the risk they pose to developing VTE. For example, the current version of PSI-12 includes surgical Medicare Severity Diagnosis Related Groups (MS-DRGs) for procedures with a high risk of VTE such as orthopedic, abdominal, or thoracic procedures.^14,15, However, it is worth noting that procedures such as tracheostomy and ocular surgery are also included.¹⁴ The variation in risk for development of VTE is reflected in the Caprini score, a perioperative risk stratification tool. These latter procedures only contribute one point, whereas trauma would add five points each to the total score, and make the patient a high VTE risk from the procedure alone.¹⁶ With regard to PSI-12, in theory, scores could vary significantly between centers even if the quality of care is the same, based on the volume and risk of procedures performed.

While most of our cases of VTE occurred within higher risk surgical subspecialties, 15 (10%) of our cases were within the clinical specialties of interventional radiology, cardiology, gastroenterology, and bone marrow transplant. One procedure of notable conflict includes bronchoalveolar lavage (BAL), which is included as an operating room procedure per the current version of PSI-12 software,¹⁷ but is no longer recognized by CMS as a surgical MS-DRG for reimbursement. The PSI-12 observed-to-expected rates take the DRG and comorbidity codes into consideration, but which DRG is selected does not always reflect the procedure type or the risk of VTE associated with the procedure.

Regarding the type of VTE event, our data revealed that PE was the predominant event, accounting for 62% of cases. This is different compared with other studies, such as the population-based studies in which PE and DVT account for approximately 40%-42% of events, respectively, and approximately 15% with concurrent PE and DVT. ^9,18 Borzecki et al. however, noted similar rates to our study, with 55% as PE only, 38% as DVT only, and 8% had both PE and DVT. This study also found a positive correlation between PSI-12 rates and VTE imaging rates, such that if more CT scans were completed, more PEs could be found.¹⁹ Our data identified 16 of the 97 cases of PE were subsegmental PE, a subset of VTE whose clinical significance has been questioned.²⁰ Excluding these cases brings the percentage closer to 52%. Also, screening ultrasounds are not performed at our institution. Asymptomatic or minimally symptomatic DVTs could be missed.

Further, this study also highlights that the reliability of PSI-12 is dependent on accurate documentation and coding. This is most evident when reviewing studies that evaluated the positive predictive value (PPV) of this measure.^9,12,21 A study of 28 Veteran’s Affairs hospitals from 2003 to 2007 used the AHRQ version 3 software to assess the PPV of PSI-12. Out of the 112 cases flagged by the AHRQ PSI software, only 48 were true events of postoperative DVT, yielding a PPV of only 43%. False-positive results were primarily patients with VTE present on admission and cases that were diagnosed after admission but prior to the index procedure. They also noted that coding inaccuracies were present in 38% of cases.⁹ Similarly, Henderson et al. conducted a retrospective review of 112 postsurgical discharges noting a PPV of 54%, with most false-positives resulting from superficial clots identified by PSI-12 related to coding ambiguity.¹² Our data similarly showed false-positive results as seven cases were excluded based on chart review and documentation correction, and 4.5% were preprocedural. However, there is some discordance with previous studies, as our data yields a PPV of 91% for VTE when accounting for preprocedural events as well as those excluded based on chart review. One explanation is an improvement in documentation strategies and coding, as our institution has adopted strategies to review cases and clarify documentation prior to billing to improve accuracy, and inclusion of a checkbox to indicate that a diagnosis was present on admission. Another possibility is improved accuracy with ICD-10 coding, as shown in a paper by Quan et al. that found a 79% PPV of PSI-12, which improved to 89% when cases present on admission were excluded.²¹ Lastly, we used newer versions of the AHRQ software, which could have improved the accuracy of detection. Despite these improvements in PSI-12 identifying true cases of postoperative VTE, as our data show, there is still much to be desired in terms of identifying issues with the quality of care and inclusion of PSI-12 in pay-for-performance.

Our study has several limitations. As a retrospective review, a major limiting factor is that data obtained are subject to the accuracy of documentation within the provider and nursing notes. As such, we focused on broad topics such as VTE events, the type of VTE event, and timing of the event in relation to admission and procedure. We actively review cases at discharge prior to billing to correct documentation if required. However, a prolonged hospital stay could lead to a review of the case weeks to months later. A real-time alert for a VTE event in a patient with surgery could improve documentation.

Further, although mechanical only prophylaxis was present on chart review, it is impossible to know both the rate of compliance with this method and whether it contributed to some events. Lastly, our study evaluated primarily the PPV of PSI-12. We did not directly evaluate the negative predictive value of PSI-12, which could mean there are cases of preventable VTE that are missed entirely.

Despite these limitations, our goals of evaluating the validity of PSI-12 and identifying areas for improved measurement and techniques for DVT prophylaxis were met. As previously discussed, our data suggest that PSI-12 has several limitations in identifying quality of care issues with the prevention of DVT. For example, PSI-12 included many cases in which pharmacologic prophylaxis was appropriately not given. Approximately 50% of cases occurred in patients who were thrombocytopenic (platelets < 100,000), and 29% of events occurred in trauma patients. In addition, 33% of cases identified were within three days of the procedure, which, in cases of high-bleeding risk procedures, is an appropriate time to refrain from pharmacologic anticoagulation.¹³ In cases such as these, it may be worth evaluating alternative forms of mechanical only prophylaxis, or other strategies to mitigate this risk.

This study also highlights areas for further research. Many of the VTE events in this study occurred while patients were treated with appropriate prophylaxis, as other studies have also shown.^22,23 Gangireddy et al. looked at demographic and clinical information surrounding postoperative VTE and found several other risk factors that incur a higher risk of VTE, including steroid use, infections, and myocardial infarction.²⁴ Perhaps future studies could target these groups as potential points for increased prophylactic strategies. As noted by Lau et al. appropriate VTE prophylaxis involves risk stratification, ordering the appropriate prophylaxis by clinicians, patient acceptance of prescribed therapy, and nursing administration of prescribed therapy.²⁵ As exemplified by our data, despite appropriate prophylaxis being ordered, eight events were associated solely with the refusal of pharmacologic prophylaxis. An ideal VTE metric would identify patients with a VTE who had a defective VTE prophylactic process,²⁵ but the cost associated with manual data abstraction may limit the inclusion of a process metric into pay-for-performance. Additional research also needs to identify whether modifications to PSI-12 can improve its accuracy and predictive value, such as the exclusion of VTE prior to a procedure, modifications to what counts as a procedure, or utilizing markers to assess adherence to VTE prophylactic rates. These points are especially important to consider if PSI-12 is to remain as one of the key factors in pay-for-performance outcome measures. Lastly, understanding the effectiveness and patient adherence to oral VTE prophylactic regimens could also decrease the rates of refusal of VTE prophylaxis.

Overall, VTE remains a large contributor to morbidity and mortality among hospitalized surgical patients. While there is utility in PSI-12 as a screening tool to identify these events and potential areas for improved processes to decrease VTE, its usefulness for detection of true quality issues and its utilization in pay-for-performance is questionable. The universally high rates of VTE prophylaxis question the need for a VTE prophylactic metric. However, if these metrics are going to continue to be used in determining payments, modifications to the current processes and algorithms are needed to improve accuracy in identifying issues in quality of care.

 

Perioperative venous thromboembolism (VTE) is a major contributor to the morbidity and mortality of hospitalized patients. The historical incidence of postoperative VTE varied between 15%-80% depending on the type of procedure and monitoring strategies. Higher incidences of VTE occurred with major surgery (15%-40%), knee or hip arthroplasty (40%-60%), and trauma (60%-80%).¹ The use of VTE prophylaxis with subcutaneous heparin reduced DVTs by 70% and PEs by 50%.² A recent study from Olmstead County showed improved adherence to inpatient VTE prophylaxis from 2005 to 2010 but no difference in VTE incidence. However, 52% of VTE events were associated with hospitalization.³ As such, VTE continues to be a healthcare-associated adverse event, and surgery remains a significant risk factor for thrombosis.⁴

The Agency for Healthcare Research and Quality (AHRQ) released Patient Safety Indicators (PSI) in 2003 to provide a means of screening for adverse events.⁵ Over time, PSIs have been adopted as a measure of hospital performance and are utilized in several pay-for-performance programs. PSI-90 is a composite measure of several other PSIs and is a core metric in the Centers for Medicare and Medicaid Services (CMS) Hospital-Acquired Condition Reduction Program and the Hospital Value-Based Purchasing Program which impacts up to 2% of a hospital’s Medicare payments.^6,7 One component of PSI-90 is PSI-12, which captures perioperative VTE. PSI-12 events are identified using software that screens medical records based on International Classification of Diseases (ICD)-9/10 codes for thrombosis and procedure codes at time of discharge.⁸

Over the last several years, there has been concern regarding the validity of including PSI-12 in pay-for-performance metrics. Common areas for concern include PSI-12’s accuracy in detecting true postoperative VTE⁹ in addition to surveillance bias.^10,11 However, some note that PSI-12 is useful when applied with its original intent: as a screening tool for hospitals to identify specific areas to implement improvements.^9,12The aim of our study was to review all PSI-12 events at our institution to evaluate the accuracy of PSI-12 and identify areas for improvement to prevent VTE events in surgical patients. While several other studies have looked at the positive predictive values, accuracy, or surveillance bias of PSI-12, to the best of our knowledge, few, if any, previous studies have reported PSI-12 events in relation to their timing, type of prophylaxis used, and mitigating factors to identify areas for quality improvement.

METHODS

PSI-12 events were identified between June 2015 and June 2017 using AHRQ software (version 5). Cases were also identified through Vizient and reviewed to ensure congruence between the methods. Patients’ electronic medical records were reviewed for patient demographics, type of VTE event, platelet count at VTE diagnosis, procedure type, and both the timing and type of VTE prophylaxis. Summary statistics were calculated.

 

We considered perioperative VTE pharmacologic prophylaxis appropriate if started within 24 hours of a low bleeding risk procedure or 72 hours of a high-bleeding risk procedure.¹³ Mechanical prophylaxis was considered appropriate if pharmacologic prophylaxis was not used because of procedure risk, thrombocytopenia, or active bleeding. The medication administration record was reviewed to determine if prophylaxis was ordered, given, and/or refused.

RESULTS

During the two-year period, 18,084 surgeries were performed, and 161 cases of VTE events were identified. A detailed chart review and correction of documentation led to the exclusion of seven cases (4%) because the VTE event occurred prior to admission (n = 5) or were incidental findings that did not meet the Uniform Hospital Discharge Data Set definition for reporting (n = 2). In total, 154 (0.9% of all surgeries) cases were considered PSI-12 events. Pulmonary embolism (PE) occurred in most cases (n = 97, 62.9%), followed by deep vein thrombosis (DVT) (n = 37, 24.0%). Twenty cases (12.9%) experienced concurrent PE and DVT. Within the PE group, 16 cases (14%) were subsegmental PE only. Eight patients (14% of DVT cases) had only a distal DVT. The mean age of patients was 56 years (+/− 16 years), and the majority (59%) were male. The clinical specialties with the most events included neurosurgery (21%), orthopedics (14%), general surgery (13%), and trauma (11%). Fourteen patients (9%) died during the hospitalization, and of these, six (43%) had either sudden death or death attributed to PE (Table 1).

Cases were also reviewed for the type of VTE prophylactic strategy administered at the time of the event. The top three prophylactic strategies were subcutaneous unfractionated heparin (61%), mechanical prophylaxis only (51%), and enoxaparin (31%). Nine cases of VTE occurred during therapeutic anticoagulation (6%; Table 1).

We also evaluated the timing of VTE in relation to hospitalization and procedure. Overall, the median length of hospital stay was 21 days (range: 11-39 days). VTE occurred early in the hospitalization; 21% of cases of VTE occurred within three days of admission, and 43.5% occurred within seven days of admission (Figure 1). With regard to VTE timing in relation to the procedure, 4.5% of cases of VTE occurred prior to the procedure, 33% occurred within three days of the procedure, and 53% occurred within seven days (Figure 2).

Absence of guideline-appropriate VTE prophylaxis was identified in only nine (6%) cases: seven patients had delayed initiation of pharmacologic prophylaxis, and two had pharmacologic prophylaxis held for unknown reasons. When accounting for pharmacologic prophylaxis missed based on patient refusal (n = 10 patients), the number of patients without guideline- appropriate VTE prophylaxis increased to 17 cases (11%), as two of the cases with patient refusal were found to have other quality issues present. Pharmacologic prophylaxis was given to 125 patients during their hospitalization. A median of 8% of ordered doses was refused, and an additional 8% of doses were held for a procedure (Table 2). We evaluated other factors that could have influenced the rate or type of VTE prophylactic strategies toward the use of mechanical prophylaxis, including thrombocytopenia and trauma. Although 11% of cases were treated primarily by the trauma teams (Table 1), a trauma-related procedure accounted for 29% of PSI-12 cases. Thrombocytopenia (platelets of less than 100,000) occurred in 53 cases (34%), with 27 patients (18%) having a platelet count of less than 50,000.

 

DISCUSSION

Utilizing AHRQ version 5 software for PSI-12, our institution identified 154 cases of perioperative VTE. Most of these cases were a pulmonary embolism, occurred within a week of admission, and were associated with surgical specialties that portend a higher risk of VTE. Very few of these cases were deficient in guideline-directed VTE prophylaxis, and several cases had associated factors such as trauma and thrombocytopenia that may have appropriately influenced the decision to use mechanical only prophylaxis. Sixteen percent of pharmacologic VTE prophylactic doses were refused or held for a procedure, a known but rarely quantified influence on rates of pharmacologic prophylaxis. The use of patient-level data and adjudication of the clinical decision that affected the administration of VTE prophylaxis is a major strength of this work. In all, our data raise several questions about the accuracy of PSI-12 in identifying preventable postoperative VTE, especially as it is utilized as a marker for pay-for-performance measures in addition to identifying further areas for research and improvement.

Our results align with previous studies that suggest that PSI-12 is an inaccurate measure of performance quality. A study by Bilimoria et al. in 2013 noted that surveillance biases associated with PSI-12, showing that hospitals with higher compliance to appropriate VTE prophylaxis paradoxically had worse outcomes.¹⁰ Furthermore, Blay et al. recently published a study evaluating PSI-90 scores for hospitals with and without the VTE measure included. Their results indicated that larger hospitals (teaching hospitals, level I trauma centers, etc) caring for sicker patients were noted to improve by 8%-25% when the VTE measure was removed.¹¹ Similarly, our data indicate that the PSI-12 may not be an accurate measure of quality performance, as only 6% of cases were noted to be deficient in appropriate guideline-directed prophylaxis. Even when accounting for the refusal of doses of pharmacologic prophylaxis, this figure only increased to 11%.

Procedures included in the current PSI-12 algorithm also vary in the risk they pose to developing VTE. For example, the current version of PSI-12 includes surgical Medicare Severity Diagnosis Related Groups (MS-DRGs) for procedures with a high risk of VTE such as orthopedic, abdominal, or thoracic procedures.^14,15, However, it is worth noting that procedures such as tracheostomy and ocular surgery are also included.¹⁴ The variation in risk for development of VTE is reflected in the Caprini score, a perioperative risk stratification tool. These latter procedures only contribute one point, whereas trauma would add five points each to the total score, and make the patient a high VTE risk from the procedure alone.¹⁶ With regard to PSI-12, in theory, scores could vary significantly between centers even if the quality of care is the same, based on the volume and risk of procedures performed.

While most of our cases of VTE occurred within higher risk surgical subspecialties, 15 (10%) of our cases were within the clinical specialties of interventional radiology, cardiology, gastroenterology, and bone marrow transplant. One procedure of notable conflict includes bronchoalveolar lavage (BAL), which is included as an operating room procedure per the current version of PSI-12 software,¹⁷ but is no longer recognized by CMS as a surgical MS-DRG for reimbursement. The PSI-12 observed-to-expected rates take the DRG and comorbidity codes into consideration, but which DRG is selected does not always reflect the procedure type or the risk of VTE associated with the procedure.

Regarding the type of VTE event, our data revealed that PE was the predominant event, accounting for 62% of cases. This is different compared with other studies, such as the population-based studies in which PE and DVT account for approximately 40%-42% of events, respectively, and approximately 15% with concurrent PE and DVT. ^9,18 Borzecki et al. however, noted similar rates to our study, with 55% as PE only, 38% as DVT only, and 8% had both PE and DVT. This study also found a positive correlation between PSI-12 rates and VTE imaging rates, such that if more CT scans were completed, more PEs could be found.¹⁹ Our data identified 16 of the 97 cases of PE were subsegmental PE, a subset of VTE whose clinical significance has been questioned.²⁰ Excluding these cases brings the percentage closer to 52%. Also, screening ultrasounds are not performed at our institution. Asymptomatic or minimally symptomatic DVTs could be missed.

Further, this study also highlights that the reliability of PSI-12 is dependent on accurate documentation and coding. This is most evident when reviewing studies that evaluated the positive predictive value (PPV) of this measure.^9,12,21 A study of 28 Veteran’s Affairs hospitals from 2003 to 2007 used the AHRQ version 3 software to assess the PPV of PSI-12. Out of the 112 cases flagged by the AHRQ PSI software, only 48 were true events of postoperative DVT, yielding a PPV of only 43%. False-positive results were primarily patients with VTE present on admission and cases that were diagnosed after admission but prior to the index procedure. They also noted that coding inaccuracies were present in 38% of cases.⁹ Similarly, Henderson et al. conducted a retrospective review of 112 postsurgical discharges noting a PPV of 54%, with most false-positives resulting from superficial clots identified by PSI-12 related to coding ambiguity.¹² Our data similarly showed false-positive results as seven cases were excluded based on chart review and documentation correction, and 4.5% were preprocedural. However, there is some discordance with previous studies, as our data yields a PPV of 91% for VTE when accounting for preprocedural events as well as those excluded based on chart review. One explanation is an improvement in documentation strategies and coding, as our institution has adopted strategies to review cases and clarify documentation prior to billing to improve accuracy, and inclusion of a checkbox to indicate that a diagnosis was present on admission. Another possibility is improved accuracy with ICD-10 coding, as shown in a paper by Quan et al. that found a 79% PPV of PSI-12, which improved to 89% when cases present on admission were excluded.²¹ Lastly, we used newer versions of the AHRQ software, which could have improved the accuracy of detection. Despite these improvements in PSI-12 identifying true cases of postoperative VTE, as our data show, there is still much to be desired in terms of identifying issues with the quality of care and inclusion of PSI-12 in pay-for-performance.

Our study has several limitations. As a retrospective review, a major limiting factor is that data obtained are subject to the accuracy of documentation within the provider and nursing notes. As such, we focused on broad topics such as VTE events, the type of VTE event, and timing of the event in relation to admission and procedure. We actively review cases at discharge prior to billing to correct documentation if required. However, a prolonged hospital stay could lead to a review of the case weeks to months later. A real-time alert for a VTE event in a patient with surgery could improve documentation.

Further, although mechanical only prophylaxis was present on chart review, it is impossible to know both the rate of compliance with this method and whether it contributed to some events. Lastly, our study evaluated primarily the PPV of PSI-12. We did not directly evaluate the negative predictive value of PSI-12, which could mean there are cases of preventable VTE that are missed entirely.

Despite these limitations, our goals of evaluating the validity of PSI-12 and identifying areas for improved measurement and techniques for DVT prophylaxis were met. As previously discussed, our data suggest that PSI-12 has several limitations in identifying quality of care issues with the prevention of DVT. For example, PSI-12 included many cases in which pharmacologic prophylaxis was appropriately not given. Approximately 50% of cases occurred in patients who were thrombocytopenic (platelets < 100,000), and 29% of events occurred in trauma patients. In addition, 33% of cases identified were within three days of the procedure, which, in cases of high-bleeding risk procedures, is an appropriate time to refrain from pharmacologic anticoagulation.¹³ In cases such as these, it may be worth evaluating alternative forms of mechanical only prophylaxis, or other strategies to mitigate this risk.

This study also highlights areas for further research. Many of the VTE events in this study occurred while patients were treated with appropriate prophylaxis, as other studies have also shown.^22,23 Gangireddy et al. looked at demographic and clinical information surrounding postoperative VTE and found several other risk factors that incur a higher risk of VTE, including steroid use, infections, and myocardial infarction.²⁴ Perhaps future studies could target these groups as potential points for increased prophylactic strategies. As noted by Lau et al. appropriate VTE prophylaxis involves risk stratification, ordering the appropriate prophylaxis by clinicians, patient acceptance of prescribed therapy, and nursing administration of prescribed therapy.²⁵ As exemplified by our data, despite appropriate prophylaxis being ordered, eight events were associated solely with the refusal of pharmacologic prophylaxis. An ideal VTE metric would identify patients with a VTE who had a defective VTE prophylactic process,²⁵ but the cost associated with manual data abstraction may limit the inclusion of a process metric into pay-for-performance. Additional research also needs to identify whether modifications to PSI-12 can improve its accuracy and predictive value, such as the exclusion of VTE prior to a procedure, modifications to what counts as a procedure, or utilizing markers to assess adherence to VTE prophylactic rates. These points are especially important to consider if PSI-12 is to remain as one of the key factors in pay-for-performance outcome measures. Lastly, understanding the effectiveness and patient adherence to oral VTE prophylactic regimens could also decrease the rates of refusal of VTE prophylaxis.

Overall, VTE remains a large contributor to morbidity and mortality among hospitalized surgical patients. While there is utility in PSI-12 as a screening tool to identify these events and potential areas for improved processes to decrease VTE, its usefulness for detection of true quality issues and its utilization in pay-for-performance is questionable. The universally high rates of VTE prophylaxis question the need for a VTE prophylactic metric. However, if these metrics are going to continue to be used in determining payments, modifications to the current processes and algorithms are needed to improve accuracy in identifying issues in quality of care.