User login
Satisfaction With Department of Veterans Affairs Prosthetics and Support Services as Reported by Women and Men Veterans
Satisfaction With Department of Veterans Affairs Prosthetics and Support Services as Reported by Women and Men Veterans
Limb loss is a significant and growing concern in the United States. Nearly 2 million Americans are living with limb loss, and up to 185,000 people undergo amputations annually.1-4 Of these patients, about 35% are women.5 The Veterans Health Administration (VHA) provides about 10% of US amputations.6-8 Between 2015 and 2019, the number of prosthetic devices provided to female veterans increased from 3.3 million to 4.6 million.5,9,10
Previous research identified disparities in prosthetic care between men and women, both within and outside the VHA. These disparities include slower prosthesis prescription and receipt among women, in addition to differences in self-reported mobility, satisfaction, rates of prosthesis rejection, and challenges related to prosthesis appearance and fit.5,10,11 Recent studies suggest women tend to have worse outcomes following amputation, and are underrepresented in amputation research.12,13 However, these disparities are poorly described in a large, national sample. Because women represent a growing portion of patients with limb loss in the VHA, understanding their needs is critical.14
The Johnny Isakson and David P. Roe, MD Veterans Health Care and Benefits Improvement Act of 2020 was enacted, in part, to improve the care provided to women veterans.15 The law required the VHA to conduct a survey of ≥ 50,000 veterans to assess the satisfaction of women veterans with prostheses provided by the VHA. To comply with this legislation and understand how women veterans rate their prostheses and related care in the VHA, the US Department of Veterans Affairs (VA) Center for Collaborative Evaluation (VACE) conducted a large national survey of veterans with limb loss that oversampled women veterans. This article describes the survey results, including characteristics of female veterans with limb loss receiving care from the VHA, assesses their satisfaction with prostheses and prosthetic care, and highlights where their responses differ from those of male veterans.
Methods
We conducted a cross-sectional, mixedmode survey of eligible amputees in the VHA Support Service Capital Assets Amputee Data Cube. We identified a cohort of veterans with any major amputation (above the ankle or wrist) or partial hand or foot amputation who received VHA care between October 1, 2019, and September 30, 2020. The final cohort yielded 46,646 potentially eligible veterans. Thirty-three had invalid contact information, leaving 46,613 veterans who were asked to participate, including 1356 women.
Survey
We created a survey instrument de novo that included questions from validated instruments, including the Trinity Amputation Prosthesis and Experience Scales to assess prosthetic device satisfaction, the Prosthesis Evaluation Questionnaire to assess quality of life (QOL) satisfaction, and the Orthotics Prosthetics Users Survey to assess prosthesis-related care satisfaction. 16-18 Additional questions were incorporated from a survey of veterans with upper limb amputation to assess the importance of cosmetic considerations related to the prosthesis and comfort with prosthesis use in intimate relationships.19 Questions were also included to assess amputation type, year of amputation, if a prosthesis was currently used, reasons for ceasing use of a prosthesis, reasons for never using a prosthesis, the types of prostheses used, intensity of prosthesis use, satisfaction with time required to receive a prosthetic limb, and if the prosthesis reflected the veteran’s selfidentified gender. Veterans were asked to answer questions based on their most recent amputation.
We tested the survey using cognitive interviews with 6 veterans to refine the survey and better understand how veterans interpreted the questions. Pilot testers completed the survey and participated in individual interviews with experienced interviewers (CL and RRK) to describe how they selected their responses.20 This feedback was used to refine the survey. The online survey was programmed using Qualtrics Software and manually translated into Spanish.
Given the multimodal design, surveys were distributed by email, text message, and US Postal Service (USPS). Surveys were emailed to all veterans for whom a valid email address was available. If emails were undeliverable, veterans were contacted via text message or the USPS. Surveys were distributed by text message to all veterans without an email address but with a cellphone number. We were unable to consistently identify invalid numbers among all text message recipients. Invitations with a survey URL and QR code were sent via USPS to veterans who had no valid email address or cellphone number. Targeted efforts were made to increase the response rate for women. A random sample of 200 women who had not completed the survey 2 weeks prior to the closing date (15% of women in sample) was selected to receive personal phone calls. Another random sample of 400 women was selected to receive personalized outreach emails. The survey data were confidential, and responses could not be traced to identifying information.
Data Analyses
We conducted a descriptive analysis, including percentages and means for responses to variables focused on describing amputation characteristics, prosthesis characteristics, and QOL. All data, including missing values, were used to document the percentage of respondents for each question. Removing missing data from the denominator when calculating percentages could introduce bias to the analysis because we cannot be certain data are missing at random. Missing variables were removed to avoid underinflation of mean scores.
We compared responses across 2 groups: individuals who self-identified as men and individuals who self-identified as women. For each question, we assessed whether each of these groups differed significantly from the remaining sample. For example, we examined whether the percentage of men who answered affirmatively to a question was significantly higher or lower than that of individuals not identifying as male, and whether the percentage of women who answered affirmatively was significantly higher or lower than that of individuals not identifying as female. We utilized x2 tests to determine significant differences for percentage calculations and t tests to determine significant differences in means across gender.
Since conducting multiple comparisons within a dataset may result in inflating statistical significance (type 1 errors), we used a more conservative estimate of statistical significance (α = 0.01) and high significance (α = 0.001). This study was deemed quality improvement by the VHA Rehabilitation and Prosthetic Services (12RPS) and acknowledged by the VA Research Office at Eastern Colorado Health Care System and was not subject to institutional review board review.
Results
Surveys were distributed to 46,613 veterans and were completed by 4981 respondents for a 10.7% overall response rate. Survey respondents were generally similar to the eligible population invited to participate, but the proportion of women who completed the survey was higher than the proportion of women eligible to participate (2.0% of eligible population vs 16.7% of respondents), likely due to specific efforts to target women. Survey respondents were slightly younger than the general population (67.3 years vs 68.7 years), less likely to be male (97.1% vs 83.3%), showed similar representation of Operation Enduring Freedom/Operation Iraqi Freedom (OEF/OIF) veterans (4.4% vs 4.1%), and were less likely to have diabetes (58.0% vs 52.7% had diabetes) (Table 1).
The mean age of male respondents was 67.3 years, while the mean age of female respondents was 58.3 years. The majority of respondents were male (83.3%) and White (77.2%). Female respondents were less likely to have diabetes (35.4% of women vs 53.5% of men) and less likely to report that their most recent amputation resulted from diabetes (10.1% of women vs 22.2% of men). Women respondents were more likely to report an amputation due to other causes, such as adverse results of surgery, neurologic disease, suicide attempt, blood clots, tumors, rheumatoid arthritis, and revisions of previous amputations. Most women respondents did not serve during the OEF or OIF eras. The most common amputation site for women respondents was lower limb, either below the knee and above the ankle or above the knee.
Most participants use an everyday prosthesis, but women were more likely to report using a sports-specific prosthesis (Table 2). Overall, most respondents report using a prosthesis (87.7%); however, women were more likely to report not using a prosthesis (19.4% of women vs 11.1% of men; P ≤ .01). Additionally, a lower proportion of women report using a prosthesis for < 12 hours per day (30.6% of women vs 46.4% of men; P ≤ .01) or using a prosthesis every day (54.8% of women vs 74.6% of men; P ≤ .001).
In the overall sample, the mean satisfaction score with a prosthesis was 2.7 on a 5-point scale, and women had slightly lower overall satisfaction scores (2.6 for women vs 2.7 for men; P ≤ .001) (Table 3). Women also had lower satisfaction scores related to appearance, usefulness, reliability, and comfort. Women were more likely to indicate that it was very important to be able to wear jewelry and accessories (20.2% of women vs 11.6% of men; P ≤ .01), while men were less likely to indicate that it was somewhat or very important that the prosthesis not restrict clothing or shoes (95.2% of women vs 82.9% of men; P ≤ .001). Men were more likely than women to report being comfortable or very comfortable using their prosthesis in intimate contact: 40.5% vs 29.0%, respectively (P ≤ .001).
Overall, participants reported high satisfaction with appointment times, wait times, courteous treatment, opportunities to express concerns, and staff responsiveness. Men were slightly more likely than women to be satisfied with training (P ≤ 0.001) and problem discussion (P ≤ 0.01) (Table 4). There were no statistically significant differences in satisfaction or QOL ratings between women and men. The overall sample rated both QOL and satisfaction with QOL 6.7 on a 10-point scale.
Discussion
The goal of this study was to characterize the experience of veterans with limb loss receiving care in the VHA and assess their satisfaction with prostheses and prosthetic care. We received responses from nearly 5000 veterans, 158 of whom were women. Women veteran respondents were slightly younger and less likely to have an amputation due to diabetes. We did not observe significant differences in amputation level between men and women but women were less likely to use a prosthesis, reported lower intensity of prosthesis use, and were less satisfied with certain aspects of their prostheses. Women may also be less satisfied with prosthesis training and problem discussion. However, we found no differences in QOL ratings between men and women.
Findings indicating women were more likely to report not using a prosthesis and that a lower proportion of women report using a prosthesis for > 12 hours a day or every day are consistent with previous research. 21,22 Interestingly, women were more likely to report using a sports-specific prosthesis. This is notable because prior research suggests that individuals with amputations may avoid participating in sports and exercise, and a lack of access to sports-specific prostheses may inhibit physical activity.23,24 Women in this sample were slightly less satisfied with their prostheses overall and reported lower satisfaction scores regarding appearance, usefulness, reliability, and comfort, consistent with previous findings.25
A lower percentage of women in this sample reported being comfortable or very comfortable using their prosthesis during intimate contact. Previous research on prosthesis satisfaction suggests individuals who rate prosthesis satisfaction lower also report lower body image across genders. 26 While women in this sample did not rate their prosthesis satisfaction lower than men, they did report lower intensity of prosthesis use, suggesting potential issues with their prostheses this survey did not evaluate. Women indicated the importance of prostheses not restricting jewelry, accessories, clothing, or shoes. These results have significant clinical and social implications. A recent qualitative study emphasizes that women veterans feel prostheses are primarily designed for men and may not work well with their physiological needs.9 Research focused on limbs better suited to women’s bodies could result in better fitting sockets, lightweight limbs, or less bulky designs. Additional research has also explored the difficulties in accommodating a range of footwear for patients with lower limb amputation. One study found that varying footwear heights affect the function of adjustable prosthetic feet in ways that may not be optimal.27
Ratings of satisfaction with prosthesisrelated services between men and women in this sample are consistent with a recent study showing that women veterans do not have significant differences in satisfaction with prosthesis-related services.28 However, this study focused specifically on lower limb amputations, while the respondents of this study include those with both upper and lower limb amputations. Importantly, our findings that women are less likely to be satisfied with prosthesis training and problem discussions support recent qualitative findings in which women expressed a desire to work with prosthetists who listen to them, take their concerns seriously, and seek solutions that fit their needs. We did not observe a difference in QOL ratings between men and women in the sample despite lower satisfaction among women with some elements of prosthesis-related services. Previous research suggests many factors impact QOL after amputation, most notably time since amputation.16,29
Limitations
This survey was deployed in a short timeline that did not allow for careful sample selection or implementing strategies to increase response rate. Additionally, the study was conducted among veterans receiving care in the VHA, and findings may not be generalizable to limb loss in other settings. Finally, the discrepancy in number of respondents who identified as men vs women made it difficult to compare differences between the 2 groups.
Conclusions
This is the largest sample of survey respondents of veterans with limb loss to date. While the findings suggest veterans are generally satisfied with prosthetic-related services overall, they also highlight several areas for improvement with services or prostheses. Given that most veterans with limb loss are men, there is a significant discrepancy between the number of women and men respondents. Additional studies with more comparable numbers of men and women have found similar ratings of satisfaction with prostheses and services.28 Further research specifically focused on improving the experiences of women should focus on better characterizing their experiences and identifying how they differ from those of male veterans. For example, understanding how to engage female veterans with limb loss in prosthesis training and problem discussions may improve their experience with their care teams and improve their use of prostheses. Understanding experiences and needs that are specific to women could lead to the development of processes, resources, or devices that are tailored to the unique requirements of women with limb loss.
- Ziegler-Graham K, MacKenzie EJ, Ephraim PL, Travison TG, Brookmeyer R. Estimating the prevalence of limb loss in the United States: 2005 to 2050. Arch Phys Med Rehabil. 2008;89(3):422-429. doi:10.1016/j.apmr.2007.11.005
- Dillingham TR, Pezzin LE, MacKenzie EJ. Limb amputation and limb deficiency: epidemiology and recent trends in the united states. South Med J. 2002;95(8):875-883. doi:10.1097/00007611-200208000-00018
- Dillingham TR, Pezzin LE, Shore AD. Reamputation, mortality, and health care costs among persons with dysvascular lower-limb amputations. Arch Phys Med Rehabil. 2005;86(3):480-486. doi:10.1016/j.apmr.2004.06.072
- Centers for Disease Control and Prevention. Ambulatory and inpatient procedures in the United States. Accessed September 30, 2024. https://www.cdc.gov/nchs/pressroom/98facts/ambulat.htm
- Ljung J, Iacangelo A. Identifying and acknowledging a sex gap in lower-limb prosthetics. JPO. 2024;36(1):e18-e24. doi:10.1097/JPO.0000000000000470
- Feinglass J, Brown JL, LoSasso A, et al. Rates of lower-extremity amputation and arterial reconstruction in the united states, 1979 to 1996. Am J Public Health. 1999;89(8):1222- 1227. doi:10.2105/ajph.89.8.1222
- Mayfield JA, Reiber GE, Maynard C, Czerniecki JM, Caps MT, Sangeorzan BJ. Trends in lower limb amputation in the Veterans Health Administration, 1989-1998. J Rehabil Res Dev. 2000;37(1):23-30.
- Feinglass J, Pearce WH, Martin GJ, et al. Postoperative and late survival outcomes after major amputation: findings from the department of veterans affairs national surgical quality improvement program. Surgery. 2001;130(1):21-29. doi:10.1067/msy.2001.115359
- Lehavot K, Young JP, Thomas RM, et al. Voices of women veterans with lower limb prostheses: a qualitative study. J Gen Intern Med. 2022;37(3):799-805. doi:10.1007/s11606-022-07572-8
- US Government Accountability Office. COVID-19: Opportunities to improve federal response. GAO-21-60. Published November 12, 2020. Accessed September 30, 2024. https://www.gao.gov/products/gao-21-60
- Littman AJ, Peterson AC, Korpak A, et al. Differences in prosthetic prescription between men and women veterans after transtibial or transfemoral lowerextremity amputation: a longitudinal cohort study. Arch Phys Med Rehabil. 2023;104(8)1274-1281. doi:10.1016/j.amjsurg.2023.02.011
- Cimino SR, Vijayakumar A, MacKay C, Mayo AL, Hitzig SL, Guilcher SJT. Sex and gender differences in quality of life and related domains for individuals with adult acquired lower-limb amputation: a scoping review. Disabil Rehabil. 2022 Oct 23;44(22):6899-6925. doi:10.1080/09638288.2021.1974106
- DadeMatthews OO, Roper JA, Vazquez A, Shannon DM, Sefton JM. Prosthetic device and service satisfaction, quality of life, and functional performance in lower limb prosthesis clients. Prosthet Orthot Int. 2024;48(4):422-430. doi:10.1097/PXR.0000000000000285
- Hamilton AB, Schwarz EB, Thomas HN, Goldstein KM. Moving women veterans’ health research forward: a special supplement. J Gen Intern Med. 2022;37(Suppl3):665– 667. doi:10.1007/s11606-022-07606-1
- US Congress. Public Law 116-315: An Act to Improve the Lives of Veterans, S 5108 (2) (F). 116th Congress; 2021. Accessed September 30, 2024. https://www.congress.gov/116/plaws/publ315/PLAW-116publ315.pdf
- Gallagher P, MacLachlan M. The Trinity amputation and prosthesis experience scales and quality of life in people with lower-limb amputation. Arch Phys Med Rehabil. 2004;85(5):730-736. doi:10.1016/j.apmr.2003.07.009
- Legro MW, Reiber GD, Smith DG, del Aguila M, Larsen J, Boone D. Prosthesis evaluation questionnaire for persons with lower limb amputations: assessing prosthesis-related quality of life. Arch Phys Med Rehabil. 1998;79(8):931-938. doi:10.1016/s0003-9993(98)90090-9
- Legro MW, Reiber GD, Smith DG, del Aguila M, Larsen J, Boone D. Prosthesis evaluation questionnaire for persons with lower limb amputations: assessing prosthesis-related quality of life. Arch Phys Med Rehabil. 1998;79(8):931-938. doi:10.1016/s0003-9993(98)90090-9
- Heinemann AW, Bode RK, O’Reilly C. Development and measurement properties of the orthotics and prosthetics users’ survey (OPUS): a comprehensive set of clinical outcome instruments. Prosthet Orthot Int. 2003;27(3):191-206. doi:10.1080/03093640308726682
- Resnik LJ, Borgia ML, Clark MA. A national survey of prosthesis use in veterans with major upper limb amputation: comparisons by gender. PM R. 2020;12(11):1086-1098. doi:10.1002/pmrj.12351
- Collins D. Pretesting survey instruments: an overview of cognitive methods. Qual Life Res. 2003;12(3):229-238. doi:10.1023/a:1023254226592
- Østlie K, Lesjø IM, Franklin RJ, Garfelt B, Skjeldal OH, Magnus P. Prosthesis rejection in acquired major upper-limb amputees: a population-based survey. Disabil Rehabil Assist Technol. 2012;7(4):294-303. doi:10.3109/17483107.2011.635405
- Pezzin LE, Dillingham TR, MacKenzie EJ, Ephraim P, Rossbach P. Use and satisfaction with prosthetic limb devices and related services. Arch Phys Med Rehabil. 2004;85(5):723-729. doi:10.1016/j.apmr.2003.06.002
- Deans S, Burns D, McGarry A, Murray K, Mutrie N. Motivations and barriers to prosthesis users participation in physical activity, exercise and sport: a review of the literature. Prosthet Orthot Int. 2012;36(3):260-269. doi:10.1177/0309364612437905
- McDonald CL, Kahn A, Hafner BJ, Morgan SJ. Prevalence of secondary prosthesis use in lower limb prosthesis users. Disabil Rehabil. 2023;46(5):1016-1022. doi:10.1080/09638288.2023.2182919
- Baars EC, Schrier E, Dijkstra PU, Geertzen JHB. Prosthesis satisfaction in lower limb amputees: a systematic review of associated factors and questionnaires. Medicine (Baltimore). 2018;97(39):e12296. doi:10.1097/MD.0000000000012296
- Murray CD, Fox J. Body image and prosthesis satisfaction in the lower limb amputee. Disabil Rehabil. 2002;24(17):925–931. doi:10.1080/09638280210150014
- Major MJ, Quinlan J, Hansen AH, Esposito ER. Effects of women’s footwear on the mechanical function of heel-height accommodating prosthetic feet. PLoS One. 2022;17(1). doi:10.1371/journal.pone.0262910.
- Kuo PB, Lehavot K, Thomas RM, et al. Gender differences in prosthesis-related outcomes among veterans: results of a national survey of U.S. veterans. PM R. 2024;16(3):239- 249. doi:10.1002/pmrj.13028
- Asano M, Rushton P, Miller WC, Deathe BA. Predictors of quality of life among individuals who have a lower limb amputation. Prosthet Orthot Int. 2008;32(2):231-243. doi:10.1080/03093640802024955
Limb loss is a significant and growing concern in the United States. Nearly 2 million Americans are living with limb loss, and up to 185,000 people undergo amputations annually.1-4 Of these patients, about 35% are women.5 The Veterans Health Administration (VHA) provides about 10% of US amputations.6-8 Between 2015 and 2019, the number of prosthetic devices provided to female veterans increased from 3.3 million to 4.6 million.5,9,10
Previous research identified disparities in prosthetic care between men and women, both within and outside the VHA. These disparities include slower prosthesis prescription and receipt among women, in addition to differences in self-reported mobility, satisfaction, rates of prosthesis rejection, and challenges related to prosthesis appearance and fit.5,10,11 Recent studies suggest women tend to have worse outcomes following amputation, and are underrepresented in amputation research.12,13 However, these disparities are poorly described in a large, national sample. Because women represent a growing portion of patients with limb loss in the VHA, understanding their needs is critical.14
The Johnny Isakson and David P. Roe, MD Veterans Health Care and Benefits Improvement Act of 2020 was enacted, in part, to improve the care provided to women veterans.15 The law required the VHA to conduct a survey of ≥ 50,000 veterans to assess the satisfaction of women veterans with prostheses provided by the VHA. To comply with this legislation and understand how women veterans rate their prostheses and related care in the VHA, the US Department of Veterans Affairs (VA) Center for Collaborative Evaluation (VACE) conducted a large national survey of veterans with limb loss that oversampled women veterans. This article describes the survey results, including characteristics of female veterans with limb loss receiving care from the VHA, assesses their satisfaction with prostheses and prosthetic care, and highlights where their responses differ from those of male veterans.
Methods
We conducted a cross-sectional, mixedmode survey of eligible amputees in the VHA Support Service Capital Assets Amputee Data Cube. We identified a cohort of veterans with any major amputation (above the ankle or wrist) or partial hand or foot amputation who received VHA care between October 1, 2019, and September 30, 2020. The final cohort yielded 46,646 potentially eligible veterans. Thirty-three had invalid contact information, leaving 46,613 veterans who were asked to participate, including 1356 women.
Survey
We created a survey instrument de novo that included questions from validated instruments, including the Trinity Amputation Prosthesis and Experience Scales to assess prosthetic device satisfaction, the Prosthesis Evaluation Questionnaire to assess quality of life (QOL) satisfaction, and the Orthotics Prosthetics Users Survey to assess prosthesis-related care satisfaction. 16-18 Additional questions were incorporated from a survey of veterans with upper limb amputation to assess the importance of cosmetic considerations related to the prosthesis and comfort with prosthesis use in intimate relationships.19 Questions were also included to assess amputation type, year of amputation, if a prosthesis was currently used, reasons for ceasing use of a prosthesis, reasons for never using a prosthesis, the types of prostheses used, intensity of prosthesis use, satisfaction with time required to receive a prosthetic limb, and if the prosthesis reflected the veteran’s selfidentified gender. Veterans were asked to answer questions based on their most recent amputation.
We tested the survey using cognitive interviews with 6 veterans to refine the survey and better understand how veterans interpreted the questions. Pilot testers completed the survey and participated in individual interviews with experienced interviewers (CL and RRK) to describe how they selected their responses.20 This feedback was used to refine the survey. The online survey was programmed using Qualtrics Software and manually translated into Spanish.
Given the multimodal design, surveys were distributed by email, text message, and US Postal Service (USPS). Surveys were emailed to all veterans for whom a valid email address was available. If emails were undeliverable, veterans were contacted via text message or the USPS. Surveys were distributed by text message to all veterans without an email address but with a cellphone number. We were unable to consistently identify invalid numbers among all text message recipients. Invitations with a survey URL and QR code were sent via USPS to veterans who had no valid email address or cellphone number. Targeted efforts were made to increase the response rate for women. A random sample of 200 women who had not completed the survey 2 weeks prior to the closing date (15% of women in sample) was selected to receive personal phone calls. Another random sample of 400 women was selected to receive personalized outreach emails. The survey data were confidential, and responses could not be traced to identifying information.
Data Analyses
We conducted a descriptive analysis, including percentages and means for responses to variables focused on describing amputation characteristics, prosthesis characteristics, and QOL. All data, including missing values, were used to document the percentage of respondents for each question. Removing missing data from the denominator when calculating percentages could introduce bias to the analysis because we cannot be certain data are missing at random. Missing variables were removed to avoid underinflation of mean scores.
We compared responses across 2 groups: individuals who self-identified as men and individuals who self-identified as women. For each question, we assessed whether each of these groups differed significantly from the remaining sample. For example, we examined whether the percentage of men who answered affirmatively to a question was significantly higher or lower than that of individuals not identifying as male, and whether the percentage of women who answered affirmatively was significantly higher or lower than that of individuals not identifying as female. We utilized x2 tests to determine significant differences for percentage calculations and t tests to determine significant differences in means across gender.
Since conducting multiple comparisons within a dataset may result in inflating statistical significance (type 1 errors), we used a more conservative estimate of statistical significance (α = 0.01) and high significance (α = 0.001). This study was deemed quality improvement by the VHA Rehabilitation and Prosthetic Services (12RPS) and acknowledged by the VA Research Office at Eastern Colorado Health Care System and was not subject to institutional review board review.
Results
Surveys were distributed to 46,613 veterans and were completed by 4981 respondents for a 10.7% overall response rate. Survey respondents were generally similar to the eligible population invited to participate, but the proportion of women who completed the survey was higher than the proportion of women eligible to participate (2.0% of eligible population vs 16.7% of respondents), likely due to specific efforts to target women. Survey respondents were slightly younger than the general population (67.3 years vs 68.7 years), less likely to be male (97.1% vs 83.3%), showed similar representation of Operation Enduring Freedom/Operation Iraqi Freedom (OEF/OIF) veterans (4.4% vs 4.1%), and were less likely to have diabetes (58.0% vs 52.7% had diabetes) (Table 1).
The mean age of male respondents was 67.3 years, while the mean age of female respondents was 58.3 years. The majority of respondents were male (83.3%) and White (77.2%). Female respondents were less likely to have diabetes (35.4% of women vs 53.5% of men) and less likely to report that their most recent amputation resulted from diabetes (10.1% of women vs 22.2% of men). Women respondents were more likely to report an amputation due to other causes, such as adverse results of surgery, neurologic disease, suicide attempt, blood clots, tumors, rheumatoid arthritis, and revisions of previous amputations. Most women respondents did not serve during the OEF or OIF eras. The most common amputation site for women respondents was lower limb, either below the knee and above the ankle or above the knee.
Most participants use an everyday prosthesis, but women were more likely to report using a sports-specific prosthesis (Table 2). Overall, most respondents report using a prosthesis (87.7%); however, women were more likely to report not using a prosthesis (19.4% of women vs 11.1% of men; P ≤ .01). Additionally, a lower proportion of women report using a prosthesis for < 12 hours per day (30.6% of women vs 46.4% of men; P ≤ .01) or using a prosthesis every day (54.8% of women vs 74.6% of men; P ≤ .001).
In the overall sample, the mean satisfaction score with a prosthesis was 2.7 on a 5-point scale, and women had slightly lower overall satisfaction scores (2.6 for women vs 2.7 for men; P ≤ .001) (Table 3). Women also had lower satisfaction scores related to appearance, usefulness, reliability, and comfort. Women were more likely to indicate that it was very important to be able to wear jewelry and accessories (20.2% of women vs 11.6% of men; P ≤ .01), while men were less likely to indicate that it was somewhat or very important that the prosthesis not restrict clothing or shoes (95.2% of women vs 82.9% of men; P ≤ .001). Men were more likely than women to report being comfortable or very comfortable using their prosthesis in intimate contact: 40.5% vs 29.0%, respectively (P ≤ .001).
Overall, participants reported high satisfaction with appointment times, wait times, courteous treatment, opportunities to express concerns, and staff responsiveness. Men were slightly more likely than women to be satisfied with training (P ≤ 0.001) and problem discussion (P ≤ 0.01) (Table 4). There were no statistically significant differences in satisfaction or QOL ratings between women and men. The overall sample rated both QOL and satisfaction with QOL 6.7 on a 10-point scale.
Discussion
The goal of this study was to characterize the experience of veterans with limb loss receiving care in the VHA and assess their satisfaction with prostheses and prosthetic care. We received responses from nearly 5000 veterans, 158 of whom were women. Women veteran respondents were slightly younger and less likely to have an amputation due to diabetes. We did not observe significant differences in amputation level between men and women but women were less likely to use a prosthesis, reported lower intensity of prosthesis use, and were less satisfied with certain aspects of their prostheses. Women may also be less satisfied with prosthesis training and problem discussion. However, we found no differences in QOL ratings between men and women.
Findings indicating women were more likely to report not using a prosthesis and that a lower proportion of women report using a prosthesis for > 12 hours a day or every day are consistent with previous research. 21,22 Interestingly, women were more likely to report using a sports-specific prosthesis. This is notable because prior research suggests that individuals with amputations may avoid participating in sports and exercise, and a lack of access to sports-specific prostheses may inhibit physical activity.23,24 Women in this sample were slightly less satisfied with their prostheses overall and reported lower satisfaction scores regarding appearance, usefulness, reliability, and comfort, consistent with previous findings.25
A lower percentage of women in this sample reported being comfortable or very comfortable using their prosthesis during intimate contact. Previous research on prosthesis satisfaction suggests individuals who rate prosthesis satisfaction lower also report lower body image across genders. 26 While women in this sample did not rate their prosthesis satisfaction lower than men, they did report lower intensity of prosthesis use, suggesting potential issues with their prostheses this survey did not evaluate. Women indicated the importance of prostheses not restricting jewelry, accessories, clothing, or shoes. These results have significant clinical and social implications. A recent qualitative study emphasizes that women veterans feel prostheses are primarily designed for men and may not work well with their physiological needs.9 Research focused on limbs better suited to women’s bodies could result in better fitting sockets, lightweight limbs, or less bulky designs. Additional research has also explored the difficulties in accommodating a range of footwear for patients with lower limb amputation. One study found that varying footwear heights affect the function of adjustable prosthetic feet in ways that may not be optimal.27
Ratings of satisfaction with prosthesisrelated services between men and women in this sample are consistent with a recent study showing that women veterans do not have significant differences in satisfaction with prosthesis-related services.28 However, this study focused specifically on lower limb amputations, while the respondents of this study include those with both upper and lower limb amputations. Importantly, our findings that women are less likely to be satisfied with prosthesis training and problem discussions support recent qualitative findings in which women expressed a desire to work with prosthetists who listen to them, take their concerns seriously, and seek solutions that fit their needs. We did not observe a difference in QOL ratings between men and women in the sample despite lower satisfaction among women with some elements of prosthesis-related services. Previous research suggests many factors impact QOL after amputation, most notably time since amputation.16,29
Limitations
This survey was deployed in a short timeline that did not allow for careful sample selection or implementing strategies to increase response rate. Additionally, the study was conducted among veterans receiving care in the VHA, and findings may not be generalizable to limb loss in other settings. Finally, the discrepancy in number of respondents who identified as men vs women made it difficult to compare differences between the 2 groups.
Conclusions
This is the largest sample of survey respondents of veterans with limb loss to date. While the findings suggest veterans are generally satisfied with prosthetic-related services overall, they also highlight several areas for improvement with services or prostheses. Given that most veterans with limb loss are men, there is a significant discrepancy between the number of women and men respondents. Additional studies with more comparable numbers of men and women have found similar ratings of satisfaction with prostheses and services.28 Further research specifically focused on improving the experiences of women should focus on better characterizing their experiences and identifying how they differ from those of male veterans. For example, understanding how to engage female veterans with limb loss in prosthesis training and problem discussions may improve their experience with their care teams and improve their use of prostheses. Understanding experiences and needs that are specific to women could lead to the development of processes, resources, or devices that are tailored to the unique requirements of women with limb loss.
Limb loss is a significant and growing concern in the United States. Nearly 2 million Americans are living with limb loss, and up to 185,000 people undergo amputations annually.1-4 Of these patients, about 35% are women.5 The Veterans Health Administration (VHA) provides about 10% of US amputations.6-8 Between 2015 and 2019, the number of prosthetic devices provided to female veterans increased from 3.3 million to 4.6 million.5,9,10
Previous research identified disparities in prosthetic care between men and women, both within and outside the VHA. These disparities include slower prosthesis prescription and receipt among women, in addition to differences in self-reported mobility, satisfaction, rates of prosthesis rejection, and challenges related to prosthesis appearance and fit.5,10,11 Recent studies suggest women tend to have worse outcomes following amputation, and are underrepresented in amputation research.12,13 However, these disparities are poorly described in a large, national sample. Because women represent a growing portion of patients with limb loss in the VHA, understanding their needs is critical.14
The Johnny Isakson and David P. Roe, MD Veterans Health Care and Benefits Improvement Act of 2020 was enacted, in part, to improve the care provided to women veterans.15 The law required the VHA to conduct a survey of ≥ 50,000 veterans to assess the satisfaction of women veterans with prostheses provided by the VHA. To comply with this legislation and understand how women veterans rate their prostheses and related care in the VHA, the US Department of Veterans Affairs (VA) Center for Collaborative Evaluation (VACE) conducted a large national survey of veterans with limb loss that oversampled women veterans. This article describes the survey results, including characteristics of female veterans with limb loss receiving care from the VHA, assesses their satisfaction with prostheses and prosthetic care, and highlights where their responses differ from those of male veterans.
Methods
We conducted a cross-sectional, mixedmode survey of eligible amputees in the VHA Support Service Capital Assets Amputee Data Cube. We identified a cohort of veterans with any major amputation (above the ankle or wrist) or partial hand or foot amputation who received VHA care between October 1, 2019, and September 30, 2020. The final cohort yielded 46,646 potentially eligible veterans. Thirty-three had invalid contact information, leaving 46,613 veterans who were asked to participate, including 1356 women.
Survey
We created a survey instrument de novo that included questions from validated instruments, including the Trinity Amputation Prosthesis and Experience Scales to assess prosthetic device satisfaction, the Prosthesis Evaluation Questionnaire to assess quality of life (QOL) satisfaction, and the Orthotics Prosthetics Users Survey to assess prosthesis-related care satisfaction. 16-18 Additional questions were incorporated from a survey of veterans with upper limb amputation to assess the importance of cosmetic considerations related to the prosthesis and comfort with prosthesis use in intimate relationships.19 Questions were also included to assess amputation type, year of amputation, if a prosthesis was currently used, reasons for ceasing use of a prosthesis, reasons for never using a prosthesis, the types of prostheses used, intensity of prosthesis use, satisfaction with time required to receive a prosthetic limb, and if the prosthesis reflected the veteran’s selfidentified gender. Veterans were asked to answer questions based on their most recent amputation.
We tested the survey using cognitive interviews with 6 veterans to refine the survey and better understand how veterans interpreted the questions. Pilot testers completed the survey and participated in individual interviews with experienced interviewers (CL and RRK) to describe how they selected their responses.20 This feedback was used to refine the survey. The online survey was programmed using Qualtrics Software and manually translated into Spanish.
Given the multimodal design, surveys were distributed by email, text message, and US Postal Service (USPS). Surveys were emailed to all veterans for whom a valid email address was available. If emails were undeliverable, veterans were contacted via text message or the USPS. Surveys were distributed by text message to all veterans without an email address but with a cellphone number. We were unable to consistently identify invalid numbers among all text message recipients. Invitations with a survey URL and QR code were sent via USPS to veterans who had no valid email address or cellphone number. Targeted efforts were made to increase the response rate for women. A random sample of 200 women who had not completed the survey 2 weeks prior to the closing date (15% of women in sample) was selected to receive personal phone calls. Another random sample of 400 women was selected to receive personalized outreach emails. The survey data were confidential, and responses could not be traced to identifying information.
Data Analyses
We conducted a descriptive analysis, including percentages and means for responses to variables focused on describing amputation characteristics, prosthesis characteristics, and QOL. All data, including missing values, were used to document the percentage of respondents for each question. Removing missing data from the denominator when calculating percentages could introduce bias to the analysis because we cannot be certain data are missing at random. Missing variables were removed to avoid underinflation of mean scores.
We compared responses across 2 groups: individuals who self-identified as men and individuals who self-identified as women. For each question, we assessed whether each of these groups differed significantly from the remaining sample. For example, we examined whether the percentage of men who answered affirmatively to a question was significantly higher or lower than that of individuals not identifying as male, and whether the percentage of women who answered affirmatively was significantly higher or lower than that of individuals not identifying as female. We utilized x2 tests to determine significant differences for percentage calculations and t tests to determine significant differences in means across gender.
Since conducting multiple comparisons within a dataset may result in inflating statistical significance (type 1 errors), we used a more conservative estimate of statistical significance (α = 0.01) and high significance (α = 0.001). This study was deemed quality improvement by the VHA Rehabilitation and Prosthetic Services (12RPS) and acknowledged by the VA Research Office at Eastern Colorado Health Care System and was not subject to institutional review board review.
Results
Surveys were distributed to 46,613 veterans and were completed by 4981 respondents for a 10.7% overall response rate. Survey respondents were generally similar to the eligible population invited to participate, but the proportion of women who completed the survey was higher than the proportion of women eligible to participate (2.0% of eligible population vs 16.7% of respondents), likely due to specific efforts to target women. Survey respondents were slightly younger than the general population (67.3 years vs 68.7 years), less likely to be male (97.1% vs 83.3%), showed similar representation of Operation Enduring Freedom/Operation Iraqi Freedom (OEF/OIF) veterans (4.4% vs 4.1%), and were less likely to have diabetes (58.0% vs 52.7% had diabetes) (Table 1).
The mean age of male respondents was 67.3 years, while the mean age of female respondents was 58.3 years. The majority of respondents were male (83.3%) and White (77.2%). Female respondents were less likely to have diabetes (35.4% of women vs 53.5% of men) and less likely to report that their most recent amputation resulted from diabetes (10.1% of women vs 22.2% of men). Women respondents were more likely to report an amputation due to other causes, such as adverse results of surgery, neurologic disease, suicide attempt, blood clots, tumors, rheumatoid arthritis, and revisions of previous amputations. Most women respondents did not serve during the OEF or OIF eras. The most common amputation site for women respondents was lower limb, either below the knee and above the ankle or above the knee.
Most participants use an everyday prosthesis, but women were more likely to report using a sports-specific prosthesis (Table 2). Overall, most respondents report using a prosthesis (87.7%); however, women were more likely to report not using a prosthesis (19.4% of women vs 11.1% of men; P ≤ .01). Additionally, a lower proportion of women report using a prosthesis for < 12 hours per day (30.6% of women vs 46.4% of men; P ≤ .01) or using a prosthesis every day (54.8% of women vs 74.6% of men; P ≤ .001).
In the overall sample, the mean satisfaction score with a prosthesis was 2.7 on a 5-point scale, and women had slightly lower overall satisfaction scores (2.6 for women vs 2.7 for men; P ≤ .001) (Table 3). Women also had lower satisfaction scores related to appearance, usefulness, reliability, and comfort. Women were more likely to indicate that it was very important to be able to wear jewelry and accessories (20.2% of women vs 11.6% of men; P ≤ .01), while men were less likely to indicate that it was somewhat or very important that the prosthesis not restrict clothing or shoes (95.2% of women vs 82.9% of men; P ≤ .001). Men were more likely than women to report being comfortable or very comfortable using their prosthesis in intimate contact: 40.5% vs 29.0%, respectively (P ≤ .001).
Overall, participants reported high satisfaction with appointment times, wait times, courteous treatment, opportunities to express concerns, and staff responsiveness. Men were slightly more likely than women to be satisfied with training (P ≤ 0.001) and problem discussion (P ≤ 0.01) (Table 4). There were no statistically significant differences in satisfaction or QOL ratings between women and men. The overall sample rated both QOL and satisfaction with QOL 6.7 on a 10-point scale.
Discussion
The goal of this study was to characterize the experience of veterans with limb loss receiving care in the VHA and assess their satisfaction with prostheses and prosthetic care. We received responses from nearly 5000 veterans, 158 of whom were women. Women veteran respondents were slightly younger and less likely to have an amputation due to diabetes. We did not observe significant differences in amputation level between men and women but women were less likely to use a prosthesis, reported lower intensity of prosthesis use, and were less satisfied with certain aspects of their prostheses. Women may also be less satisfied with prosthesis training and problem discussion. However, we found no differences in QOL ratings between men and women.
Findings indicating women were more likely to report not using a prosthesis and that a lower proportion of women report using a prosthesis for > 12 hours a day or every day are consistent with previous research. 21,22 Interestingly, women were more likely to report using a sports-specific prosthesis. This is notable because prior research suggests that individuals with amputations may avoid participating in sports and exercise, and a lack of access to sports-specific prostheses may inhibit physical activity.23,24 Women in this sample were slightly less satisfied with their prostheses overall and reported lower satisfaction scores regarding appearance, usefulness, reliability, and comfort, consistent with previous findings.25
A lower percentage of women in this sample reported being comfortable or very comfortable using their prosthesis during intimate contact. Previous research on prosthesis satisfaction suggests individuals who rate prosthesis satisfaction lower also report lower body image across genders. 26 While women in this sample did not rate their prosthesis satisfaction lower than men, they did report lower intensity of prosthesis use, suggesting potential issues with their prostheses this survey did not evaluate. Women indicated the importance of prostheses not restricting jewelry, accessories, clothing, or shoes. These results have significant clinical and social implications. A recent qualitative study emphasizes that women veterans feel prostheses are primarily designed for men and may not work well with their physiological needs.9 Research focused on limbs better suited to women’s bodies could result in better fitting sockets, lightweight limbs, or less bulky designs. Additional research has also explored the difficulties in accommodating a range of footwear for patients with lower limb amputation. One study found that varying footwear heights affect the function of adjustable prosthetic feet in ways that may not be optimal.27
Ratings of satisfaction with prosthesisrelated services between men and women in this sample are consistent with a recent study showing that women veterans do not have significant differences in satisfaction with prosthesis-related services.28 However, this study focused specifically on lower limb amputations, while the respondents of this study include those with both upper and lower limb amputations. Importantly, our findings that women are less likely to be satisfied with prosthesis training and problem discussions support recent qualitative findings in which women expressed a desire to work with prosthetists who listen to them, take their concerns seriously, and seek solutions that fit their needs. We did not observe a difference in QOL ratings between men and women in the sample despite lower satisfaction among women with some elements of prosthesis-related services. Previous research suggests many factors impact QOL after amputation, most notably time since amputation.16,29
Limitations
This survey was deployed in a short timeline that did not allow for careful sample selection or implementing strategies to increase response rate. Additionally, the study was conducted among veterans receiving care in the VHA, and findings may not be generalizable to limb loss in other settings. Finally, the discrepancy in number of respondents who identified as men vs women made it difficult to compare differences between the 2 groups.
Conclusions
This is the largest sample of survey respondents of veterans with limb loss to date. While the findings suggest veterans are generally satisfied with prosthetic-related services overall, they also highlight several areas for improvement with services or prostheses. Given that most veterans with limb loss are men, there is a significant discrepancy between the number of women and men respondents. Additional studies with more comparable numbers of men and women have found similar ratings of satisfaction with prostheses and services.28 Further research specifically focused on improving the experiences of women should focus on better characterizing their experiences and identifying how they differ from those of male veterans. For example, understanding how to engage female veterans with limb loss in prosthesis training and problem discussions may improve their experience with their care teams and improve their use of prostheses. Understanding experiences and needs that are specific to women could lead to the development of processes, resources, or devices that are tailored to the unique requirements of women with limb loss.
- Ziegler-Graham K, MacKenzie EJ, Ephraim PL, Travison TG, Brookmeyer R. Estimating the prevalence of limb loss in the United States: 2005 to 2050. Arch Phys Med Rehabil. 2008;89(3):422-429. doi:10.1016/j.apmr.2007.11.005
- Dillingham TR, Pezzin LE, MacKenzie EJ. Limb amputation and limb deficiency: epidemiology and recent trends in the united states. South Med J. 2002;95(8):875-883. doi:10.1097/00007611-200208000-00018
- Dillingham TR, Pezzin LE, Shore AD. Reamputation, mortality, and health care costs among persons with dysvascular lower-limb amputations. Arch Phys Med Rehabil. 2005;86(3):480-486. doi:10.1016/j.apmr.2004.06.072
- Centers for Disease Control and Prevention. Ambulatory and inpatient procedures in the United States. Accessed September 30, 2024. https://www.cdc.gov/nchs/pressroom/98facts/ambulat.htm
- Ljung J, Iacangelo A. Identifying and acknowledging a sex gap in lower-limb prosthetics. JPO. 2024;36(1):e18-e24. doi:10.1097/JPO.0000000000000470
- Feinglass J, Brown JL, LoSasso A, et al. Rates of lower-extremity amputation and arterial reconstruction in the united states, 1979 to 1996. Am J Public Health. 1999;89(8):1222- 1227. doi:10.2105/ajph.89.8.1222
- Mayfield JA, Reiber GE, Maynard C, Czerniecki JM, Caps MT, Sangeorzan BJ. Trends in lower limb amputation in the Veterans Health Administration, 1989-1998. J Rehabil Res Dev. 2000;37(1):23-30.
- Feinglass J, Pearce WH, Martin GJ, et al. Postoperative and late survival outcomes after major amputation: findings from the department of veterans affairs national surgical quality improvement program. Surgery. 2001;130(1):21-29. doi:10.1067/msy.2001.115359
- Lehavot K, Young JP, Thomas RM, et al. Voices of women veterans with lower limb prostheses: a qualitative study. J Gen Intern Med. 2022;37(3):799-805. doi:10.1007/s11606-022-07572-8
- US Government Accountability Office. COVID-19: Opportunities to improve federal response. GAO-21-60. Published November 12, 2020. Accessed September 30, 2024. https://www.gao.gov/products/gao-21-60
- Littman AJ, Peterson AC, Korpak A, et al. Differences in prosthetic prescription between men and women veterans after transtibial or transfemoral lowerextremity amputation: a longitudinal cohort study. Arch Phys Med Rehabil. 2023;104(8)1274-1281. doi:10.1016/j.amjsurg.2023.02.011
- Cimino SR, Vijayakumar A, MacKay C, Mayo AL, Hitzig SL, Guilcher SJT. Sex and gender differences in quality of life and related domains for individuals with adult acquired lower-limb amputation: a scoping review. Disabil Rehabil. 2022 Oct 23;44(22):6899-6925. doi:10.1080/09638288.2021.1974106
- DadeMatthews OO, Roper JA, Vazquez A, Shannon DM, Sefton JM. Prosthetic device and service satisfaction, quality of life, and functional performance in lower limb prosthesis clients. Prosthet Orthot Int. 2024;48(4):422-430. doi:10.1097/PXR.0000000000000285
- Hamilton AB, Schwarz EB, Thomas HN, Goldstein KM. Moving women veterans’ health research forward: a special supplement. J Gen Intern Med. 2022;37(Suppl3):665– 667. doi:10.1007/s11606-022-07606-1
- US Congress. Public Law 116-315: An Act to Improve the Lives of Veterans, S 5108 (2) (F). 116th Congress; 2021. Accessed September 30, 2024. https://www.congress.gov/116/plaws/publ315/PLAW-116publ315.pdf
- Gallagher P, MacLachlan M. The Trinity amputation and prosthesis experience scales and quality of life in people with lower-limb amputation. Arch Phys Med Rehabil. 2004;85(5):730-736. doi:10.1016/j.apmr.2003.07.009
- Legro MW, Reiber GD, Smith DG, del Aguila M, Larsen J, Boone D. Prosthesis evaluation questionnaire for persons with lower limb amputations: assessing prosthesis-related quality of life. Arch Phys Med Rehabil. 1998;79(8):931-938. doi:10.1016/s0003-9993(98)90090-9
- Legro MW, Reiber GD, Smith DG, del Aguila M, Larsen J, Boone D. Prosthesis evaluation questionnaire for persons with lower limb amputations: assessing prosthesis-related quality of life. Arch Phys Med Rehabil. 1998;79(8):931-938. doi:10.1016/s0003-9993(98)90090-9
- Heinemann AW, Bode RK, O’Reilly C. Development and measurement properties of the orthotics and prosthetics users’ survey (OPUS): a comprehensive set of clinical outcome instruments. Prosthet Orthot Int. 2003;27(3):191-206. doi:10.1080/03093640308726682
- Resnik LJ, Borgia ML, Clark MA. A national survey of prosthesis use in veterans with major upper limb amputation: comparisons by gender. PM R. 2020;12(11):1086-1098. doi:10.1002/pmrj.12351
- Collins D. Pretesting survey instruments: an overview of cognitive methods. Qual Life Res. 2003;12(3):229-238. doi:10.1023/a:1023254226592
- Østlie K, Lesjø IM, Franklin RJ, Garfelt B, Skjeldal OH, Magnus P. Prosthesis rejection in acquired major upper-limb amputees: a population-based survey. Disabil Rehabil Assist Technol. 2012;7(4):294-303. doi:10.3109/17483107.2011.635405
- Pezzin LE, Dillingham TR, MacKenzie EJ, Ephraim P, Rossbach P. Use and satisfaction with prosthetic limb devices and related services. Arch Phys Med Rehabil. 2004;85(5):723-729. doi:10.1016/j.apmr.2003.06.002
- Deans S, Burns D, McGarry A, Murray K, Mutrie N. Motivations and barriers to prosthesis users participation in physical activity, exercise and sport: a review of the literature. Prosthet Orthot Int. 2012;36(3):260-269. doi:10.1177/0309364612437905
- McDonald CL, Kahn A, Hafner BJ, Morgan SJ. Prevalence of secondary prosthesis use in lower limb prosthesis users. Disabil Rehabil. 2023;46(5):1016-1022. doi:10.1080/09638288.2023.2182919
- Baars EC, Schrier E, Dijkstra PU, Geertzen JHB. Prosthesis satisfaction in lower limb amputees: a systematic review of associated factors and questionnaires. Medicine (Baltimore). 2018;97(39):e12296. doi:10.1097/MD.0000000000012296
- Murray CD, Fox J. Body image and prosthesis satisfaction in the lower limb amputee. Disabil Rehabil. 2002;24(17):925–931. doi:10.1080/09638280210150014
- Major MJ, Quinlan J, Hansen AH, Esposito ER. Effects of women’s footwear on the mechanical function of heel-height accommodating prosthetic feet. PLoS One. 2022;17(1). doi:10.1371/journal.pone.0262910.
- Kuo PB, Lehavot K, Thomas RM, et al. Gender differences in prosthesis-related outcomes among veterans: results of a national survey of U.S. veterans. PM R. 2024;16(3):239- 249. doi:10.1002/pmrj.13028
- Asano M, Rushton P, Miller WC, Deathe BA. Predictors of quality of life among individuals who have a lower limb amputation. Prosthet Orthot Int. 2008;32(2):231-243. doi:10.1080/03093640802024955
- Ziegler-Graham K, MacKenzie EJ, Ephraim PL, Travison TG, Brookmeyer R. Estimating the prevalence of limb loss in the United States: 2005 to 2050. Arch Phys Med Rehabil. 2008;89(3):422-429. doi:10.1016/j.apmr.2007.11.005
- Dillingham TR, Pezzin LE, MacKenzie EJ. Limb amputation and limb deficiency: epidemiology and recent trends in the united states. South Med J. 2002;95(8):875-883. doi:10.1097/00007611-200208000-00018
- Dillingham TR, Pezzin LE, Shore AD. Reamputation, mortality, and health care costs among persons with dysvascular lower-limb amputations. Arch Phys Med Rehabil. 2005;86(3):480-486. doi:10.1016/j.apmr.2004.06.072
- Centers for Disease Control and Prevention. Ambulatory and inpatient procedures in the United States. Accessed September 30, 2024. https://www.cdc.gov/nchs/pressroom/98facts/ambulat.htm
- Ljung J, Iacangelo A. Identifying and acknowledging a sex gap in lower-limb prosthetics. JPO. 2024;36(1):e18-e24. doi:10.1097/JPO.0000000000000470
- Feinglass J, Brown JL, LoSasso A, et al. Rates of lower-extremity amputation and arterial reconstruction in the united states, 1979 to 1996. Am J Public Health. 1999;89(8):1222- 1227. doi:10.2105/ajph.89.8.1222
- Mayfield JA, Reiber GE, Maynard C, Czerniecki JM, Caps MT, Sangeorzan BJ. Trends in lower limb amputation in the Veterans Health Administration, 1989-1998. J Rehabil Res Dev. 2000;37(1):23-30.
- Feinglass J, Pearce WH, Martin GJ, et al. Postoperative and late survival outcomes after major amputation: findings from the department of veterans affairs national surgical quality improvement program. Surgery. 2001;130(1):21-29. doi:10.1067/msy.2001.115359
- Lehavot K, Young JP, Thomas RM, et al. Voices of women veterans with lower limb prostheses: a qualitative study. J Gen Intern Med. 2022;37(3):799-805. doi:10.1007/s11606-022-07572-8
- US Government Accountability Office. COVID-19: Opportunities to improve federal response. GAO-21-60. Published November 12, 2020. Accessed September 30, 2024. https://www.gao.gov/products/gao-21-60
- Littman AJ, Peterson AC, Korpak A, et al. Differences in prosthetic prescription between men and women veterans after transtibial or transfemoral lowerextremity amputation: a longitudinal cohort study. Arch Phys Med Rehabil. 2023;104(8)1274-1281. doi:10.1016/j.amjsurg.2023.02.011
- Cimino SR, Vijayakumar A, MacKay C, Mayo AL, Hitzig SL, Guilcher SJT. Sex and gender differences in quality of life and related domains for individuals with adult acquired lower-limb amputation: a scoping review. Disabil Rehabil. 2022 Oct 23;44(22):6899-6925. doi:10.1080/09638288.2021.1974106
- DadeMatthews OO, Roper JA, Vazquez A, Shannon DM, Sefton JM. Prosthetic device and service satisfaction, quality of life, and functional performance in lower limb prosthesis clients. Prosthet Orthot Int. 2024;48(4):422-430. doi:10.1097/PXR.0000000000000285
- Hamilton AB, Schwarz EB, Thomas HN, Goldstein KM. Moving women veterans’ health research forward: a special supplement. J Gen Intern Med. 2022;37(Suppl3):665– 667. doi:10.1007/s11606-022-07606-1
- US Congress. Public Law 116-315: An Act to Improve the Lives of Veterans, S 5108 (2) (F). 116th Congress; 2021. Accessed September 30, 2024. https://www.congress.gov/116/plaws/publ315/PLAW-116publ315.pdf
- Gallagher P, MacLachlan M. The Trinity amputation and prosthesis experience scales and quality of life in people with lower-limb amputation. Arch Phys Med Rehabil. 2004;85(5):730-736. doi:10.1016/j.apmr.2003.07.009
- Legro MW, Reiber GD, Smith DG, del Aguila M, Larsen J, Boone D. Prosthesis evaluation questionnaire for persons with lower limb amputations: assessing prosthesis-related quality of life. Arch Phys Med Rehabil. 1998;79(8):931-938. doi:10.1016/s0003-9993(98)90090-9
- Legro MW, Reiber GD, Smith DG, del Aguila M, Larsen J, Boone D. Prosthesis evaluation questionnaire for persons with lower limb amputations: assessing prosthesis-related quality of life. Arch Phys Med Rehabil. 1998;79(8):931-938. doi:10.1016/s0003-9993(98)90090-9
- Heinemann AW, Bode RK, O’Reilly C. Development and measurement properties of the orthotics and prosthetics users’ survey (OPUS): a comprehensive set of clinical outcome instruments. Prosthet Orthot Int. 2003;27(3):191-206. doi:10.1080/03093640308726682
- Resnik LJ, Borgia ML, Clark MA. A national survey of prosthesis use in veterans with major upper limb amputation: comparisons by gender. PM R. 2020;12(11):1086-1098. doi:10.1002/pmrj.12351
- Collins D. Pretesting survey instruments: an overview of cognitive methods. Qual Life Res. 2003;12(3):229-238. doi:10.1023/a:1023254226592
- Østlie K, Lesjø IM, Franklin RJ, Garfelt B, Skjeldal OH, Magnus P. Prosthesis rejection in acquired major upper-limb amputees: a population-based survey. Disabil Rehabil Assist Technol. 2012;7(4):294-303. doi:10.3109/17483107.2011.635405
- Pezzin LE, Dillingham TR, MacKenzie EJ, Ephraim P, Rossbach P. Use and satisfaction with prosthetic limb devices and related services. Arch Phys Med Rehabil. 2004;85(5):723-729. doi:10.1016/j.apmr.2003.06.002
- Deans S, Burns D, McGarry A, Murray K, Mutrie N. Motivations and barriers to prosthesis users participation in physical activity, exercise and sport: a review of the literature. Prosthet Orthot Int. 2012;36(3):260-269. doi:10.1177/0309364612437905
- McDonald CL, Kahn A, Hafner BJ, Morgan SJ. Prevalence of secondary prosthesis use in lower limb prosthesis users. Disabil Rehabil. 2023;46(5):1016-1022. doi:10.1080/09638288.2023.2182919
- Baars EC, Schrier E, Dijkstra PU, Geertzen JHB. Prosthesis satisfaction in lower limb amputees: a systematic review of associated factors and questionnaires. Medicine (Baltimore). 2018;97(39):e12296. doi:10.1097/MD.0000000000012296
- Murray CD, Fox J. Body image and prosthesis satisfaction in the lower limb amputee. Disabil Rehabil. 2002;24(17):925–931. doi:10.1080/09638280210150014
- Major MJ, Quinlan J, Hansen AH, Esposito ER. Effects of women’s footwear on the mechanical function of heel-height accommodating prosthetic feet. PLoS One. 2022;17(1). doi:10.1371/journal.pone.0262910.
- Kuo PB, Lehavot K, Thomas RM, et al. Gender differences in prosthesis-related outcomes among veterans: results of a national survey of U.S. veterans. PM R. 2024;16(3):239- 249. doi:10.1002/pmrj.13028
- Asano M, Rushton P, Miller WC, Deathe BA. Predictors of quality of life among individuals who have a lower limb amputation. Prosthet Orthot Int. 2008;32(2):231-243. doi:10.1080/03093640802024955
Satisfaction With Department of Veterans Affairs Prosthetics and Support Services as Reported by Women and Men Veterans
Satisfaction With Department of Veterans Affairs Prosthetics and Support Services as Reported by Women and Men Veterans
Common Ground: Primary Care and Specialty Clinicians’ Perceptions of E-Consults in the Veterans Health Administration
Electronic consultation (e-consult) is designed to increase access to specialty care by facilitating communication between primary care and specialty clinicians without the need for outpatient face-to-face encounters.1–4 In 2011, the US Department of Veterans Affairs (VA) implemented an e-consult program as a component of its overall strategy to increase access to specialty services, reduce costs of care, and reduce appointment travel burden on patients.
E-consult has substantially increased within the VA since its implementation.5,6 Consistent with limited evaluations from other health care systems, evaluations of the VA e-consult program demonstrated reduced costs, reduced travel time for patients, and improved access to specialty care.2,5–11 However, there is wide variation in e-consult use across VA specialties, facilities, and regions.5,6,12,13 For example, hematology, preoperative evaluation, neurosurgery, endocrinology, and infectious diseases use e-consults more frequently when compared with in-person consults in the VA.6 Reasons for this variation or specific barriers and facilitators of using e-consults have not been described.
Prior qualitative studies report that primary care practitioners (PCPs) describe e-consults as convenient, educational, beneficial for patient care, and useful for improving patient access to specialty care.8,14,15 One study identified limited PCP knowledge of e-consults as a barrier to use.16 Specialists have reported that e-consult improves clinical communication, but increases their workload.1,14,17,18 These studies did not assess perspectives from both clinicians who initiate e-consults and those who respond to them. This is the first qualitative study to assess e-consult perceptions from perspectives of both PCPs and specialists among a large, national sample of VA clinicians who use e-consults. The objective of this study was to understand perspectives of e-consults between PCPs and specialists that may be relevant to increasing adoption in the VA.
Methods
The team (CL, ML, PG, 2 analysts under the guidance of GS and JS and support from RRK, and a biostatistician) conducted semistructured interviews with PCPs, specialists, and specialty division leaders who were employed by VA in 2016 and 2017. Specialties of interest were identified by the VA Office of Specialty Care and included cardiology, endocrinology, gastroenterology, and hematology.
E-Consult Procedures
Within the VA, the specific procedures used to initiate, triage and manage e-consults are coordinated at VA medical centers (VAMCs) and at the Veterans Integrated Service Network (VISN) regional level. E-consult can be requested by any clinician. Generally, e-consults are initiated by PCPs through standardized, specialty-specific templates. Recipients, typically specialists, respond by answering questions, suggesting additional testing and evaluation, or requesting an in-person visit. Communication is documented in the patient’s electronic health record (EHR). Specialists receive different levels of workload credit for responding to e-consults similar to a relative value unit reimbursement model. Training in the use of e-consults is available to practitioners but may vary at local and regional levels.
Recruitment
Our sample included PCPs, specialists, and specialty care division leaders. We first quantified e-consult rates (e-consults per 100 patient visits) between July 2016 and June 2017 at VA facilities within primary care and the 4 priority specialties and identified the 30 sites with the highest e-consult rates and 30 sites with the lowest e-consult rates. Sites with < 500 total visits, < 3 specialties, or without any e-consult visit during the study period were excluded. E-consult rates at community-based outpatient clinics were included with associated VAMCs. We then stratified PCPs by whether they were high or low users of e-consults (determined by the top and bottom users within each site) and credentials (MD vs nurse practitioner [NP] or physician assistant [PA]). Specialists were sampled based on their rate of use relative to colleagues within their site and the use rate of their division. We sampled division chiefs and individuals who had > 300 total visits and 1 e-consult during the study period. To recruit participants, the primary investigator sent an initial email and 2 reminder emails. The team followed up with respondents to schedule an interview.
Interview guides were designed to elicit rich descriptions of barriers and facilitators to e-consult use (eAppendix available at doi:10.12788/fp.0214). The team used the Practical Robust Implementation and Sustainability Model (PRISM), which considers factors along 6 domains for intervention planning, implementation, and sustainment.19 Telephone interviews lasted about 20 minutes and were conducted between September 2017 and March 2018. Interviews were recorded and transcribed verbatim.
Analysis
The team used an iterative, team-based, inductive/deductive approach to conventional content analysis.20,21 Initial code categories were created so that we could identify e-consult best practices—facilitators of e-consult that were recommended by both PCPs and specialists. Inductive codes or labels applied to identify meaningful quotations, phrases, or key terms were used to identify emergent ideas and were added throughout coding after discussion among team members. Consensus was reached using a team-based approach.21 Four analysts independently coded the same 3 transcripts and met to discuss points of divergence and convergence. Analyses continued with emergent themes, categories, and conclusions. Atlas.ti. v.7 was used for coding and data management.22
Results
We conducted 34 interviews with clinicians (Table 1) from 13 VISNs. Four best-practice themes emerged among both PCPs and specialists, including that e-consults (1) are best suited for certain clinical questions and patients; (2) require relevant background information from requesting clinicians and clear recommendations from responding clinicians; (3) are a novel opportunity to provide efficient, transparent care; and (4) may not be fully adopted due to low awareness. Supporting quotations for the following findings are provided in Table 2.
Specific Clinical Questions and Patients
PCPs described specific patients and questions for which they most frequently used e-consults, such as for medication changes (Q1), determining treatment steps (Q2,3), and or clarifying laboratory or imaging findings. PCPs frequently used e-consults for patients who did not require a physical examination or when specialists could make recommendations without seeing patients face-to-face (Q3). An important use of e-consults described by PCPs was for treating conditions they could manage within primary care if additional guidance were available (Q4). Several PCPs and specialists also noted that e-consults were particularly useful for patients who were unable to travel or did not want face-to-face appointments (Q5). Notably, PCPs and specialists mentioned situations for which e-consults were inappropriate, including when a detailed history or physical examination was needed, or if a complex condition was suspected (Q6).
Background Data and Clear Recommendations
Participants described necessary data that should be included in high-quality e-consults. Specialists voiced frustration in time-consuming chart reviews that were often necessary when these data were not provided by the requestor. In some cases, specialists were unable to access necessary EHR data, which delayed responses (Q7). PCPs noted that the most useful responses carefully considered the question, used current patient information to determine treatments, provided clear recommendations, and defined who was responsible for next steps (Q8). PCPs and specialists stated that e-consult templates that required relevant information facilitated high-quality e-consults. Neither wanted to waste the other clinician's time (Q8).
A Novel Opportunity
Many PCPs felt that e-consults improved communication (eg, efficiency, response time), established new communication between clinicians, and reduced patients’ appointment burden (Q10, Q11). Many specialists felt that e-consults improved documentation of communication between clinicians and increased transparency of clinical decisions (Q12). Additionally, many specialists mentioned that e-consults capture previously informal curbside consults, enabling them to receive workload credit (Q13).
Lack of Awareness
Some noted that the biggest barrier to e-consults was not being aware of them generally, or which specialties offer e-consults (Q14). One PCP described e-consults as the best kept secret and found value in sharing the utility of e-consults with colleagues (Q15). All participants, including those who did not frequently use e-consults, felt that e-consults improved the quality of care by providing more timely care or better answers to clinical questions (Q16). Several practitioners also felt that e-consults increased access to specialty care. For example, specialists reported that e-consults enabled them to better manage patient load by using e-consults to answer relatively simple questions, reserving face-to-face consults for more complex patients (Q17).
Discussion
The objective of this study was to identify potential best practices for e-consults that may help increase their quality and use within the VA. We built on prior studies that offered insights on PCP and specialists’ overall satisfaction with e-consult by identifying several themes relevant to the further adoption of e-consults in the VA and elsewhere without a face-to-face visit.8,13,14,16–18 Future work may be beneficial in identifying whether the study themes identified can explain variation in e-consult use or whether addressing these factors might lead to increased or higher quality e-consult use. We are unaware of any qualitative study of comparable scale in a different health care system. Further, this is the first study to assess perspectives on e-consults among those who initiate and respond to them within the same health care system. Perhaps the most important finding from this study is that e-consults are generally viewed favorably, which is a necessary leverage point to increase their adoption within the system.
Clinicians reported several benefits to e-consults, including timely responses to clinical questions, efficient communication, allow for documentation of specialist recommendations, and help capture workload. These benefits are consistent with prior literature that indicates both PCPs and specialists in the VA and other health care systems feel that e-consults improves communication, decreases unnecessary visits, and improves quality of care.1,14,17,18 In particular, clinicians reported that e-consults improve their practice efficiency and efficacy. This is of critical importance given the pressures of providing timely access to primary and specialty care within the VA. Interestingly, many VA practitioners were unaware which specialties offered e-consults within their facilities, reflecting previous work showing that PCPs are often unaware of e-consult options.16 This may partially explain variation in e-consult use. Increasing awareness and educating clinicians on the benefits of e-consults may help promote use among non- and low users.
A common theme reported by both groups was the importance of providing necessary information within e-consult questions and responses. Specialists felt there was a need to ensure that PCPs provide relevant and patient-specific information that would enable them to efficiently and accurately answer questions without the need for extensive EHR review. This reflects previous work showing that specialists are often unable to respond to e-consult requests because they do not contain sufficient information.22 PCPs described a need to ensure that specialists’ responses included information that was detailed enough to make clinical decisions without the need for a reconsult. This highlights a common challenge to medical consultation, in that necessary or relevant information may not be apparent to all clinicians. To address this, there may be a role in developing enhanced, flexible templating that elicits necessary patient-specific information. Such a template may automatically pull relevant data from the EHR and prompt clinicians to provide important information. We did not assess how perspectives of templates varied, and further work could help define precisely what constitutes an effective template, including how it should capture appropriate patient data and how this impacts acceptability or use of e-consults generally. Collaboratively developed service agreements and e-consult templates could help guide PCPs and specialists to engage in efficient communication.
Another theme among both groups was that e-consult is most appropriate within specific clinical scenarios. Examples included review of laboratory results, questions about medication changes, or for patients who were reluctant to travel to appointments. Identifying and promoting specific opportunities for e-consults may help increase their use and align e-consult practices with scenarios that are likely to provide the most benefit to patients. For example, it could be helpful to understand the distance patients must travel for specialty care. Providing that information during clinical encounters could trigger clinicians to consider e-consults as an option. Future work might aim to identify clinical scenarios that clinicians feel are not well suited for e-consults and determine how to adapt them for those scenarios.
Limitations
Generalizability of these findings is limited given the qualitative study design. Participants’ descriptions of experiences with e-consults reflect the experiences of clinicians in the VA and may not reflect clinicians in other settings. We also interviewed a sample of clinicians who were already using e-consults. Important information could be learned from future work with those who have not yet adopted e-consult procedures or adopted and abandoned them.
Conclusions
E-consult is perceived as beneficial by VA PCPs and specialists. Participants suggested using e-consults for appropriate questions or patients and including necessary information and next steps in both the initial e-consult and response. Finding ways to facilitate e-consults with these suggestions in mind may increase delivery of high-quality e-consults. Future work could compare the findings of this work to similar work assessing clinicians perceptions of e-consults outside of the VA.
1. Battaglia C, Lambert-Kerzner A, Aron DC, et al. Evaluation of e-consults in the VHA: provider perspectives. Fed Pract. 2015;32(7):42-48.
2. Haverhals LM, Sayre G, Helfrich CD, et al. E-consult implementation: lessons learned using consolidated framework for implementation research. Am J Manag Care. 2015;21(12):e640-e647. Published 2015 Dec 1.
3. Sewell JL, Telischak KS, Day LW, Kirschner N, Weissman A. Preconsultation exchange in the United States: use, awareness, and attitudes. Am J Manag Care. 2014;20(12):e556-e564. Published 2014 Dec 1.
4. Horner K, Wagner E, Tufano J. Electronic consultations between primary and specialty care clinicians: early insights. Issue Brief (Commonw Fund). 2011;23:1-14.
5. Kirsh S, Carey E, Aron DC, et al. Impact of a national specialty e-consultation implementation project on access. Am J Manag Care. 2015;21(12):e648-654. Published 2015 Dec 1.
6. Saxon DR, Kaboli PJ, Haraldsson B, Wilson C, Ohl M, Augustine MR. Growth of electronic consultations in the Veterans Health Administration. Am J Manag Care. 2021;27(1):12-19. doi:10.37765/ajmc.2021.88572
7. Olayiwola JN, Anderson D, Jepeal N, et al. Electronic consultations to improve the primary care-specialty care interface for cardiology in the medically underserved: a cluster-randomized controlled trial. Ann Fam Med. 2016;14(2):133-140. doi:10.1370/afm.1869
8. Schettini P, Shah KP, O’Leary CP, et al. Keeping care connected: e-Consultation program improves access to nephrology care. J Telemed Telecare. 2019;25(3):142-150. doi:10.1177/1357633X17748350
9. Whittington MD, Ho PM, Kirsh SR, et al. Cost savings associated with electronic specialty consultations. Am J Manag Care. 2021;27(1):e16-e23. Published 2021 Jan 1. doi:10.37765/ajmc.2021.88579
10. Shipherd JC, Kauth MR, Matza A. Nationwide interdisciplinary e-consultation on transgender care in the Veterans Health Administration. Telemed J E Health. 2016;22(12):1008-1012. doi:10.1089/tmj.2016.0013
11. Strymish J, Gupte G, Afable MK, et al. Electronic consultations (E-consults): advancing infectious disease care in a large Veterans Affairs Healthcare System. Clin Infect Dis. 2017;64(8):1123-1125. doi:10.1093/cid/cix058
12. Williams KM, Kirsh S, Aron D, et al. Evaluation of the Veterans Health Administration’s Specialty Care Transformational Initiatives to promote patient-centered delivery of specialty care: a mixed-methods approach. Telemed J E-Health. 2017;23(7):577-589. doi:10.1089/tmj.2016.0166
13. US Department of Veterans Affairs, Veterans Health Administration, Specialty Care Transformational Initiative Evaluation Center. Evaluation of specialty care initiatives. Published 2013.
14. Vimalananda VG, Gupte G, Seraj SM, et al. Electronic consultations (e-consults) to improve access to specialty care: a systematic review and narrative synthesis. J Telemed Telecare. 2015;21(6):323-330. doi:10.1177/1357633X15582108
15. Lee M, Leonard C, Greene P, et al. Perspectives of VA primary care clinicians toward electronic consultation-related workload burden. JAMA Netw Open. 2020;3(10):e2018104. Published 2020 Oct 1. doi:10.1001/jamanetworkopen.2020.18104
16. Deeds SA, Dowdell KJ, Chew LD, Ackerman SL. Implementing an opt-in eConsult program at seven academic medical centers: a qualitative analysis of primary care provider experiences. J Gen Intern Med. 2019;34(8):1427-1433. doi:10.1007/s11606-019-05067-7
17. Rodriguez KL, Burkitt KH, Bayliss NK, et al. Veteran, primary care provider, and specialist satisfaction with electronic consultation. JMIR Med Inform. 2015;3(1):e5. Published 2015 Jan 14. doi:10.2196/medinform.3725
18. Gupte G, Vimalananda V, Simon SR, DeVito K, Clark J, Orlander JD. Disruptive innovation: implementation of electronic consultations in a Veterans Affairs Health Care System. JMIR Med Inform. 2016;4(1):e6. Published 2016 Feb 12. doi:10.2196/medinform.4801
19. Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf. 2008;34(4):228-243. doi:10.1016/s1553-7250(08)34030-6
20. Patton MQ. Qualitative Research and Evaluation Methods. 3rd ed. Sage Publications; 2002.
21. Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv Res. 2007;42(4):1758-1772. doi:10.1111/j.1475-6773.2006.00684.x
22. Kim EJ, Orlander JD, Afable M, et al. Cardiology electronic consultation (e-consult) use by primary care providers at VA medical centres in New England. J Telemed Telecare. 2019;25(6):370-377. doi:10.1177/1357633X18774468
Electronic consultation (e-consult) is designed to increase access to specialty care by facilitating communication between primary care and specialty clinicians without the need for outpatient face-to-face encounters.1–4 In 2011, the US Department of Veterans Affairs (VA) implemented an e-consult program as a component of its overall strategy to increase access to specialty services, reduce costs of care, and reduce appointment travel burden on patients.
E-consult has substantially increased within the VA since its implementation.5,6 Consistent with limited evaluations from other health care systems, evaluations of the VA e-consult program demonstrated reduced costs, reduced travel time for patients, and improved access to specialty care.2,5–11 However, there is wide variation in e-consult use across VA specialties, facilities, and regions.5,6,12,13 For example, hematology, preoperative evaluation, neurosurgery, endocrinology, and infectious diseases use e-consults more frequently when compared with in-person consults in the VA.6 Reasons for this variation or specific barriers and facilitators of using e-consults have not been described.
Prior qualitative studies report that primary care practitioners (PCPs) describe e-consults as convenient, educational, beneficial for patient care, and useful for improving patient access to specialty care.8,14,15 One study identified limited PCP knowledge of e-consults as a barrier to use.16 Specialists have reported that e-consult improves clinical communication, but increases their workload.1,14,17,18 These studies did not assess perspectives from both clinicians who initiate e-consults and those who respond to them. This is the first qualitative study to assess e-consult perceptions from perspectives of both PCPs and specialists among a large, national sample of VA clinicians who use e-consults. The objective of this study was to understand perspectives of e-consults between PCPs and specialists that may be relevant to increasing adoption in the VA.
Methods
The team (CL, ML, PG, 2 analysts under the guidance of GS and JS and support from RRK, and a biostatistician) conducted semistructured interviews with PCPs, specialists, and specialty division leaders who were employed by VA in 2016 and 2017. Specialties of interest were identified by the VA Office of Specialty Care and included cardiology, endocrinology, gastroenterology, and hematology.
E-Consult Procedures
Within the VA, the specific procedures used to initiate, triage and manage e-consults are coordinated at VA medical centers (VAMCs) and at the Veterans Integrated Service Network (VISN) regional level. E-consult can be requested by any clinician. Generally, e-consults are initiated by PCPs through standardized, specialty-specific templates. Recipients, typically specialists, respond by answering questions, suggesting additional testing and evaluation, or requesting an in-person visit. Communication is documented in the patient’s electronic health record (EHR). Specialists receive different levels of workload credit for responding to e-consults similar to a relative value unit reimbursement model. Training in the use of e-consults is available to practitioners but may vary at local and regional levels.
Recruitment
Our sample included PCPs, specialists, and specialty care division leaders. We first quantified e-consult rates (e-consults per 100 patient visits) between July 2016 and June 2017 at VA facilities within primary care and the 4 priority specialties and identified the 30 sites with the highest e-consult rates and 30 sites with the lowest e-consult rates. Sites with < 500 total visits, < 3 specialties, or without any e-consult visit during the study period were excluded. E-consult rates at community-based outpatient clinics were included with associated VAMCs. We then stratified PCPs by whether they were high or low users of e-consults (determined by the top and bottom users within each site) and credentials (MD vs nurse practitioner [NP] or physician assistant [PA]). Specialists were sampled based on their rate of use relative to colleagues within their site and the use rate of their division. We sampled division chiefs and individuals who had > 300 total visits and 1 e-consult during the study period. To recruit participants, the primary investigator sent an initial email and 2 reminder emails. The team followed up with respondents to schedule an interview.
Interview guides were designed to elicit rich descriptions of barriers and facilitators to e-consult use (eAppendix available at doi:10.12788/fp.0214). The team used the Practical Robust Implementation and Sustainability Model (PRISM), which considers factors along 6 domains for intervention planning, implementation, and sustainment.19 Telephone interviews lasted about 20 minutes and were conducted between September 2017 and March 2018. Interviews were recorded and transcribed verbatim.
Analysis
The team used an iterative, team-based, inductive/deductive approach to conventional content analysis.20,21 Initial code categories were created so that we could identify e-consult best practices—facilitators of e-consult that were recommended by both PCPs and specialists. Inductive codes or labels applied to identify meaningful quotations, phrases, or key terms were used to identify emergent ideas and were added throughout coding after discussion among team members. Consensus was reached using a team-based approach.21 Four analysts independently coded the same 3 transcripts and met to discuss points of divergence and convergence. Analyses continued with emergent themes, categories, and conclusions. Atlas.ti. v.7 was used for coding and data management.22
Results
We conducted 34 interviews with clinicians (Table 1) from 13 VISNs. Four best-practice themes emerged among both PCPs and specialists, including that e-consults (1) are best suited for certain clinical questions and patients; (2) require relevant background information from requesting clinicians and clear recommendations from responding clinicians; (3) are a novel opportunity to provide efficient, transparent care; and (4) may not be fully adopted due to low awareness. Supporting quotations for the following findings are provided in Table 2.
Specific Clinical Questions and Patients
PCPs described specific patients and questions for which they most frequently used e-consults, such as for medication changes (Q1), determining treatment steps (Q2,3), and or clarifying laboratory or imaging findings. PCPs frequently used e-consults for patients who did not require a physical examination or when specialists could make recommendations without seeing patients face-to-face (Q3). An important use of e-consults described by PCPs was for treating conditions they could manage within primary care if additional guidance were available (Q4). Several PCPs and specialists also noted that e-consults were particularly useful for patients who were unable to travel or did not want face-to-face appointments (Q5). Notably, PCPs and specialists mentioned situations for which e-consults were inappropriate, including when a detailed history or physical examination was needed, or if a complex condition was suspected (Q6).
Background Data and Clear Recommendations
Participants described necessary data that should be included in high-quality e-consults. Specialists voiced frustration in time-consuming chart reviews that were often necessary when these data were not provided by the requestor. In some cases, specialists were unable to access necessary EHR data, which delayed responses (Q7). PCPs noted that the most useful responses carefully considered the question, used current patient information to determine treatments, provided clear recommendations, and defined who was responsible for next steps (Q8). PCPs and specialists stated that e-consult templates that required relevant information facilitated high-quality e-consults. Neither wanted to waste the other clinician's time (Q8).
A Novel Opportunity
Many PCPs felt that e-consults improved communication (eg, efficiency, response time), established new communication between clinicians, and reduced patients’ appointment burden (Q10, Q11). Many specialists felt that e-consults improved documentation of communication between clinicians and increased transparency of clinical decisions (Q12). Additionally, many specialists mentioned that e-consults capture previously informal curbside consults, enabling them to receive workload credit (Q13).
Lack of Awareness
Some noted that the biggest barrier to e-consults was not being aware of them generally, or which specialties offer e-consults (Q14). One PCP described e-consults as the best kept secret and found value in sharing the utility of e-consults with colleagues (Q15). All participants, including those who did not frequently use e-consults, felt that e-consults improved the quality of care by providing more timely care or better answers to clinical questions (Q16). Several practitioners also felt that e-consults increased access to specialty care. For example, specialists reported that e-consults enabled them to better manage patient load by using e-consults to answer relatively simple questions, reserving face-to-face consults for more complex patients (Q17).
Discussion
The objective of this study was to identify potential best practices for e-consults that may help increase their quality and use within the VA. We built on prior studies that offered insights on PCP and specialists’ overall satisfaction with e-consult by identifying several themes relevant to the further adoption of e-consults in the VA and elsewhere without a face-to-face visit.8,13,14,16–18 Future work may be beneficial in identifying whether the study themes identified can explain variation in e-consult use or whether addressing these factors might lead to increased or higher quality e-consult use. We are unaware of any qualitative study of comparable scale in a different health care system. Further, this is the first study to assess perspectives on e-consults among those who initiate and respond to them within the same health care system. Perhaps the most important finding from this study is that e-consults are generally viewed favorably, which is a necessary leverage point to increase their adoption within the system.
Clinicians reported several benefits to e-consults, including timely responses to clinical questions, efficient communication, allow for documentation of specialist recommendations, and help capture workload. These benefits are consistent with prior literature that indicates both PCPs and specialists in the VA and other health care systems feel that e-consults improves communication, decreases unnecessary visits, and improves quality of care.1,14,17,18 In particular, clinicians reported that e-consults improve their practice efficiency and efficacy. This is of critical importance given the pressures of providing timely access to primary and specialty care within the VA. Interestingly, many VA practitioners were unaware which specialties offered e-consults within their facilities, reflecting previous work showing that PCPs are often unaware of e-consult options.16 This may partially explain variation in e-consult use. Increasing awareness and educating clinicians on the benefits of e-consults may help promote use among non- and low users.
A common theme reported by both groups was the importance of providing necessary information within e-consult questions and responses. Specialists felt there was a need to ensure that PCPs provide relevant and patient-specific information that would enable them to efficiently and accurately answer questions without the need for extensive EHR review. This reflects previous work showing that specialists are often unable to respond to e-consult requests because they do not contain sufficient information.22 PCPs described a need to ensure that specialists’ responses included information that was detailed enough to make clinical decisions without the need for a reconsult. This highlights a common challenge to medical consultation, in that necessary or relevant information may not be apparent to all clinicians. To address this, there may be a role in developing enhanced, flexible templating that elicits necessary patient-specific information. Such a template may automatically pull relevant data from the EHR and prompt clinicians to provide important information. We did not assess how perspectives of templates varied, and further work could help define precisely what constitutes an effective template, including how it should capture appropriate patient data and how this impacts acceptability or use of e-consults generally. Collaboratively developed service agreements and e-consult templates could help guide PCPs and specialists to engage in efficient communication.
Another theme among both groups was that e-consult is most appropriate within specific clinical scenarios. Examples included review of laboratory results, questions about medication changes, or for patients who were reluctant to travel to appointments. Identifying and promoting specific opportunities for e-consults may help increase their use and align e-consult practices with scenarios that are likely to provide the most benefit to patients. For example, it could be helpful to understand the distance patients must travel for specialty care. Providing that information during clinical encounters could trigger clinicians to consider e-consults as an option. Future work might aim to identify clinical scenarios that clinicians feel are not well suited for e-consults and determine how to adapt them for those scenarios.
Limitations
Generalizability of these findings is limited given the qualitative study design. Participants’ descriptions of experiences with e-consults reflect the experiences of clinicians in the VA and may not reflect clinicians in other settings. We also interviewed a sample of clinicians who were already using e-consults. Important information could be learned from future work with those who have not yet adopted e-consult procedures or adopted and abandoned them.
Conclusions
E-consult is perceived as beneficial by VA PCPs and specialists. Participants suggested using e-consults for appropriate questions or patients and including necessary information and next steps in both the initial e-consult and response. Finding ways to facilitate e-consults with these suggestions in mind may increase delivery of high-quality e-consults. Future work could compare the findings of this work to similar work assessing clinicians perceptions of e-consults outside of the VA.
Electronic consultation (e-consult) is designed to increase access to specialty care by facilitating communication between primary care and specialty clinicians without the need for outpatient face-to-face encounters.1–4 In 2011, the US Department of Veterans Affairs (VA) implemented an e-consult program as a component of its overall strategy to increase access to specialty services, reduce costs of care, and reduce appointment travel burden on patients.
E-consult has substantially increased within the VA since its implementation.5,6 Consistent with limited evaluations from other health care systems, evaluations of the VA e-consult program demonstrated reduced costs, reduced travel time for patients, and improved access to specialty care.2,5–11 However, there is wide variation in e-consult use across VA specialties, facilities, and regions.5,6,12,13 For example, hematology, preoperative evaluation, neurosurgery, endocrinology, and infectious diseases use e-consults more frequently when compared with in-person consults in the VA.6 Reasons for this variation or specific barriers and facilitators of using e-consults have not been described.
Prior qualitative studies report that primary care practitioners (PCPs) describe e-consults as convenient, educational, beneficial for patient care, and useful for improving patient access to specialty care.8,14,15 One study identified limited PCP knowledge of e-consults as a barrier to use.16 Specialists have reported that e-consult improves clinical communication, but increases their workload.1,14,17,18 These studies did not assess perspectives from both clinicians who initiate e-consults and those who respond to them. This is the first qualitative study to assess e-consult perceptions from perspectives of both PCPs and specialists among a large, national sample of VA clinicians who use e-consults. The objective of this study was to understand perspectives of e-consults between PCPs and specialists that may be relevant to increasing adoption in the VA.
Methods
The team (CL, ML, PG, 2 analysts under the guidance of GS and JS and support from RRK, and a biostatistician) conducted semistructured interviews with PCPs, specialists, and specialty division leaders who were employed by VA in 2016 and 2017. Specialties of interest were identified by the VA Office of Specialty Care and included cardiology, endocrinology, gastroenterology, and hematology.
E-Consult Procedures
Within the VA, the specific procedures used to initiate, triage and manage e-consults are coordinated at VA medical centers (VAMCs) and at the Veterans Integrated Service Network (VISN) regional level. E-consult can be requested by any clinician. Generally, e-consults are initiated by PCPs through standardized, specialty-specific templates. Recipients, typically specialists, respond by answering questions, suggesting additional testing and evaluation, or requesting an in-person visit. Communication is documented in the patient’s electronic health record (EHR). Specialists receive different levels of workload credit for responding to e-consults similar to a relative value unit reimbursement model. Training in the use of e-consults is available to practitioners but may vary at local and regional levels.
Recruitment
Our sample included PCPs, specialists, and specialty care division leaders. We first quantified e-consult rates (e-consults per 100 patient visits) between July 2016 and June 2017 at VA facilities within primary care and the 4 priority specialties and identified the 30 sites with the highest e-consult rates and 30 sites with the lowest e-consult rates. Sites with < 500 total visits, < 3 specialties, or without any e-consult visit during the study period were excluded. E-consult rates at community-based outpatient clinics were included with associated VAMCs. We then stratified PCPs by whether they were high or low users of e-consults (determined by the top and bottom users within each site) and credentials (MD vs nurse practitioner [NP] or physician assistant [PA]). Specialists were sampled based on their rate of use relative to colleagues within their site and the use rate of their division. We sampled division chiefs and individuals who had > 300 total visits and 1 e-consult during the study period. To recruit participants, the primary investigator sent an initial email and 2 reminder emails. The team followed up with respondents to schedule an interview.
Interview guides were designed to elicit rich descriptions of barriers and facilitators to e-consult use (eAppendix available at doi:10.12788/fp.0214). The team used the Practical Robust Implementation and Sustainability Model (PRISM), which considers factors along 6 domains for intervention planning, implementation, and sustainment.19 Telephone interviews lasted about 20 minutes and were conducted between September 2017 and March 2018. Interviews were recorded and transcribed verbatim.
Analysis
The team used an iterative, team-based, inductive/deductive approach to conventional content analysis.20,21 Initial code categories were created so that we could identify e-consult best practices—facilitators of e-consult that were recommended by both PCPs and specialists. Inductive codes or labels applied to identify meaningful quotations, phrases, or key terms were used to identify emergent ideas and were added throughout coding after discussion among team members. Consensus was reached using a team-based approach.21 Four analysts independently coded the same 3 transcripts and met to discuss points of divergence and convergence. Analyses continued with emergent themes, categories, and conclusions. Atlas.ti. v.7 was used for coding and data management.22
Results
We conducted 34 interviews with clinicians (Table 1) from 13 VISNs. Four best-practice themes emerged among both PCPs and specialists, including that e-consults (1) are best suited for certain clinical questions and patients; (2) require relevant background information from requesting clinicians and clear recommendations from responding clinicians; (3) are a novel opportunity to provide efficient, transparent care; and (4) may not be fully adopted due to low awareness. Supporting quotations for the following findings are provided in Table 2.
Specific Clinical Questions and Patients
PCPs described specific patients and questions for which they most frequently used e-consults, such as for medication changes (Q1), determining treatment steps (Q2,3), and or clarifying laboratory or imaging findings. PCPs frequently used e-consults for patients who did not require a physical examination or when specialists could make recommendations without seeing patients face-to-face (Q3). An important use of e-consults described by PCPs was for treating conditions they could manage within primary care if additional guidance were available (Q4). Several PCPs and specialists also noted that e-consults were particularly useful for patients who were unable to travel or did not want face-to-face appointments (Q5). Notably, PCPs and specialists mentioned situations for which e-consults were inappropriate, including when a detailed history or physical examination was needed, or if a complex condition was suspected (Q6).
Background Data and Clear Recommendations
Participants described necessary data that should be included in high-quality e-consults. Specialists voiced frustration in time-consuming chart reviews that were often necessary when these data were not provided by the requestor. In some cases, specialists were unable to access necessary EHR data, which delayed responses (Q7). PCPs noted that the most useful responses carefully considered the question, used current patient information to determine treatments, provided clear recommendations, and defined who was responsible for next steps (Q8). PCPs and specialists stated that e-consult templates that required relevant information facilitated high-quality e-consults. Neither wanted to waste the other clinician's time (Q8).
A Novel Opportunity
Many PCPs felt that e-consults improved communication (eg, efficiency, response time), established new communication between clinicians, and reduced patients’ appointment burden (Q10, Q11). Many specialists felt that e-consults improved documentation of communication between clinicians and increased transparency of clinical decisions (Q12). Additionally, many specialists mentioned that e-consults capture previously informal curbside consults, enabling them to receive workload credit (Q13).
Lack of Awareness
Some noted that the biggest barrier to e-consults was not being aware of them generally, or which specialties offer e-consults (Q14). One PCP described e-consults as the best kept secret and found value in sharing the utility of e-consults with colleagues (Q15). All participants, including those who did not frequently use e-consults, felt that e-consults improved the quality of care by providing more timely care or better answers to clinical questions (Q16). Several practitioners also felt that e-consults increased access to specialty care. For example, specialists reported that e-consults enabled them to better manage patient load by using e-consults to answer relatively simple questions, reserving face-to-face consults for more complex patients (Q17).
Discussion
The objective of this study was to identify potential best practices for e-consults that may help increase their quality and use within the VA. We built on prior studies that offered insights on PCP and specialists’ overall satisfaction with e-consult by identifying several themes relevant to the further adoption of e-consults in the VA and elsewhere without a face-to-face visit.8,13,14,16–18 Future work may be beneficial in identifying whether the study themes identified can explain variation in e-consult use or whether addressing these factors might lead to increased or higher quality e-consult use. We are unaware of any qualitative study of comparable scale in a different health care system. Further, this is the first study to assess perspectives on e-consults among those who initiate and respond to them within the same health care system. Perhaps the most important finding from this study is that e-consults are generally viewed favorably, which is a necessary leverage point to increase their adoption within the system.
Clinicians reported several benefits to e-consults, including timely responses to clinical questions, efficient communication, allow for documentation of specialist recommendations, and help capture workload. These benefits are consistent with prior literature that indicates both PCPs and specialists in the VA and other health care systems feel that e-consults improves communication, decreases unnecessary visits, and improves quality of care.1,14,17,18 In particular, clinicians reported that e-consults improve their practice efficiency and efficacy. This is of critical importance given the pressures of providing timely access to primary and specialty care within the VA. Interestingly, many VA practitioners were unaware which specialties offered e-consults within their facilities, reflecting previous work showing that PCPs are often unaware of e-consult options.16 This may partially explain variation in e-consult use. Increasing awareness and educating clinicians on the benefits of e-consults may help promote use among non- and low users.
A common theme reported by both groups was the importance of providing necessary information within e-consult questions and responses. Specialists felt there was a need to ensure that PCPs provide relevant and patient-specific information that would enable them to efficiently and accurately answer questions without the need for extensive EHR review. This reflects previous work showing that specialists are often unable to respond to e-consult requests because they do not contain sufficient information.22 PCPs described a need to ensure that specialists’ responses included information that was detailed enough to make clinical decisions without the need for a reconsult. This highlights a common challenge to medical consultation, in that necessary or relevant information may not be apparent to all clinicians. To address this, there may be a role in developing enhanced, flexible templating that elicits necessary patient-specific information. Such a template may automatically pull relevant data from the EHR and prompt clinicians to provide important information. We did not assess how perspectives of templates varied, and further work could help define precisely what constitutes an effective template, including how it should capture appropriate patient data and how this impacts acceptability or use of e-consults generally. Collaboratively developed service agreements and e-consult templates could help guide PCPs and specialists to engage in efficient communication.
Another theme among both groups was that e-consult is most appropriate within specific clinical scenarios. Examples included review of laboratory results, questions about medication changes, or for patients who were reluctant to travel to appointments. Identifying and promoting specific opportunities for e-consults may help increase their use and align e-consult practices with scenarios that are likely to provide the most benefit to patients. For example, it could be helpful to understand the distance patients must travel for specialty care. Providing that information during clinical encounters could trigger clinicians to consider e-consults as an option. Future work might aim to identify clinical scenarios that clinicians feel are not well suited for e-consults and determine how to adapt them for those scenarios.
Limitations
Generalizability of these findings is limited given the qualitative study design. Participants’ descriptions of experiences with e-consults reflect the experiences of clinicians in the VA and may not reflect clinicians in other settings. We also interviewed a sample of clinicians who were already using e-consults. Important information could be learned from future work with those who have not yet adopted e-consult procedures or adopted and abandoned them.
Conclusions
E-consult is perceived as beneficial by VA PCPs and specialists. Participants suggested using e-consults for appropriate questions or patients and including necessary information and next steps in both the initial e-consult and response. Finding ways to facilitate e-consults with these suggestions in mind may increase delivery of high-quality e-consults. Future work could compare the findings of this work to similar work assessing clinicians perceptions of e-consults outside of the VA.
1. Battaglia C, Lambert-Kerzner A, Aron DC, et al. Evaluation of e-consults in the VHA: provider perspectives. Fed Pract. 2015;32(7):42-48.
2. Haverhals LM, Sayre G, Helfrich CD, et al. E-consult implementation: lessons learned using consolidated framework for implementation research. Am J Manag Care. 2015;21(12):e640-e647. Published 2015 Dec 1.
3. Sewell JL, Telischak KS, Day LW, Kirschner N, Weissman A. Preconsultation exchange in the United States: use, awareness, and attitudes. Am J Manag Care. 2014;20(12):e556-e564. Published 2014 Dec 1.
4. Horner K, Wagner E, Tufano J. Electronic consultations between primary and specialty care clinicians: early insights. Issue Brief (Commonw Fund). 2011;23:1-14.
5. Kirsh S, Carey E, Aron DC, et al. Impact of a national specialty e-consultation implementation project on access. Am J Manag Care. 2015;21(12):e648-654. Published 2015 Dec 1.
6. Saxon DR, Kaboli PJ, Haraldsson B, Wilson C, Ohl M, Augustine MR. Growth of electronic consultations in the Veterans Health Administration. Am J Manag Care. 2021;27(1):12-19. doi:10.37765/ajmc.2021.88572
7. Olayiwola JN, Anderson D, Jepeal N, et al. Electronic consultations to improve the primary care-specialty care interface for cardiology in the medically underserved: a cluster-randomized controlled trial. Ann Fam Med. 2016;14(2):133-140. doi:10.1370/afm.1869
8. Schettini P, Shah KP, O’Leary CP, et al. Keeping care connected: e-Consultation program improves access to nephrology care. J Telemed Telecare. 2019;25(3):142-150. doi:10.1177/1357633X17748350
9. Whittington MD, Ho PM, Kirsh SR, et al. Cost savings associated with electronic specialty consultations. Am J Manag Care. 2021;27(1):e16-e23. Published 2021 Jan 1. doi:10.37765/ajmc.2021.88579
10. Shipherd JC, Kauth MR, Matza A. Nationwide interdisciplinary e-consultation on transgender care in the Veterans Health Administration. Telemed J E Health. 2016;22(12):1008-1012. doi:10.1089/tmj.2016.0013
11. Strymish J, Gupte G, Afable MK, et al. Electronic consultations (E-consults): advancing infectious disease care in a large Veterans Affairs Healthcare System. Clin Infect Dis. 2017;64(8):1123-1125. doi:10.1093/cid/cix058
12. Williams KM, Kirsh S, Aron D, et al. Evaluation of the Veterans Health Administration’s Specialty Care Transformational Initiatives to promote patient-centered delivery of specialty care: a mixed-methods approach. Telemed J E-Health. 2017;23(7):577-589. doi:10.1089/tmj.2016.0166
13. US Department of Veterans Affairs, Veterans Health Administration, Specialty Care Transformational Initiative Evaluation Center. Evaluation of specialty care initiatives. Published 2013.
14. Vimalananda VG, Gupte G, Seraj SM, et al. Electronic consultations (e-consults) to improve access to specialty care: a systematic review and narrative synthesis. J Telemed Telecare. 2015;21(6):323-330. doi:10.1177/1357633X15582108
15. Lee M, Leonard C, Greene P, et al. Perspectives of VA primary care clinicians toward electronic consultation-related workload burden. JAMA Netw Open. 2020;3(10):e2018104. Published 2020 Oct 1. doi:10.1001/jamanetworkopen.2020.18104
16. Deeds SA, Dowdell KJ, Chew LD, Ackerman SL. Implementing an opt-in eConsult program at seven academic medical centers: a qualitative analysis of primary care provider experiences. J Gen Intern Med. 2019;34(8):1427-1433. doi:10.1007/s11606-019-05067-7
17. Rodriguez KL, Burkitt KH, Bayliss NK, et al. Veteran, primary care provider, and specialist satisfaction with electronic consultation. JMIR Med Inform. 2015;3(1):e5. Published 2015 Jan 14. doi:10.2196/medinform.3725
18. Gupte G, Vimalananda V, Simon SR, DeVito K, Clark J, Orlander JD. Disruptive innovation: implementation of electronic consultations in a Veterans Affairs Health Care System. JMIR Med Inform. 2016;4(1):e6. Published 2016 Feb 12. doi:10.2196/medinform.4801
19. Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf. 2008;34(4):228-243. doi:10.1016/s1553-7250(08)34030-6
20. Patton MQ. Qualitative Research and Evaluation Methods. 3rd ed. Sage Publications; 2002.
21. Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv Res. 2007;42(4):1758-1772. doi:10.1111/j.1475-6773.2006.00684.x
22. Kim EJ, Orlander JD, Afable M, et al. Cardiology electronic consultation (e-consult) use by primary care providers at VA medical centres in New England. J Telemed Telecare. 2019;25(6):370-377. doi:10.1177/1357633X18774468
1. Battaglia C, Lambert-Kerzner A, Aron DC, et al. Evaluation of e-consults in the VHA: provider perspectives. Fed Pract. 2015;32(7):42-48.
2. Haverhals LM, Sayre G, Helfrich CD, et al. E-consult implementation: lessons learned using consolidated framework for implementation research. Am J Manag Care. 2015;21(12):e640-e647. Published 2015 Dec 1.
3. Sewell JL, Telischak KS, Day LW, Kirschner N, Weissman A. Preconsultation exchange in the United States: use, awareness, and attitudes. Am J Manag Care. 2014;20(12):e556-e564. Published 2014 Dec 1.
4. Horner K, Wagner E, Tufano J. Electronic consultations between primary and specialty care clinicians: early insights. Issue Brief (Commonw Fund). 2011;23:1-14.
5. Kirsh S, Carey E, Aron DC, et al. Impact of a national specialty e-consultation implementation project on access. Am J Manag Care. 2015;21(12):e648-654. Published 2015 Dec 1.
6. Saxon DR, Kaboli PJ, Haraldsson B, Wilson C, Ohl M, Augustine MR. Growth of electronic consultations in the Veterans Health Administration. Am J Manag Care. 2021;27(1):12-19. doi:10.37765/ajmc.2021.88572
7. Olayiwola JN, Anderson D, Jepeal N, et al. Electronic consultations to improve the primary care-specialty care interface for cardiology in the medically underserved: a cluster-randomized controlled trial. Ann Fam Med. 2016;14(2):133-140. doi:10.1370/afm.1869
8. Schettini P, Shah KP, O’Leary CP, et al. Keeping care connected: e-Consultation program improves access to nephrology care. J Telemed Telecare. 2019;25(3):142-150. doi:10.1177/1357633X17748350
9. Whittington MD, Ho PM, Kirsh SR, et al. Cost savings associated with electronic specialty consultations. Am J Manag Care. 2021;27(1):e16-e23. Published 2021 Jan 1. doi:10.37765/ajmc.2021.88579
10. Shipherd JC, Kauth MR, Matza A. Nationwide interdisciplinary e-consultation on transgender care in the Veterans Health Administration. Telemed J E Health. 2016;22(12):1008-1012. doi:10.1089/tmj.2016.0013
11. Strymish J, Gupte G, Afable MK, et al. Electronic consultations (E-consults): advancing infectious disease care in a large Veterans Affairs Healthcare System. Clin Infect Dis. 2017;64(8):1123-1125. doi:10.1093/cid/cix058
12. Williams KM, Kirsh S, Aron D, et al. Evaluation of the Veterans Health Administration’s Specialty Care Transformational Initiatives to promote patient-centered delivery of specialty care: a mixed-methods approach. Telemed J E-Health. 2017;23(7):577-589. doi:10.1089/tmj.2016.0166
13. US Department of Veterans Affairs, Veterans Health Administration, Specialty Care Transformational Initiative Evaluation Center. Evaluation of specialty care initiatives. Published 2013.
14. Vimalananda VG, Gupte G, Seraj SM, et al. Electronic consultations (e-consults) to improve access to specialty care: a systematic review and narrative synthesis. J Telemed Telecare. 2015;21(6):323-330. doi:10.1177/1357633X15582108
15. Lee M, Leonard C, Greene P, et al. Perspectives of VA primary care clinicians toward electronic consultation-related workload burden. JAMA Netw Open. 2020;3(10):e2018104. Published 2020 Oct 1. doi:10.1001/jamanetworkopen.2020.18104
16. Deeds SA, Dowdell KJ, Chew LD, Ackerman SL. Implementing an opt-in eConsult program at seven academic medical centers: a qualitative analysis of primary care provider experiences. J Gen Intern Med. 2019;34(8):1427-1433. doi:10.1007/s11606-019-05067-7
17. Rodriguez KL, Burkitt KH, Bayliss NK, et al. Veteran, primary care provider, and specialist satisfaction with electronic consultation. JMIR Med Inform. 2015;3(1):e5. Published 2015 Jan 14. doi:10.2196/medinform.3725
18. Gupte G, Vimalananda V, Simon SR, DeVito K, Clark J, Orlander JD. Disruptive innovation: implementation of electronic consultations in a Veterans Affairs Health Care System. JMIR Med Inform. 2016;4(1):e6. Published 2016 Feb 12. doi:10.2196/medinform.4801
19. Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf. 2008;34(4):228-243. doi:10.1016/s1553-7250(08)34030-6
20. Patton MQ. Qualitative Research and Evaluation Methods. 3rd ed. Sage Publications; 2002.
21. Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv Res. 2007;42(4):1758-1772. doi:10.1111/j.1475-6773.2006.00684.x
22. Kim EJ, Orlander JD, Afable M, et al. Cardiology electronic consultation (e-consult) use by primary care providers at VA medical centres in New England. J Telemed Telecare. 2019;25(6):370-377. doi:10.1177/1357633X18774468